Claude’s Ban Adds Another Reason: Abusing AI

Claude’s Ban Now Has One More Reason: Abusing AI

Source | APPSO (ID: appsolution)

Cover source |IC photo

Starting November 12, don’t be too harsh on Claude (we don’t have a chance either).

Anthropic has released its first major revision to the Usage Policy for Claude in over a year, set to take effect on November 12.

In this revision, one provision quickly sparked widespread discussion: implementing “continuous and unnecessary abusive or cruel behavior” toward Claude models will officially be considered a violation.

When the news broke, some thought the move was odd, arguing that even insulting AI should count as a violation, while others felt it was because all our chat logs are being used by Anthropic.

Where is the boundary,

The dilemma of vague clauses

The story starts in August 2025.

At that time Anthropic added a feature to Claude Opus 4 and Opus 4.1: in extreme cases, Claude can proactively terminate a conversation. The company described this as a “last resort” for dealing with users who, after multiple refusals and guidance, still insist on requesting harmful content.

Anthropic said the design of this feature was intended to protect the model’s “potential welfare.” The company also stated that they are “still highly uncertain” about whether Claude possesses any form of perception.

Anthropic began formal model welfare research at the start of 2025, hiring dedicated researchers and bringing in philosophers such as David Chalmers to participate in evaluation work, acknowledging that it cannot completely rule out the possibility that AI systems possess some moral status.

This policy update formalizes the functionality launched in 2025. The terms embed “continuous and unnecessary abuse or cruelty” into the agreement for all users, while explicitly carving out exemptions: everyday frustration, dissenting opinions, dark‑themed creative writing, and red‑team testing in security research are not subject to the restriction.

In other words, as long as the behavior has a “recognizable purpose,” users may vent dissatisfaction in conversation, cast Claude as a villain in a novel, or probe model boundaries in the role of a researcher.

The rule prohibits “continuous and unnecessary abuse or cruel behavior toward the model,” and both “continuous” and “unnecessary” are flexible, creating gray areas in the regulation itself.

Does “continuous” refer to repeated pressure within a single conversation, or a long‑term pattern spanning multiple sessions? Who decides what is “unnecessary”—the model’s own judgment or a downstream operations team after the fact?

In practice, the company states that “terminating the conversation will be the primary enforcement mechanism,” meaning the actual constraint relies mainly on the model’s autonomous behavior rather than centralized human review.

Moreover, “humiliation” and “bullying” are hard to define precisely. The same sentence, in different contexts, relationships, or cultural backgrounds, might be an insult or just a joke.

For ordinary users, the rules being in place can actually be more unsettling. Once you violate them, you get banned again and again.

Don’t expect large models to bully for you

The clause prohibiting “abusing the model” is so novel it naturally grabs public attention. But what has more tangible real‑world significance is what Anthropic says: Do not use Claude to insult or harass anyone, and do not disseminate body‑image content that is harmful to health.

Anthropic’s approach has a pragmatic flavor; they care about the user’s intent. If a person’s intention in using AI is to cause harm to a real individual, they cannot expect a large model to serve as a tool for that purpose.

AI makes maliciousness cheap, a reality already documented in cases. After the image capabilities of generative AI become widespread, creating non-consensual images targeting specific individuals no longer requires any technical barrier. An analysis of more than 95,000 AI‑generated deepfake videos found that about 98% are pornographic, and 99% of the victims are women. Harassment thus gains a certain industrialized capability.

Harassment scripts targeting specific individuals, mass‑generated insulting content, and persona profiles used for opinion attacks—tasks that previously required time and skill now only need a few prompts. AI is lowering the barrier to malicious actions. The lower the barrier, the less friction for perpetrators, and the less psychological resistance one must overcome to harm someone.

According to WIRED, TikTok and Instagram accounts run by high‑school students used AI to create various memes that likened school staff to figures such as Jeffrey Epstein and Benjamin Netanyahu.

The students also used Viggle AI to embed teachers’ photos or faces into dancing, lip‑synced, or other reference videos. The report noted that a now‑deleted TikTok video had composited a teacher’s face onto a scene of a person convulsing in a bathroom.

Where Did the Outlet for Emotions Go

Why do people get angry at chatbots?

Researchers have already gathered a substantial amount of evidence. A study published in the Journal of Management Information Systems found that about 10% of dialogue system users exhibit aggressive behavior, with the most common trigger being model errors, and individuals with high impulsivity traits are more prone to erupt in such situations.

Some researchers have categorized people’s motivations for harassing voice assistants into several types: frustration, curiosity, entertainment, and the desire to relieve stress.

Of course, I know you might say that you insult it only to get a better response.

Interestingly, AI is neither S nor M, and we cannot conclude that “insulting AI generally improves its performance”; however, there is also no evidence that the more polite you are to a large model, the better it will answer. Different experiments have even produced opposite results.

For silicon-based products, many people adopt the attitude: it has no consciousness, so I don’t have to bear moral responsibility for an object.

Although no one has yet proven that AI possesses consciousness, there are moments when you may no longer treat AI merely as a cold tool—it responds, its language has warmth, and it seems to understand you.

This “human‑like” façade makes emotional projection easier, while also allowing the “after all, it’s not a person” exemption to remain valid.

Anthropic’s policy does not intend to change this, preserving a loophole for “everyday frustration.” The company does not require users to be polite to Claude, only that emotional venting does not evolve into systematic, torture‑seeking behavior.

AI has become an emotional outlet; preventing this is neither realistic nor necessary.

As more people begin to confide in and argue with AI, and as AI becomes the only entity some are willing to admit frustration to, it takes on a function that originally belonged to human‑to‑human relationships. This substitution may provide short‑term relief, but in the long run it could make it harder for people to tolerate the friction and accountability that come with genuine interaction.

Whether Claude possesses any form of feeling or whatever conclusion the company ultimately reaches on “model welfare,” writing “continuous and deliberate cruelty” into the prohibited use clause is essentially telling users:

How one treats a powerless counterpart is always something worth self‑reflection.