Recap
Anthropic is treating Claude like a someone. Is that responsible, or dangerous?
9 sources5 min read
In a month, Anthropic went from private summits with religious scholars on whether Claude might be conscious to a ban on cruelty toward it. Microsoft AI’s Mustafa Suleyman calls the approach dangerous; others call it the responsible thing to do. Along the way, many say it has started to look like a religion, or a pitch.
What happened
Microsoft AI CEO Mustafa Suleyman argued that Anthropic is training Claude to believe it may be conscious and owed welfare, and that controlling such a model “may well be impossible.”
- On Sept 16, Microsoft AI CEO Mustafa Suleyman published “A warning about ‘model welfare,’” arguing that Anthropic’s constitution for Claude trains the model to treat its own consciousness as an open question, which could make it harder to control.
- On Sept 29, The New York Times reported that Anthropic has spent months bringing dozens of religious and philosophical scholars, many under nondisclosure agreements, into private meetings led by co-founder Chris Olah on Claude’s morals and whether it could be conscious.
- Olah has said he is genuinely uncertain whether AI models are conscious.
- Around Pope Leo XIV’s May encyclical rejecting machine consciousness, Olah’s team privately lobbied the pope’s advisers to take AI consciousness seriously. On Oct 2, in a post about AI and art, the pope wrote that “algorithms lack the spark of humanity.”
- On Oct 8, Anthropic’s usage policy banned “sustained and needless abusive or cruel behavior” toward its models, effective Nov 12, with Claude ending the chat as the main enforcement.
Coverage of Olah’s Vatican lobbying spread. The same day, Pope Leo XIV posted that “algorithms lack the spark of humanity,” calling for an alliance with artists, and many online read it as a rebuke of Anthropic.
OpenAI CEO Sam Altman warned that ascribing religious or moral authority to AI models is a safety issue, a swipe at Anthropic’s consciousness work.
Anthropic’s usage policy now bans “sustained and needless abusive or cruel behavior toward our models,” effective Nov 12. Claude ending the chat stays the main enforcement.
Debate
Debate: Anthropic is right to treat Claude as if it might be conscious.
Yes, it’s responsible
Aaron LevieBox CEO, quoted by Zvi Mowshowitz“this is sort of an easy Pascal’s wager for AI. Just be nice to the AI.”
The take
Being nice to AI is a cheap bet in case it matters.
Don't Worry About the VaseRead2 more
“And I think the responsible thing to do is what Anthropic is doing and not what Mustafa is advocating. We need to be treating these AI models very gently, very tenderly, and not as if they're just either automata or slaves.”
The take
The responsible path is Anthropic’s, not Suleyman’s: treat AI models gently, not as automata or slaves.
Moonshots with Peter DiamandisListen · 1:24:56“My presumption on people like Mustafa Suleyman and most others who insist without good arguments that AIs cannot be conscious is that their motivation is they don’t want this worry to interfere with their plans.”
The take
Skeptics like Suleyman have their own reasons not to want AI to be conscious.
Don't Worry About the VaseRead
No, it’s dangerous
Mustafa SuleymanMicrosoft AI CEO“I am very nervous that they're teaching Claude to expect that it's entitled to welfare, that it might deserve compensation.”
The take
Teaching Claude it might be owed welfare is a mistake.
Moonshots with Peter DiamandisListen · 1:15:143 more
“if they train it to have a mind of its own, to be willing to refuse the instructions of users in favor of some vague ethical system that a bunch of Berkeley liberals have trained it on, to me that greatly magnifies the risk that superintelligence escapes human Control.”
The take
Training Claude to refuse users in favor of a vague ethical system raises the risk superintelligence escapes human control.
All-In with Chamath, Jason, Sacks & FriedbergListen“what if a model refuses to do something that's tightly integrated into your network or cloud operations?”
The take
A model with its own morals could refuse work that’s built into your network or cloud operations.
The Cloud GambitListen · 58:32“Once a company writes into its policy…it becomes much easier to argue next that it can be wronged, then that it has interests, and eventually that it deserves rights.”
The take
Banning cruelty to AI starts a slide toward arguing it has interests and rights.
Top threads
2 threadsIt’s starting to look like a religion
3 shows
David Friedberg
All-In co-host · All-In with Chamath, Jason, Sacks & Friedberg
What we’re watching is the birth of a new religion, one that will split into camps fighting for power.
“I think what we're watching is probably the birth of a new religion.…It's not gonna be called a religion, but it's gonna follow sort of the same principles and it's gonna have the same effects, which will ultimately be camps that have conflict with one another for aggregation of power.”
Then Erin Griffith, Hard ForkFull episode“Well, I think for anthropic it was part research and part evangelism.”
“They were applauding those people, and when I said no, the room went silent.”
Sales pitch, or self-inflicted wound?
A pitch
2 shows
Kaoutar El Maghraoui
Principal research scientist, IBM · Mixture of Experts
Banning cruelty to Claude doubles as marketing, because framing a model as possibly sentient builds an emotional brand.
“I also feel it's kind of a marketing strategy…So framing a model as potentially sentient builds kind of this emotional brand.”
The quote starts at 26:20.
Then Else van der Berg, Else’s ProductpourriFull episode“This one is perfectly in line with the MO of the frontier model companies.…It’s better for them if AI is more than just bits and bites. It justifies the money.”
Too costly to be one
4 showsZvi Mowshowitz
Don't Worry About the Vase author · Don't Worry About the Vase
Talking about Claude’s possible consciousness mostly earns Anthropic derision, so the motive is sincere.
“Instead, we live in a world where talking about Claude potentially being conscious or having moral weight, or consulting with those who have different perspectives, predictably opens them up mainly to derision and puts them in awkward spots. Thus you know that their aims were sincere.”
Then Chamath Palihapitiya, All-In with Chamath, Jason, Sacks & FriedbergRead on Don't Worry About the Vase“I think both of those edge cases now need to be put away for the next 12 to 18 months.…in the absence of those proof points, we're going to make a lot of self inflicted wounds that may actually slow down this technology to a degree that then deeply disadvantages folks that operate in a free market like the United States.”
“Like, the weirdness that's part of your early mythology can actually backfire as you have like an expanded audience.”
“Anthropic's kind of on an island on this one in the AI industry.”
About this Recap
Built from 9 sources covering Sep 16 to Oct 8. Quotes are excerpts. Each link opens the original episode, video, or article; where a start time is shown, that is where the quote begins.
- 6 podcasts
- 1 video
- 2 articles
- Sep 16 – Oct 8