Anthropic Wants to Pause AI Development — After Reaching a $965 Billion Valuation

On June 4, 2026, the Anthropic Institute published a report titled "When AI Builds Itself," accompanied by a call for the world's leading AI companies to establish a coordinated mechanism to slow or temporarily halt the development of frontier AI systems [1]. The stated reason: AI models are approaching "recursive self-improvement" — the ability to autonomously design and build their own successors — and society is not ready [2].

The proposal landed like a grenade in an industry already under intense geopolitical and commercial pressure. Within hours, it drew pointed responses from competitors, White House officials, and analysts who questioned whether the most valuable AI company on Earth was genuinely sounding an alarm or pulling up the ladder behind it [3].

What Anthropic Is Actually Proposing

The report, authored by Anthropic co-founder Jack Clark and Anthropic Institute director Marina Favaro, stops short of demanding an immediate halt. Instead, it argues that the industry should build a "brake pedal" — a pre-negotiated framework under which multiple frontier AI labs in multiple countries would agree to stop development simultaneously under verifiable conditions [4].

"The AI industry right now has a gas pedal, but it doesn't have a brake pedal in the car, and we want to do some of the work to build that pedal," Clark told the BBC [5].

The proposal is conditional: Anthropic says it would slow down or pause only if other frontier developers — OpenAI, Google DeepMind, Meta AI, and xAI among them — implement verifiable measures at the same time [1]. The company has not specified how long a pause would last, what safety benchmarks would need to be met before development resumes, or who would adjudicate whether those thresholds have been crossed. The report acknowledges that verification infrastructure "could take decades to establish" [6].

Anthropic plans to convene government officials, scientists, advocacy groups, and competing AI firms "in coming months" to work out the details [3].

The Risk: Recursive Self-Improvement

The specific capability Anthropic flags is recursive self-improvement — an AI system that can "fully autonomously" design and develop its own successor without human involvement [6].

The company offers its own operations as evidence of the trajectory. As of May 2026, more than 80% of the code merged into Anthropic's codebase was authored by Claude, up from low single digits before Claude Code launched in February 2025 [6]. Engineers at Anthropic are now shipping roughly eight times as much code per quarter as in 2024 [6]. Claude's success rate on open-ended programming tasks jumped from around 15% in late 2025 to over 70% by spring 2026 [6].

Source: Anthropic Institute

Data as of Jun 4, 2026CSV

The company also disclosed that an internal model called "Mythos Preview" achieved a 52x speedup in code optimization and selected better research directions than human researchers 64% of the time [6]. Task complexity has scaled from four-minute tasks in March 2024 to 12-hour tasks by March 2026, with week-long autonomous tasks projected for 2027 [6].

Anthropic frames these advances as evidence that the threshold for full recursive self-improvement could arrive "sooner than most institutions are prepared for" [7], though the company acknowledges "this threshold has not yet been crossed" [1].

The Credibility Problem: February's Dropped Safety Pledge

The proposal's reception cannot be separated from Anthropic's own recent history. In February 2026, the company formally dropped the central commitment of its Responsible Scaling Policy (RSP) — a pledge, in place since 2023, never to train an AI system unless it could guarantee in advance that safety measures were adequate [8].

The revised RSP v3.0 replaced that categorical pause trigger with a softer dual condition: the company would only halt development if it simultaneously held an AI capability lead and identified a material catastrophic risk [9]. The new policy explicitly stated: "If one AI developer paused development to implement safety measures while others moved forward training and deploying AI systems without strong mitigations, that could result in a world that is less safe" [9].

The timing is hard to ignore. Four months ago, Anthropic argued that unilateral pauses were counterproductive and removed its own hard safety limits. Now it is calling for a coordinated global pause — a proposal that, by its own admission, requires verification infrastructure that does not yet exist and may take decades to build.

CNN reported that the RSP change came amid pressure from the Pentagon, which was evaluating Anthropic's models for defense applications [10]. The OECD flagged the revision as raising "catastrophic risk concerns" [11].

Follow the Money

Anthropic's financial position makes the pause call harder to read as purely altruistic.

The company raised $65 billion in a Series H round in May 2026 at a $965 billion valuation, surpassing OpenAI to become the most valuable AI startup in the world [12]. That nearly tripled its valuation from a $380 billion Series G just three months earlier [13]. Its annualized revenue run rate reached $47 billion as of late May 2026, up from $10 billion in annual revenue the prior year [12]. Eight of the Fortune 10 are Claude customers [12].

Source: TechCrunch, CNBC

Data as of May 28, 2026CSV

Days before publishing the pause proposal, Anthropic filed confidential IPO paperwork [7]. The company had recently launched its Claude 4-series models, including Claude Opus 4, which it marketed as its most capable system to date.

The Strategic Critique

Critics argue that the timing is not coincidental. OpenAI CEO Sam Altman offered the sharpest public response: "It is clearly incredible marketing to say, 'We have built a bomb, we are about to drop it on your head. We will sell you a bomb shelter for $100 million'" [3].

David Sacks, a venture capitalist and Trump adviser on AI policy, has accused Anthropic of running a "regulatory capture agenda" — using safety rhetoric to encourage regulations that would burden cheaper open-source competitors while protecting proprietary systems like Claude [3].

Holger Mueller of Constellation Research questioned the competitive logic: Does the pause proposal aim to "freeze the status quo so it can catch up, or simply retain its lead?" [1]

Rob Enderle of the Enderle Group called implementation "practically impossible, because the economic and national security stakes are simply too high for any superpower to willingly hit the brakes now" [1].

White House officials have also pushed back, saying Anthropic's focus on worst-case scenarios overstates current risks and functions as a strategy for slowing rivals under safety cover [3].

The steelman case for strategic self-interest is straightforward: Anthropic has just released its most capable models, secured nearly $1 trillion in valuation, and filed for an IPO. A pause that freezes frontier development at this moment locks in Anthropic's position while competitors — particularly Meta's open-source Llama models, which operate under a different commercial logic — would be disproportionately affected.

The Defense: Why Anthropic Says This Is Different

Anthropic's defenders argue that the company has more to lose commercially from a pause than to gain, given its revenue growth trajectory. The proposal explicitly requires multilateral participation — Anthropic is not calling for a unilateral halt, which would simply hand the lead to competitors [1].

The company also points to its track record: it has published more safety research than any other frontier lab, pioneered constitutional AI methods, and invested in interpretability research [14]. Clark's disclosure that 80% of Anthropic's own code is AI-authored is itself a form of transparency few competitors have matched.

Dario Amodei, Anthropic's CEO, has argued in previous essays that the competitive dynamics make unilateral action impossible — "we have geopolitical adversaries building the same technology at a similar pace" — which is precisely why coordinated mechanisms are necessary [15].

What Independent Researchers Say

The academic AI safety community has grown rapidly, with over 44,500 papers on AI safety and alignment published in 2025 alone, though the pace has slowed in 2026 [16].

Source: OpenAlex

Data as of Jan 1, 2026CSV

Independent researchers have offered mixed reactions to the pause concept. The fundamental critique, articulated by safety researchers since the 2023 pause letter, is that safety risks accumulate primarily from the deployment of existing models — not from new training runs. Pausing development does nothing to address harms from the billions of interactions already happening daily with current systems [17].

The verification problem is also severe. Unlike nuclear facilities, AI training runs occur in distributed data centers using commercially available hardware. As one analyst noted, "tracking decentralized computing resources, private data centers and algorithmic research globally is far more difficult than monitoring something physical, like nuclear facilities" [1].

A 2023 paper from researchers at the University of Oxford proposed a "coordinated pausing" framework with evaluation-based triggers, but acknowledged that no enforcement mechanism currently exists [18].

The 2023 Precedent: What Happened Last Time

Anthropic's proposal inevitably invites comparison to the March 2023 open letter from the Future of Life Institute, which called on "all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4" [19]. The letter gathered over 30,000 signatures, including from Elon Musk, Yoshua Bengio, and Steve Wozniak [19].

No signatory complied. Six months later, Axios reported that "no one took a six-month pause in AI work" [20]. Musk himself launched xAI to build competing AI systems. OpenAI released GPT-4 Turbo. Google DeepMind shipped Gemini. Anthropic released Claude 2 and Claude 3.

The letter did, however, shift public discourse. It normalized discussion of AI existential risk, contributed to the White House securing voluntary safety commitments from major labs, and accelerated regulatory efforts in the EU and China [17].

The lesson: voluntary pauses in a competitive market do not hold. Anthropic's proposal attempts to address this by conditioning any pause on multilateral, verifiable participation — but it offers no mechanism to achieve that verification.

The Regulatory Landscape

The realistic path to enforcing any pause runs through governments, and the landscape is fragmented.

The EU AI Act, which entered full enforcement in 2025, regulates AI applications by risk category but does not include provisions to halt development of frontier models [3]. The US has no comprehensive federal AI legislation; the Trump administration has generally favored deregulation and voluntary commitments over binding rules [3]. President Trump has said he discussed AI safety cooperation with China during a recent visit to Beijing, but no formal agreements have emerged [3].

China has implemented its own AI regulations focused on content control and algorithmic transparency, but has shown no interest in pausing frontier development — indeed, Chinese labs including DeepSeek and Baidu have accelerated their efforts [3].

An internationally binding treaty on AI development would require years of negotiation and face the same enforcement challenges as arms control agreements, with the added difficulty that AI capabilities are harder to verify than physical weapons systems [1].

Anthropic has said it will bring together stakeholders "in coming months" to work toward a coordination framework, but has not disclosed which governments or international bodies have been formally briefed on the proposal [3].

What the Proposal Does and Does Not Do

Anthropic's pause call is best understood not as a concrete policy proposal but as an opening position in a negotiation that has barely begun. The company has identified a real technical concern — recursive self-improvement — and published detailed internal metrics to substantiate it. But the proposal lacks the specifics that would make it actionable: no defined duration, no safety benchmarks, no enforcement mechanism, no adjudication body, and no clear answer to the question of what happens if one major player defects.

The company's own recent behavior — dropping its internal safety pledge in February, accelerating model releases, and filing for an IPO — creates a credibility gap that the proposal's critics have been quick to exploit.

Whether Anthropic's call is sincere, strategic, or both, it has restarted a conversation the industry has been avoiding since the 2023 pause letter failed. The question is whether the conversation will produce anything more durable this time — or whether, as before, the labs will keep building while the debate continues.

Key Takeaways

• The proposal is conditional and vague: Anthropic will pause only if competitors do too, under verification systems that do not exist and may take decades to build.
• The financial timing is conspicuous: The call comes days after confidential IPO filings and months after Anthropic reached a $965 billion valuation.
• The credibility gap is real: Anthropic dropped its own flagship safety pledge in February 2026, four months before calling for an industry-wide pause.
• The 2023 precedent is not encouraging: The last major pause effort produced zero compliance from any signatory.
• The technical concern has substance: Claude now writes 80% of Anthropic's code, and the trajectory toward recursive self-improvement is supported by published metrics.
• Enforcement is an unsolved problem: No government, treaty, or international body currently has the authority or capacity to verify or enforce a pause on AI development.