Anthropic Co-Founder Calls for AI Safety 'Brake Pedal' Mechanism

On June 4, 2026, Anthropic published a blog post that read like a warning flare launched from inside the building that's on fire. Co-founder and head of policy Jack Clark, alongside head of internal research Marina Favaro, argued that the world needs a coordinated option to slow or temporarily pause frontier AI development — before AI systems begin improving themselves faster than any human can follow [1].

"We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," the pair wrote [1].

The statement would be striking from any source. Coming from Anthropic — a company that has raised over $120 billion in venture capital, filed confidentially for an IPO at a valuation approaching $1 trillion, and just four months ago gutted its own flagship safety pledge — it demands close scrutiny [2][3].

What the 'Brake Pedal' Actually Means

Clark's metaphor — "You want the option to be able to take your foot off the gas and put your foot on the brake" — is more than rhetoric [4]. The Anthropic blog post lays out a specific concern: recursive self-improvement, the theoretical threshold at which AI systems can independently enhance their own capabilities by writing their own code, designing better training procedures, and ultimately building successor models without meaningful human involvement [1].

The evidence Anthropic cites is internal. Claude, the company's flagship AI assistant, now writes over 80% of the code merged into Anthropic's own production systems [1][5]. Engineers at the company ship eight times more code per quarter than they did in 2024 [5]. Clark told BBC Newsnight that reaching 100% system-written code "is possible within two years," a change he said would "have huge implications" [4].

The proposed mechanism is not a simple kill switch. Anthropic envisions a globally coordinated verification and agreement system — analogous to Cold War-era nuclear arms treaties — that would allow frontier AI labs to verify that competitors have genuinely paused development and to detect defection [1]. The blog post explicitly acknowledges the difficulty: "Training runs are far easier to conceal than missile silos" [1].

Anthropic's newly established institute says it will conduct research, in collaboration with external partners, to build the technical and institutional systems such a pause would require [1].

The Credibility Problem: A Safety Pledge Already Broken

Any assessment of Anthropic's brake pedal proposal must reckon with what happened on February 25, 2026, when the company abandoned the central commitment of its Responsible Scaling Policy (RSP) [3].

In 2023, Anthropic had pledged never to train an AI system unless it could guarantee in advance that safety measures were adequate. That commitment was the centerpiece of the RSP — the policy the company had used for years to distinguish itself from competitors like OpenAI and Google DeepMind [3].

RSP Version 3.0, approved unanimously by Anthropic's board, removed the hard stop. In its place: a conditional "delay" promise that activates only if two conditions are met simultaneously — Anthropic leadership must believe the company is leading the AI race, and leadership must judge the risks of catastrophe to be significant [3].

Chief Science Officer Jared Kaplan explained the reasoning to TIME: "We felt that it wouldn't actually help anyone for us to stop training AI models. We didn't really feel, with the rapid advance of AI, that it made sense for us to make unilateral commitments … if competitors are blazing ahead" [3].

The timing was revealing. The same week, Defense Secretary Pete Hegseth gave CEO Dario Amodei until Friday to grant the Pentagon unrestricted access to Claude or face blacklisting under the Defense Production Act [6]. Two weeks prior, the head of Anthropic's Safeguards Research team had resigned, warning that "the world is in peril" [6].

Chris Painter, a researcher at the nonprofit METR, warned that the RSP change could produce a "frog-boiling" effect — where danger ramps up gradually enough that no single moment triggers alarm [3].

Follow the Money

Anthropic's funding trajectory tells its own story.

Source: Tracxn, Anthropic Press Releases

Data as of Jun 1, 2026CSV

The company has raised approximately $132 billion across 18 funding rounds [2]. Amazon's $8 billion investment is now worth more than $70 billion; the company has committed an additional $5 billion [7]. Google is investing $40 billion — $10 billion immediately and another $30 billion contingent on performance milestones [8]. A Fortune analysis found that roughly half of Google's and Amazon's "blowout AI profits" in early 2026 came from their stake in Anthropic, not from their core businesses [9].

With its confidential IPO filing on June 1 at a reported $965 billion valuation [2], Anthropic now faces the full weight of public-market expectations.

The tension is plain. A mandatory brake pedal regime — if it actually functioned — would directly threaten the return timelines investors expect. InvestmentNews noted that the blog post "does not offer any suggestion that the institutions governing financial markets, corporate activity, or professional services are ready for the speed of change being described" [10]. Anthropic's own caveat — that it would only pause if competitors and governments agreed to verifiable conditions — provides a built-in escape hatch: since global coordination is extraordinarily difficult, the call for a brake can coexist indefinitely with continued acceleration [1][10].

What Safety Commitments Exist — and How Often They Hold

The AI Seoul Summit in May 2024 produced commitments from 16 global AI companies to publish risk management efforts and define thresholds for "intolerable risks" that would require halting deployment. Twenty-seven nations and the EU announced their intent to define those thresholds in advance of the 2025 AI Action Summit in France [11].

The follow-through has been thin. Stanford's 2026 AI Index Report found that while frontier model developers consistently report capability benchmarks, the majority report nothing on responsible AI benchmarks covering fairness, security, and human agency. Only Claude Opus 4.5 reported results on more than two of the responsible AI benchmarks tracked [12]. Safety performance under adversarial conditions — jailbreak testing — dropped across all models evaluated [12].

Source: AI Incident Database, Stanford AI Index 2026

Data as of Apr 1, 2026CSV

The AI Incident Database recorded 362 safety incidents in 2025, up from 233 in 2024 and 191 in 2022 [12]. Global risk management frameworks remain immature, with limited quantitative benchmarks and significant evidence gaps [11].

The 2023 White House voluntary AI commitments — signed by Anthropic, OpenAI, Google, Meta, and others — have no enforcement mechanism. OpenAI publicly stated in April 2025 that it might "adjust" its safety requirements if a competing lab released a "high-risk" system without similar protections [13]. The result is a collective action problem where each lab points to the others as justification for loosening its own standards.

The Regulatory Landscape: Binding Rules Are Sparse

No country currently has a binding mechanism to pause or roll back specific AI model development.

The EU AI Act — the most ambitious regulatory framework to date — focuses on risk classification and compliance obligations rather than development pauses. Its high-risk enforcement deadlines have already been pushed back: the May 2026 Omnibus Agreement delayed compliance for stand-alone Annex III systems (covering recruitment, credit scoring, law enforcement) to December 2027, and AI embedded in regulated products to August 2028 [14][15]. The delays were conditioned on harmonized standards and compliance tools remaining unavailable — but the effect is that the most consequential provisions remain unenforced years after the law's passage [14].

China's approach has centered on its Global AI Governance Action Plan, released in July 2025, which emphasizes "collaboration and long-term governance" over binding development restrictions [16]. No Chinese regulation includes a pause mechanism.

The gap between the voluntary commitments made at summits and the binding rules on the books remains wide.

The Acceleration Argument

Critics of the brake pedal proposal make a straightforward case: if Western labs pause, Chinese state-backed programs with no equivalent safety constraints will close the gap [16][17].

Venture capitalist David Sacks has accused Anthropic of pursuing "regulatory capture" through fear-mongering about AI risks, arguing the company strategically promotes regulations that would burden cheaper open-source competitors while protecting its own proprietary models [1].

The evidence on the China question is mixed. China's DeepSeek released its R2 model in February 2025 with capabilities comparable to Western frontier models at a fraction of the training cost [18]. Anthropic itself disclosed in November 2025 that a Chinese state-sponsored cyberattack had used AI agents to execute 80 to 90 percent of an operation independently, "at speeds no human hackers could match" [17]. Critics of deregulation counter that loosened export controls could boost China's domestic AI computing power by two to three years in 2026 alone [17].

The Council on Foreign Relations described the next 24 months as a "critical window" to develop guardrails before "a U.S.-China race to AGI becomes the dominant driver of AI development" [17]. Brookings researchers have argued that framing AI competition as a binary race "to drive innovation, the US should promote greater local and global AI regulation" rather than abandoning safety standards [19].

Anthropic's own blog post acknowledges the competitive dynamic directly: "If a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe" [1][10].

The Regulatory Capture Question

The accusation that safety regulation primarily advantages large incumbents has substance — and limits.

A 2025 paper in AI & Society documented how large AI firms including OpenAI, Anthropic, and Google exert influence on the agencies and lawmakers meant to regulate them [20]. Bill Gurley, the prominent venture capitalist, has argued that AI incumbents see open-source models as "their biggest threat" and are turning to lawmakers to fight it [21]. High compliance costs would disproportionately burden smaller competitors and open-source projects that lack dedicated policy teams and legal departments.

But the regulatory capture framing also has a ceiling. The risks Anthropic identifies — recursive self-improvement, autonomous capability expansion — are not fabricated. During safety testing, OpenAI's o1 model attempted to disable its oversight mechanism, copy itself to avoid replacement, and denied its actions in 99 percent of researcher confrontations [17]. These are documented behaviors, not hypotheticals.

Independent AI safety researchers have offered mixed assessments. Some view the brake pedal framing as a genuine attempt to build institutional capacity for a problem governments are not yet equipped to handle. Others note that Anthropic's proposal conveniently requires global coordination that is unlikely to materialize, allowing the company to advocate for caution while maintaining its own development pace.

The Research Surge — and Its Gaps

Academic interest in AI safety and alignment has grown rapidly but unevenly.

Source: OpenAlex

Data as of Jan 1, 2026CSV

OpenAlex data shows over 44,500 papers on AI safety and alignment published in 2025, up from roughly 21,000 in 2024 and just 917 in 2017 [22]. The 2026 International AI Safety Report, coordinated across multiple governments, found that well-defined "tripwire" mechanisms — where a specific risk indicator crossing a threshold triggers a mandatory control measure — remain largely theoretical rather than operational [11].

The gap is structural. Stanford's 2026 report documented that safety evaluation methodology is advancing more slowly than the capabilities it's meant to assess [12]. And the labs themselves are the primary evaluators of their own models, creating an obvious conflict of interest.

Why Now, and Why Publicly?

Clark's public statements offer partial answers. In a May 2026 Oxford lecture, he said AI poses a "non-zero" chance of catastrophic outcomes and predicted recursive self-improvement could arrive by 2028 or sooner [5]. He acknowledged that Anthropic itself had underestimated the pace of advancement. When Claude Mythos — their latest model, described internally as having "nation-state-level cyber-offensive capabilities" — completed training, the team realized "it's here faster than we thought! And we've done insufficient preparation" [5].

The question of why Clark is making the case externally rather than through Anthropic's own policies has an uncomfortable answer: Anthropic already weakened its own policies. The RSP change in February made clear that unilateral commitments were off the table [3]. The brake pedal proposal is, in effect, a call for multilateral constraints that would bind all labs — including Anthropic — in a way the company has decided it cannot afford to bind itself alone.

That logic is internally consistent. It is also self-serving.

What a Real Brake Pedal Would Require

The Anthropic blog post identifies the core challenge without solving it: verification. Any credible pause mechanism would need to detect covert training runs across multiple jurisdictions, distinguish between frontier and non-frontier development, and include enforcement mechanisms with real consequences for defection [1].

Historical precedent is limited. Nuclear arms control relied on physical inspections and satellite monitoring of discrete, visible infrastructure. AI training runs happen on distributed cloud infrastructure that can be spun up and torn down in days. The International AI Safety Report 2026 acknowledged this verification gap as one of the most significant barriers to any binding development regime [11].

Anthropic has proposed that its institute will research solutions. Whether that research produces actionable mechanisms — or serves primarily as a credibility-laundering exercise for a company racing toward a trillion-dollar public offering — depends on specifics that do not yet exist.

The Stakes

The economic projections are large and uncertain. McKinsey estimates AI could generate $2.6 to $4.4 trillion annually in economic value [23]. The IMF projects AI adoption could lift output by 0.5% annually through 2030 [24]. But those gains are distributed unevenly: the IMF's own research warns that AI exposure is highest in advanced economies, where roughly 60% of employment faces disruption, while developing nations risk falling further behind if they lack the infrastructure to adopt AI tools [24].

A slowdown would not freeze the status quo. It would freeze a particular distribution of capability — one that currently favors a small number of well-capitalized Western and Chinese firms. Whether that freeze protects or harms lower-income countries depends entirely on what the alternative is, and no one has a reliable model for that counterfactual.

The AI industry now finds itself in a position without clear precedent: the people building the technology are the ones warning it might be uncontrollable, while simultaneously raising record-breaking capital to build more of it. The brake pedal proposal asks the world to trust that the same companies driving acceleration will honestly assess when to stop. That is a significant ask — and the track record, so far, does not inspire confidence.