Federal AI Safety Institute Signs National Security Testing Agreements with Google, Microsoft, and xAI

On May 5, 2026, the Center for AI Standards and Innovation — the federal body housed within the National Institute of Standards and Technology, formerly known as the U.S. AI Safety Institute — announced new agreements with Google DeepMind, Microsoft, and xAI [1]. The deals grant CAISI access to frontier AI models before public release, including versions with safety guardrails reduced or removed, so government researchers can probe them for national security risks [1].

The three agreements join existing partnerships with OpenAI and Anthropic, first established in 2024 and since renegotiated to align with President Trump's AI Action Plan [2]. Five companies now account for the vast majority of frontier AI development worldwide, and all five have agreed to let a single government office test their systems before deployment [3].

CAISI Director Chris Fall framed the effort as a scientific imperative: "Independent, rigorous measurement science is essential to understanding frontier AI and its national security implications. These expanded industry collaborations help us scale our work in the public interest" [1].

The question is whether the institute has the resources, authority, and independence to make that claim meaningful.

What the Agreements Actually Require

The agreements are voluntary memoranda of understanding, not legally binding contracts [4]. They establish a framework for pre-deployment evaluation and post-deployment assessment of frontier models, but they carry no statutory basis [3]. Companies can withdraw at any time, and the government lacks authority to block a model's release based on test results [3].

Under the terms, developers provide CAISI with models that have "reduced or removed safeguards" to enable thorough evaluation of national security-related capabilities and risks [1]. The agreements support testing in classified environments, meaning some evaluations occur within facilities where results can be restricted from public disclosure [1].

The TRAINS Taskforce — a group of interagency experts — participates in evaluations. Its members include representatives from the Department of Defense (including the NSA and the Chief Digital and Artificial Intelligence Office), the Department of Energy and its national laboratories, the Department of Homeland Security, CISA, and the National Institutes of Health [5]. These agencies lend subject matter expertise in areas like cybersecurity, biosecurity, and nuclear risk, and collaborate on developing new evaluation benchmarks and conducting joint red-teaming exercises [5].

What the press release does not specify — and what no public document clarifies — is what happens when a model raises serious concerns. There is no disclosed formal process to delay or block deployment. The agreements "support information-sharing, driving voluntary product improvements" [1], language that suggests CAISI's role is advisory. If a company discovers its model has dangerous capabilities, it could, legally, decline to submit it for evaluation and release it anyway [3].

A Budget That Doesn't Match the Mission

CAISI operates on an annual budget of approximately $10 million [6]. The UK's AI Security Institute (formerly the UK AI Safety Institute), established at the same time in November 2023, received £100 million ($127 million) in public funding — roughly 12 times the American body's resources [7][6].

Source: FAS, NIST, UK Gov

Data as of May 5, 2026CSV

The European Union's AI Office, which has regulatory powers under the AI Act including the ability to request information from model providers and apply sanctions, operates with a broader mandate and enforcement toolkit that no first-wave safety institute possesses [7]. Japan's AI Safety Institute received approximately $28 million. Canada announced its own AI Safety Institute with CAD $50 million in funding [7].

A proposal from the Federation of American Scientists for an expanded "CAISI+" estimated that a credible national AI security evaluation body would require an annual operating budget of $67-155 million, an initial setup cost of $155-275 million, 80-150 technical staff plus 30-60 support personnel, and 128-512 state-of-the-art GPUs housed in facilities with "nation-state-level security protections" [6].

Current CAISI funding is described as "precarious," and its host institution NIST's offices are reportedly "crumbling" [6].

The Staffing Crisis

The resource gap became acute in February 2025, when the Trump administration announced plans to dismiss approximately 497 probationary employees at NIST — a 20 percent workforce reduction [8][9]. The cuts targeted recent hires, many of whom staffed the AI Safety Institute and the CHIPS for America program [8]. Reports indicated headcount reductions of more than 50 percent in offices key to AI work [10].

This came shortly after President Trump repealed the Biden-era AI executive order on his first day back in office, and the institute's director departed [8]. The institute was then rebranded in June 2025, shifting from the "U.S. AI Safety Institute" to the "Center for AI Standards and Innovation" — a name change that Commerce Secretary Howard Lutnick described as reflecting a "pro-innovation, pro-science" orientation [11].

CAISI currently has approximately 200 government evaluators working on model assessments, according to reporting on the program's scope [3]. The asymmetry is stark: those 200 evaluators face companies employing tens of thousands of AI researchers. As one analysis put it, "the companies will always know more about their models than the evaluators do" [3].

Why xAI — and the Question of Who Benefits

The inclusion of xAI alongside Google and Microsoft raised eyebrows. Elon Musk's AI company is significantly smaller and newer than the two dominant cloud providers, and its flagship model Grok has drawn criticism for producing racist, antisemitic, and conspiratorial content [12]. Senator Elizabeth Warren sent a letter to the Pentagon in September 2025 raising concerns about the integration of Grok into government systems [12].

Yet xAI's inclusion also reflects an effort toward "comprehensive frontier coverage" of companies developing the most capable AI systems [3]. From xAI's perspective, the agreement carries clear advantages: participation in a federal testing program lends government legitimacy to its models, a valuable signal for a company competing against better-established rivals for enterprise and government contracts.

This dynamic fuels the strongest version of the critique that these agreements primarily serve commercial interests. The argument runs as follows: by participating in a voluntary federal testing program with no binding consequences, the signing companies gain a de facto government endorsement — "tested by CAISI" — without actually subjecting themselves to enforceable regulation. Companies with the lobbying reach and legal infrastructure to negotiate memoranda of understanding with federal agencies benefit from the arrangement; smaller AI developers, open-source projects, and international competitors do not.

The Trump administration's USAi program, which allows federal employees to experiment with models from OpenAI, Anthropic, Google, and Meta under $1 contracts, reinforces this pattern — entrenching a small number of large companies in government procurement pipelines [12].

Defenders of the program counter that any testing is better than none, that voluntary agreements with the largest frontier developers cover the models most likely to pose national security risks, and that the alternative — no pre-deployment evaluation at all — would leave the government entirely blind to emerging capabilities.

The Mythos Wake-Up Call

The urgency behind these agreements came into sharp focus in April 2026, when Anthropic revealed Claude Mythos Preview — a model that autonomously discovered and exploited zero-day vulnerabilities across every major operating system and browser [13]. The model identified thousands of high-severity bugs, some undetected for decades, including a 17-year-old remote code execution flaw in FreeBSD [13].

Anthropic did not explicitly train Mythos to have these capabilities. They emerged as a consequence of general improvements in code reasoning and autonomous operation [13]. Non-experts using Mythos could request remote code execution vulnerabilities overnight and find a complete, working exploit by morning [13].

Anthropic chose not to release Mythos publicly, instead forming Project Glasswing — a coalition including AWS, Apple, Microsoft, Google, CrowdStrike, and Palo Alto Networks to coordinate responsible disclosure [14]. But the episode demonstrated precisely the kind of capability that pre-deployment evaluation is supposed to catch: an AI system whose national security implications were not anticipated by its creators.

The Mythos case also complicated Anthropic's relationship with the testing program. Reporting from multiple outlets painted a conflicting picture: some sources indicated Anthropic renegotiated its existing CAISI agreement to align with the Trump AI Action Plan [2], while others reported Anthropic was notably absent from the May 5 announcement following a dispute with the administration over safeguards against mass surveillance and autonomous weapons [15].

The Classified Testing Problem

The agreements specify that testing can occur in classified environments [1]. This means some evaluation results may be restricted from public disclosure, shared with intelligence agencies through the TRAINS Taskforce's interagency channels, or withheld entirely under national security classification rules.

This creates a transparency tension at the core of the program. The public has no way to verify what CAISI finds, what it reports to companies, or whether companies act on those findings. Trade secret protections further limit disclosure — the agreements must balance "protecting the AI companies' proprietary technology" against the public interest in knowing whether a model poses risks [4].

The classified testing capability does, however, address a genuine need. Evaluating whether an AI model can assist with weapons development, help conduct cyberattacks on critical infrastructure, or generate actionable biological threat information requires controlled environments where sensitive findings do not themselves become security risks.

How the U.S. Compares Internationally

The United States is not alone in establishing pre-deployment AI testing, but it lags behind peer nations in resourcing and authority.

The UK's AI Security Institute, with its £100 million budget, has conducted evaluations of frontier models and negotiated access agreements with major developers [7]. The EU AI Office has regulatory powers that no first-wave safety institute possesses, including the ability to impose sanctions under the AI Act [7]. Singapore, Japan, and Canada have each established their own safety institutes, contributing to an international network announced at the 2023 AI Safety Summit [7].

Source: OpenAlex

Data as of Jan 1, 2026CSV

Academic research on AI safety evaluation has expanded rapidly, with over 154,000 papers published in 2025 alone according to OpenAlex data. But the gap between published research and operational government capacity to evaluate models remains wide.

The risk of international fragmentation is real. If U.S. oversight appears inadequate, foreign governments may impose their own separate — and potentially conflicting — requirements on the same models [3]. The EU's demand for access to Anthropic's Mythos for cyber defense purposes, citing concerns that "the most consequential cybersecurity tool in existence" remained under exclusive American control, previewed this dynamic [3].

What This Adds Up To

The CAISI agreements represent the most extensive federal engagement with frontier AI testing to date. Five companies, more than 40 completed evaluations, classified testing environments, and an interagency taskforce spanning defense, energy, homeland security, and health agencies [1][5].

But the program's structural limitations are significant. It is voluntary, underfunded relative to international peers, understaffed relative to the industry it oversees, and lacks any mechanism to delay or block a model's deployment. The Anthropic co-founder Jack Clark's assessment from an earlier phase of the program remains apt: "pre-deployment testing is a nice idea, but very difficult to implement" [4].

One analysis described the program this way: "What the programme provides is not comprehensive oversight. It is a window, narrow and dependent on goodwill, into what the most powerful AI systems can do before the rest of the world finds out" [3].

Whether that window is wide enough depends on what the government does with what it sees — and whether it ever gains the authority, funding, and independence to act on it.