Anthropic's letter to Newsom says courts should be able to block AI models that a review finds pose a serious risk

Anthropic told California Gov. Gavin Newsom that government officials should be able to ask a court to block an AI model that a review finds poses a serious risk, according to an Oct. 15 letter, Reuters reported Wednesday. Politico first reported the letter’s contents Tuesday evening. “When a review finds a model poses a serious risk, the government should be able to ask a court to block it,” the letter said, as quoted by Reuters. An Anthropic spokesperson confirmed to Reuters that the letter is authentic. Newsom’s office did not immediately respond to Reuters.

The letter responds to ideas in an executive order Newsom issued in September, including independent safety monitors inside AI companies and a “kill switch,” according to Politico. It says independent verification organizations (IVOs) should be able to report risks they identify to the state attorney general. If the attorney general concludes that a company’s models pose a “near-term risk of catastrophic harm,” the letter says, the attorney general “should be able to seek a court order to suspend a model’s deployment, internal use, or further development.” It adds that requiring both an IVO’s assessment and a court’s ruling “protects against this power being misused, and a court can still act quickly in an emergency.” The letter also says Anthropic supports requiring IVOs to review frontier AI companies’ risk reports and sees embedding IVOs inside those companies, with employee-like access, as a stronger form of that review. Politico reported that the letter says large AI developers should publish risk reports at least every six months.

On kill switches, the letter says Anthropic shares the loss-of-control concerns behind the executive order and that “every frontier AI company should be able to rapidly shut down the models it serves or uses internally.” It adds that any regime letting government block a model, which Anthropic says it generally supports, “only works if companies can actually take that model offline quickly when ordered to.” But it says a law focused on shutdown alone “could provide a false sense of reassurance,” and it proposes a “circuit breaker” approach that cuts off a problem “at the level where it occurred.” Politico quoted the letter saying that taking an entire model offline “only makes sense in extreme situations.” Reuters described the stance differently, reporting that Anthropic opposes requirements that developers build “kill switches” capable of shutting down entire models and prefers targeted steps, such as disabling a problematic AI agent or cutting off one customer’s access.

Reuters said Anthropic submitted the recommendations after Newsom convened a panel of AI experts in September. Politico said OpenAI sent Newsom a similar letter last week and that the California Legislature is expected to pursue AI regulation when it returns in about a month. Reuters reported that a bipartisan group of senators debated a bill in September that would let the federal government block certain AI models, and that House lawmakers proposed an “AI Kill Switch Act” in July.

Anthropic made its S-1 public Monday for a planned initial public offering. The Justice Department also filed a notice of appeal Monday of an August ruling that blocked a federal agency ban on the company.

Anthropic’s letter to Newsom says courts should be able to block AI models that a review finds pose a serious risk

Sources