How a court order to block an AI model differs from a kill switch

Two phrases keep appearing in California’s debate over frontier AI safety, and they describe different powers: a kill switch and a court-ordered block. Anthropic’s October 15 letter to Gov. Gavin Newsom, first reported by Politico on October 27 and by Reuters on October 28, asks for the first and cautions against relying on the second alone. Anthropic confirmed to Reuters that the letter is authentic. We have not read the letter itself, so the quotations below come from those two reports.

A kill switch versus a court block

A kill switch is a capability held by the company. The clearest statutory example is SB 1047, the 2024 bill Newsom vetoed. Its text would have required developers of large models to “implement the capability to promptly enact a full shutdown” before training began, and defined a full shutdown as ceasing the operation of the training run, the covered model and its derivatives under the developer’s control.

A court block is a decision by someone else. In Anthropic’s version, independent reviewers report risks to the state attorney general, and if the attorney general concludes a company’s models pose a “near-term risk of catastrophic harm,” the attorney general “should be able to seek a court order to suspend a model’s deployment, internal use, or further development,” Politico quoted the letter as saying. Reuters summarized it as officials being able to seek court orders against models that pose a “serious risk” to the public. SB 1047 did let the attorney general sue for injunctive relief. Section 22606 allowed a civil action “for a violation of this chapter” to recover, among other remedies, “injunctive or declaratory relief,” so it was an enforcement remedy for violations, not a standalone power to halt a model on a finding of risk.

The two are not opposites. Anthropic said “every frontier AI company should be able to rapidly shut down the models it serves or uses internally,” according to Politico. The company also said a law focused only on shutdown “could provide a false sense of reassurance” and that taking an entire model offline “only makes sense in extreme situations.” It proposed a “circuit breaker” approach that responds “at the level where they occur.” Reuters reported that the letter says requirements to build “kill switches” capable of shutting down entire models could provide a “false sense of reassurance” to government officials, and that Anthropic supports more targeted interventions, such as disabling a problematic AI agent or cutting off a specific customer’s access to a model. Reuters and Politico emphasize different parts of the letter: Reuters leads with its caution about kill-switch requirements, while Politico leads with its statement that every frontier company should be able to shut down models rapidly. Neither report says the letter opposes companies having that ability.

How independent review of risk reports would work

California has begun building the machinery. Newsom’s office said he signed SB 813, which sets up a framework for certifying independent verification organizations (IVOs) with “demonstrated independence from AI companies,” and AB 1405, a state registry of AI auditors.

Anthropic’s letter would put those reviewers to work. Politico reported that it supports third-party evaluators working inside frontier companies, that large developers should publish risk reports at least every six months, and that evaluators should have access “comparable to what a company’s own risk assessment staff have,” with limits for sensitive customer data. Anthropic proposed that companies pay into a state-administered fund rather than paying evaluators directly, and that the state rate evaluators on “the rigor of their reasoning and their track record of identifying risks.” The state could then require at least one highly rated evaluator or assign them by random selection or rotation.

Where California law stands

Newsom vetoed SB 1047 on September 29, 2024. His veto message said that by focusing only on the largest models the bill “could give the public a false sense of security.” He also wrote: “Safety protocols must be adopted.” A year later he signed SB 53, the Transparency in Frontier Artificial Intelligence Act. His office said it requires large frontier developers to publish a framework, creates a way to report critical safety incidents to the Office of Emergency Services, protects whistleblowers and sets a civil penalty enforceable by the attorney general.

The Future of Privacy Forum said SB 53 passed without many of SB 1047’s most controversial provisions, including its mandatory full-shutdown capability and annual third-party audits. It said critics had called the shutdown requirement technically infeasible and a barrier to open-source development.

What Newsom can do next

His September 18 executive order is not a mandate. It directs the Government Operations Agency to deliver recommendations by November 16 on the “technical feasibility and potential efficacy” of amendments, including embedded IVOs, independent verification of company filings, a kill switch whose efficacy is verified on an ongoing basis, and broader reporting of loss-of-control incidents. Changing the statute would take legislation. Politico reported the Legislature is expected to pursue AI regulation when it returns in about a month, and The Verge reported that Newsom had floated calling a special session.

Who else is weighing in

OpenAI sent Newsom its own letter on October 20. Politico said it supports “shutdown controls,” also described as circuit breakers, and embedded independent auditors, but did not go as far as authorizing a government block. It proposed requiring companies to monitor models under development for cyber and other risks.

In Washington, Reuters reported that a bipartisan group of senators debated a bill in September that would let the federal government block certain models, with companies able to challenge in federal court, and that House members proposed an “AI Kill Switch Act” in July. Reuters said lawmakers have not moved forward on the Senate proposal. The House bill remains introduced legislation.