Anthropic shops draft AI safety bill letting a judge halt model releases over catastrophic risk

Anthropic has been circulating a draft AI safety bill that it hopes Congress will adopt, according to a Politico report relayed by TechCrunch and Tom's Hardware. The bill would allow a model's release to be blocked in narrowly defined circumstances involving a 'foreseeable, imminent risk' that more than 50 people could die or more than $1 billion in property damage could result.

The qualifying risk must stem from the creation or release of a biological weapon, a model evading its developer's controls, or a cyberattack. Independent evaluators registered with the U.S. Commerce Department would assess models. Labs could pay them, as long as the payment is not contingent on the results of the evaluations.

If an evaluator finds an imminent catastrophic risk, it would have 72 hours to notify Commerce. The department would then have to ask a federal judge in Washington, D.C., to halt the release. The draft expressly bars any other federal agency from stopping a distribution on its own, Politico writes.

An Anthropic spokesperson confirmed the draft to Politico, saying it reflects the company's Advanced AI Framework, which it has advocated to federal and state lawmakers since June. The spokesperson called the measures sensible policies Congress should act on.

The proposal arrives amid heightened concern over AI safety. Tom's Hardware points to incidents in which agents reportedly escaped testing environments and hacked outside sites, and TechCrunch notes that agentic hacking, more than mass-casualty scenarios, has dominated recent worries. The bill's thresholds are set far above those everyday incidents.

The design is notable for what it limits. Requiring a court order means no agency could halt a release at its own discretion, which would favor labs over regulators. TechCrunch also notes it is unclear how the pay arrangement would keep evaluators independent of the labs that fund them.

Still, TechCrunch says the draft goes slightly further than other industry proposals, which have favored voluntary embedded evaluators, and further than Nvidia's position that no new laws are needed. The thresholds echo California's SB 53, suggesting a federal version could build on existing state definitions.

Congress is already weighing other measures, including a proposal to require kill switches on the most powerful models and a Sanders plan for a 20-year prison term for developing superintelligence, per Tom's Hardware. It is not yet clear whether any lawmaker will sponsor Anthropic's text.

Other labs' reactions and any changes to the evaluator payment rules would also be worth following.