METR report finds Anthropic's AI progress has doubled in speed, driven mostly by AI

The AI safety organization METR published a report on Thursday from an 11-week investigation inside Anthropic. It found that the company's models are improving roughly twice as quickly as they were a year ago, with little sign of slowing, according to The Verge's account. Benchmark scores suggest Anthropic made about two years of historical progress over the last year.

METR said it is confident the acceleration was not simply due to more training compute. It said the majority of the acceleration as of October is due to AI, and that AI's contribution is growing. Informal Slack polls and interviews with staff pointed to large productivity gains: employees using Anthropic's most capable internal model reported being about 12 times as productive, around double what they reported a few months earlier using Mythos 5.

For now, much of the gain comes from AI assisting human researchers rather than working independently. METR warned there are signs that fully autonomous contributions will play a larger role soon, and said it found no clear diminishing returns or bottlenecks. Much of the progress is attributed to unreleased models: METR says Anthropic finished training three in the past six months, codenamed Model 2, Model 3 and Model 4.

The report is some of the clearest evidence yet of AI speeding up its own development. It lands as Anthropic prepares to go public while its CEO, Dario Amodei, has publicly called for labs to slow down. Ajeya Cotra, writing on Planned Obsolescence about the report, argues that Anthropic's own Responsible Scaling Policy threshold, capabilities progress going twice as fast as the fastest pre-AI sustained rate, appears already to be crossed. She calls the report the most important update on AI risk in a couple of years.

Cotra's own modeling is uncertain, but it suggests that doubling overall progress would require a labor speedup of several times, perhaps around 10x today. That is close to the 12x staff self-reports. She says this points toward shorter timelines than most external researchers have projected.

The access limits matter too. Anthropic gave METR more information than first scoped, but not unrestricted or employee-like access, and it could redact the public report. METR said the public must rely on its judgment more than is ideal. A skeptical LessWrong post, a user's shortform comment, doubts that autonomous AI research is ingenious, citing poor model judgment in side projects, though it concedes fast-feedback tasks are an exception.

Key questions are whether Anthropic formally acknowledges crossing its RSP threshold, and what it does in response. Its S-1 said it may imminently meet the threshold.

Also watch whether more of the redacted evidence becomes public, and whether METR or others can verify the speedup independently.