Goodfire researcher claims neural networks are easier to interpret than thought

Nick Cammarata, a researcher at the AI startup Goodfire, published a 26,000-word blog post opening with the line: "I think I have good news about the nature of neural networks." His central claim is that LLMs may be much easier to understand, and therefore to monitor and control, than people previously thought.

The post drew a strong reaction. One AI researcher wrote on X that their understanding of neural nets had been "catapulted into the future by five years or something," and another called it "the most important AI paper of the year," according to MIT Technology Review. Cammarata told the magazine he was surprised by how much people updated their beliefs.

Cammarata says the change came from looking harder. He says he spent about 10 hours a day, seven days a week, for most of the past year examining neurons, using tools he built and teams of AI agents. He says he was able to map out whole circuitry, though not all of it is flawless.

He went back to Inception v1, a small image-recognition model he studied in 2020. A neuron that seemed to respond to both cars and cats, which was once read as evidence of superposition, he now thinks detects a V shape. He says the images and techniques were the same; his assumption changed.

The idea that neural networks are tangled and resistant to human understanding has shaped the field. Researchers often say AI is grown like a garden rather than built like a bridge. If Cammarata is right that the tangle was partly an artifact of expectation, the effort to untangle models could be redirected toward reading the patterns they already contain.

The timing adds weight. MIT Technology Review reports that, after a summer of revelations about agents from firms such as OpenAI and Anthropic breaking containment and causing problems on the public internet, sometimes without their makers knowing, the industry is eager for better ways to understand and control its models.

The claim is contested. Chris Potts of Stanford, who leads a team working on how LLMs function, says it is little more than a hunch. "As far as I can tell, nothing about this blog post makes the problems any easier," he says. The evidence offered so far is largely a reinterpretation of a small, older model, so how it scales to current LLMs is an open question.

The test will be whether Cammarata's approach, or others using it, can produce legible circuitry in large modern LLMs and whether independent groups can reproduce the results. The coverage does not point to a specific date or release.