AI Neural Networks May Be More Interpretable Than Researchers Expected

A different explanation for AI’s “black boxes”

Some neural networks may be easier to interpret than researchers have assumed. That’s the central claim in a lengthy blog post by Nick Cammarata, a researcher at AI startup Goodfire who helped shape early work on understanding neural networks at OpenAI. His analysis suggests that neurons that appear to respond to unrelated ideas may often be tracking a shared feature or a cluster of related concepts.

The claim challenges a familiar account of why neural networks are difficult to understand. It also raises a practical question for AI research: if individual neurons have more coherent roles than they seem to, could researchers make faster progress in working out what a model has learned? Cammarata’s findings offer a reason to investigate that possibility, but they don’t show that large AI systems are now fully understood or easy to control.

Mechanistic interpretability is the research effort to reverse engineer neural networks by examining their components and how those components work together. Researchers hope that a clearer account of a model’s internal operations could help them assess its behavior and guide it more reliably. So far, progress has been difficult, in part because it can be challenging to assign a clear meaning to any one neuron.

Revisiting a famous neuron

The debate traces back to a 2020 study of InceptionV1, a Google model trained to recognize images. Cammarata and colleagues found a neuron that responded to both cat faces and the fronts of cars. That result became a vivid example of a proposed explanation called superposition: when a network has fewer neurons than the concepts it needs to represent, some neurons may carry information about multiple concepts.

Under that explanation, a neuron’s responses can look scattered. If it represents ideas that don’t obviously relate to each other, observing the neuron alone may reveal little about what the network is doing. The cat-and-car example appeared to fit that picture.

Cammarata’s newer interpretation starts with a visual detail the two images share: a prominent V shape. He reports that the same neuron also responded to a pool table corner and the pointed nose of a Japanese bullet train. Those examples led him to ask whether the neuron might be responding to a shape or visual pattern rather than combining wholly unrelated concepts.

He then used a set of specialized tools and AI agents to search for similar patterns across InceptionV1. His conclusion was that neurons that seem to represent different things often have a connection that becomes clearer on closer inspection. In his account, a neuron may first respond to one feature and then take on neighboring concepts as the network learns. He calls this process “recursive generalization.”

Early results in language models

Cammarata says he has applied the same approach to newer language models. In preliminary work on Qwen3-32B, he reports that roughly 90 percent of neurons tracked one main concept, sometimes with a few related offshoots. He says he saw similar patterns in Kimi K3, a model described in the source as having a trillion parameters. A technical paper on that analysis is expected to provide more detail.

These results could matter because they suggest researchers may be able to form useful hypotheses about a model by examining individual neurons. But a neuron’s apparent role is only one piece of the picture. A model’s behavior also depends on how neurons connect and interact in larger circuits. Understanding a collection of parts doesn’t automatically explain how the whole system produces a particular response.

That distinction is one reason the blog post has prompted both interest and caution. Cammarata’s work is presented as a working hypothesis, rather than a rigorous empirical study. Chris Potts of Stanford University acknowledged the scale of the examples while questioning whether they amount to sufficient evaluation. As he put it, “even a large number here is not a substitute for rigorous evaluation.”

What the evidence can support

Jacob Hilton of the Alignment Research Center has raised a related concern: explaining individual neurons may not be enough to explain a network. Researchers also need to understand the circuits formed by connections among them. That wider view could reveal whether a neuron’s apparent role holds across different contexts, or whether its contribution changes depending on what other parts of the model are active.

Cammarata has also cautioned against reading his results as a solution to the interpretability problem. He told MIT Technology Review that “it’s not everything solved,” and pointed to open questions about how much model behavior depends on combinations of neurons rather than individual ones.

My read is that the work gives researchers a new way to examine an old puzzle, not a settled answer about how AI systems think. If the reported patterns hold up under rigorous testing, they could make some parts of neural networks easier to describe and study. The forthcoming technical paper, and further work on how neurons form circuits, should help clarify how broadly that optimism applies.

For now, the useful takeaway is limited but meaningful: a neuron that seems to respond to unrelated concepts may be tracking a feature those concepts share. That possibility gives interpretability researchers a concrete lead, while leaving the larger challenge of explaining whole models open.

Read the blog post and the related discussion among researchers. Cammarata’s original example comes from work on neural networks and visual recognition; further critiques appeared in MIT Technology Review and in posts by Jacob Hilton.