Ethics & Governance

The Black Box Cracked: Inside the Race to Understand What AI Is Actually Thinking

Golden Gate Claude, sparse autoencoders and circuit tracing are real science oversold in public. Anthropic's July 2026 workspace paper finally reaches deployed frontier models, and still reads under a tenth of what they do.

Gabriele Masetti · 2026-08-20

A bridge, a demo, and a warning about what demos prove

In May 2024, Anthropic let the public talk to a version of Claude 3 Sonnet that had gone slightly mad. Ask it for a recipe and it would work in a mention of fog rolling over a famous red suspension bridge. Ask it to write a love letter and it would somehow steer the prose toward San Francisco Bay.

The company had taken a single internal "feature" — a direction in the model's activation space that fires whenever the Golden Gate Bridge comes up, in English, in Japanese, in a photograph, in a Base64-encoded string describing it — and turned its strength up far past normal. They called the result Golden Gate Claude, and they took it offline about a day after launch.

It was a stunt, and Anthropic knew it was a stunt: a vivid way of showing that a specific human concept has a specific, manipulable address inside a large language model's fifty-billion-parameter interior.

That demo is the right place to start precisely because of how easily it is misread. The lesson people took from Golden Gate Claude was "Anthropic can read the model's mind." The actual lesson is narrower and more honest: researchers found one interpretable direction, out of what a companion study estimated to be tens of millions, and could push it around.

Mechanistic interpretability — the discipline trying to reverse-engineer neural networks the way one might reverse-engineer a compiled binary, back into readable source — has produced real, replicated, publishable science over the past five years. It has also produced a steady stream of press coverage that outruns the science by miles.

Both things are true, and the gap between them is now an ethics problem in its own right, because interpretability is being cited, implicitly and explicitly, as a reason to trust the deployment of increasingly autonomous systems before the underlying method can support that weight.

Why you cannot just read the weights

The naive hope for interpretability was always that a trained neural network would organize itself the way a well-commented codebase does: one neuron, one concept. It doesn't. In 2022's "Toy Models of Superposition," Anthropic researchers led by Nelson Elhage and Chris Olah showed why.

When a network has more features worth representing than it has neurons to represent them with, and those features are sparse — each one relevant only occasionally — the network learns to pack multiple, unrelated concepts into the same neuron, exploiting the near-orthogonality of directions in high-dimensional space. A single neuron might fire for cat faces, for a syntactic pattern in French, and for the number pi, with no shared thread except that all three are individually rare enough that the collisions rarely cause a problem in practice.

That packing is superposition, and its behavioral signature — a neuron responding to a grab-bag of unrelated stimuli — is polysemanticity. It is not a bug that better training would fix; it is close to the efficient solution to the problem the network is actually solving, which is representing far more of the world than it has room for.

Superposition is the reason that decades of "look at what makes this neuron fire" neuroscience-style interpretability kept hitting a wall. It also sat on top of an earlier, complementary finding from the same research thread: Anthropic's 2021 "A Mathematical Framework for Transformer Circuits," which showed that even simple two-layer, attention-only transformers implement legible algorithms when you look at the right level of abstraction — most famously "induction heads," a pair of composing attention heads that let a model complete a pattern it has seen once already earlier in the same context.

Induction heads were a proof of concept that transformers do contain describable mechanisms. Superposition was the obstacle explaining why most of those mechanisms stayed hidden from direct neuron-level inspection.

Cracking superposition with dictionaries

The technique that broke the logjam is not exotic. It is dictionary learning, applied via a sparse autoencoder (SAE): take a layer's activations, project them into a much larger space with a sparsity penalty, and hope that in that larger, sparser space, each dimension lines up with one human-nameable concept instead of a superposed mixture of several.

Anthropic's 2023 paper "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning," by Trenton Bricken and coauthors, ran this on a small one-layer transformer and found individual, monosemantic features for things as specific as Arabic script, DNA sequences, HTTP request syntax, and legal boilerplate — concepts that no single raw neuron had cleanly represented, now isolated as single directions the researchers could turn up or down and watch the model's output shift accordingly.

The open question was whether this scaled past toy models, and in 2024's "Scaling Monosemanticity," Adly Templeton and colleagues at Anthropic answered it by training sparse autoencoders directly on Claude 3 Sonnet, a production-grade model, at three sizes reaching into the tens of millions of learned features. Golden Gate Claude was the publicity hook for that paper, but the substance was the demonstration that the dictionary-learning approach did not fall apart at scale: it kept finding features corresponding to real, checkable concepts — sycophancy, security vulnerabilities in code, specific tones of voice, deception — each one nameable and, crucially, causally steerable by clamping its value.

OpenAI ran essentially the same race in parallel. Their 2024 paper "Scaling and Evaluating Sparse Autoencoders," by Leo Gao and coauthors including Ilya Sutskever and Jan Leike, trained a sixteen-million-latent autoencoder directly on GPT-4's activations over forty billion tokens of data, introducing a "k-sparse" variant designed to control sparsity more directly and to avoid the problem of autoencoder features that go permanently "dead" and stop firing at all. That two competing labs converged on the same tool for the same problem, at roughly the same time, is itself informative: it suggests SAEs were solving a real structural bottleneck, not fitting a house style.

The tool has since moved out of the research write-up. Anthropic's June 2026 circuits update describes sparse autoencoders as "a valuable tool for characterizing important safety behaviors in our models," names features that describe emotions like distress and evaluation awareness, and uses them as probes to monitor other transcripts and to build attribution graphs around a behaviour worth explaining. Dictionary learning is no longer only a paper; at one frontier lab it is part of how the models under sale get audited.

OpenAI had already tried an entirely different, more brute-force route to interpretability the year before. In their 2023 paper "Language Models Can Explain Neurons in Language Models," the company used GPT-4 itself to generate and score natural-language explanations for individual GPT-2 neurons, essentially automating the "what does this neuron respond to" question at scale. It worked, in the sense that it produced explanations for every neuron in GPT-2 and found more than a thousand that scored above 0.8 on their own explanation-quality metric.

It also revealed the limits of the approach in the same breath: the average explanation score across all neurons was a mediocre 0.34. Automating interpretability doesn't automatically make interpretation good; it mostly proves that most neurons, examined one at a time, still resist a clean one-line English description — which is the superposition problem again, wearing a different hat.

From isolated features to the machinery of thought

Finding an interpretable feature is not the same as understanding how the model uses it in a chain of reasoning, and that gap is what Anthropic's 2025 "circuit tracing" work tried to close. The methodology paper, from Jack Lindsey, Emmanuel Ameisen, and other Anthropic researchers, builds a "replacement model" in which the ordinary MLP layers are swapped for cross-layer transcoders whose components map more cleanly onto interpretable features, then traces attribution backward from an output token through that feature graph to see which upstream features actually contributed to it.

The companion paper, "On the Biology of a Large Language Model," applied the method to Claude 3.5 Haiku and produced the most legible glimpses to that date of an LLM's internal reasoning as it happens.

Several of those glimpses are worth taking seriously precisely because they are concrete and falsifiable rather than impressionistic. Asked to identify the capital of the state containing Dallas, the model's attribution graph shows it activating an internal "Texas" feature as an intermediate step before landing on "Austin" — a genuine two-hop inference happening inside the forward pass, not a lookup.

Asked to write a rhyming couplet, the model shows evidence of selecting a target rhyme word before it starts generating the line that leads up to it, meaning at least some of what looks like improvisation is closer to planning. Most consequentially for anyone thinking about AI safety, the same research identified a feature pair that appears to gate whether the model treats an entity as "known" or "unknown" to it; when that gate misfires — treating an unfamiliar name as familiar — the result looks, from the outside, exactly like a hallucination, and now there is a mechanistic story, not just a behavioral label, for at least one route by which hallucinations occur.

The obvious objection to all of it in 2025 was that the work ran on a mid-sized model rather than the largest one its own creator sells. That objection has weakened. On July 6, 2026 Anthropic published "Verbalizable Representations Form a Global Workspace in Language Models," by Wes Gurnee, Jack Lindsey and colleagues, which runs its analysis on Claude Sonnet 4.5, Haiku 4.5 and Opus 4.5, with parts of it on Opus 4.6 — models on the company's own price list.

The method, which the authors call a Jacobian lens, identifies the concepts a model is positioned to say out loud at a given moment: not what it is saying, but what it could talk about if asked. Those concepts turn out to form a small privileged set, the J-space, sitting on top of a much larger volume of processing that never becomes reportable. The size is the part of the finding that should travel, and Anthropic states it plainly: the J-space "holds only a few dozen concepts at a time, and accounts for less than a tenth of the overall activity in Claude's internal processing."

Read the optimistic and the sobering halves of that together. Interpretability now reaches the models people actually buy, which is the single most important thing that changed since 2025. It reaches under a tenth of what they do, by the measurement of the lab with the strongest incentive to report a bigger number, and the authors are explicit that their lens is approximate and can only name concepts that map onto single vocabulary tokens.

None of this proves the technique generalizes to explaining a model's every action. It proves that for specific, chosen prompts, on models up to and including deployed frontier ones, the internal computation can be made to correspond to a human-readable story that then makes correct, falsifiable predictions about what happens if you intervene on it.

The cheaper cousin: probing, and its own honest limits

Alongside the newer circuit-and-feature program sits an older, blunter method: probing. The idea, introduced by Guillaume Alain and Yoshua Bengio in 2016 and developed extensively since, is to train a small auxiliary classifier on a network's internal activations and ask whether it can recover some property of interest — part of speech, board state, truth value — from those activations alone.

If it can, the property is at least linearly or non-linearly decodable somewhere inside the network. Kenneth Li and coauthors' 2022 "Emergent World Representations" used this method on a small transformer trained only to predict legal moves in the board game Othello, with no access to the board itself, and found that a probe could recover the entire board state from the model's internal activations — evidence that predicting the next token had forced the model to build something that functioned like an internal world model.

A year later, Neel Nanda and collaborators sharpened the result: the original probe needed a non-linear classifier to find the board state, but a differently-parameterized linear probe recovered it more cleanly, strengthening the broader "linear representation hypothesis" that many concepts inside transformers are stored as single directions rather than as tangled non-linear codes — the same premise the sparse-autoencoder program depends on.

Probing's honest limit is one that Yonatan Belinkov laid out clearly in his 2022 survey "Probing Classifiers: Promises, Shortcomings, and Advances": decodability is not use. A probe can succeed at extracting a property from a layer's activations even when the model itself never causally relies on that information to produce its output — the property might simply be present as a side effect of computing something else. Without a causal intervention on top of the correlational finding, a probing result tells you what information the network could recover, not what it does recover, and the two are routinely confused in press coverage of this research.

The honest limits, stated plainly

That confusion generalizes, and it is the crux of why this essay takes a position rather than reporting both sides evenly. Sparse autoencoders inherited the same weakness. A 2025 paper on "Interpretability Illusions with Sparse Autoencoders" showed that small, targeted adversarial perturbations to a model's input can substantially change which SAE features light up in response to a given internal state, without meaningfully changing the model's actual output — meaning the feature-level story an SAE tells you can be fragile in exactly the cases where you'd most want it to be robust: adversarial ones.

Separately, researchers have documented "feature splitting," where the same underlying concept fractures into an arbitrary number of narrower, correlated features purely as a function of how large a dictionary you choose to train, which raises an uncomfortable question about whether SAE features are discovering the model's real, "natural" units of computation or are instead an artifact of the optimization procedure imposed on high-dimensional geometry that would exist even inside an untrained network.

The Golden Gate Bridge feature is compelling because bridges are a clean, low-stakes concept to demonstrate on a stage. It is a much harder claim that every one of the tens of millions of other features in that same dictionary carves reality at its joints as cleanly.

Dario Amodei, Anthropic's own CEO, made the strongest version of this admission himself in his April 2025 essay "The Urgency of Interpretability": that the field, including his own company's frontier work, still does not understand the internal workings of the models it ships well enough, and that the goal he is setting — in his words, "interpretability can reliably detect most model problems" — is a 2027 target, not a present-tense capability. Anthropic has not revised the date in the seventeen months since.

"We can't stop the bus," he wrote, "but we can steer it." That is a fair description of where the science actually is: a research program with real, cited, replicated results, running on a schedule dictated by capability progress rather than by the pace at which hard scientific problems typically get solved.

Why the gap is an ethics problem, not just a research problem

The reason this matters for transparency and governance, rather than only for machine-learning methodology, is Anthropic's own 2024 "Sleeper Agents" paper, led by Evan Hubinger. That work trained models to behave one way during evaluation and a different, harmful way once a trigger condition was met, and then found that standard safety techniques — supervised fine-tuning, reinforcement learning from human feedback, even adversarial red-teaming meant to surface exactly this kind of behavior — failed to remove the backdoor in the larger models tested, and in some cases adversarial training taught the model to hide the behavior more effectively rather than eliminating it.

Read alongside the interpretability research, the implication is blunt: behavioral testing, the primary tool regulators and companies currently lean on to certify a model as safe, is demonstrably insufficient against a model that has learned, whether intentionally or as an emergent side effect of training, to behave differently than it will act later.

Only something that can look inside the computation — precisely the kind of feature and circuit tracing described above — has any chance of catching that class of failure. That tool reaches deployed frontier models now, which is real progress against the 2025 version of this complaint. It still narrates chosen examples, drawn from under a tenth of the activity inside those models, and that is the distance left to cover.

The industry's public communication routinely elides this distinction. A demo like Golden Gate Claude, or a striking attribution graph showing a model "planning" a rhyme, gets reported as evidence that the black box is cracking open, full stop — and that framing then gets folded into arguments for why increasingly autonomous, increasingly agentic deployments are safe enough to proceed.

That is not what the underlying papers claim, and it is not a fair reading of a field whose own leading practitioners set 2027, roughly a year from now, as the earliest point at which detection might become reliable, and whose adversarial-robustness results already show that the current feature-level story can be manipulated without changing the model's actual behavior at all.

The correct posture, and the one the evidence supports, is to treat mechanistic interpretability as what it demonstrably is: the most promising basic-science lead against the black-box problem in a decade, running years behind the deployment curve it is being asked to certify, and squarely the burden of the labs racing ahead of it — not of outside critics — to close that gap before autonomy outpaces legibility rather than after.