Reading Hidden Representations Without Overclaiming

Stronger claims require stronger methods than merely plotting activations. Description summarises what activations look like across examples. Decodability trains a probe to predict an attribute from frozen activations. Intervention changes an activation or a pathway and measures what happens to the output.

The third level is not an abstraction waiting for someone to attempt it. In 2022 an intervention located where GPT stores a fact, and then edited it. “We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions.” That is Meng and colleagues, opening the abstract of the paper that introduced causal tracing. The tracing localised factual recall to middle-layer feed-forward modules at the subject tokens. The authors then changed those weights with Rank-One Model Editing to alter specific facts.

Notice what the levels cost and what they return. The same activations could have been plotted, and could have been probed, and neither would have named a weight anyone could change. The move from a heat map to a patched activation is not a rhetorical upgrade of the same finding. It is a different experiment with a different output.

Neural information can be distributed across many units, while one unit can participate in several features. A coordinate’s apparent meaning may also change across layers, contexts, seeds, or fine-tuning stages.

Whether a coordinate carries readable meaning is a property of the chosen basis, not of the function computed. That has been measured, not argued. The Network Dissection team took the 256 units of AlexNet conv5, trained on Places205, and spun them with a random orthogonal rotation Q drawn from SO(256). The network computed exactly what it had computed before: “Each rotated representation has exactly the same discriminative power as the original layer”. What changed was how much of it could be read — “we find that the number of unique detectors in Qf(x) is 80% fewer than the number of unique detectors in f(x)”. Same function, same accuracy, 80% fewer readable units. The paper’s introduction states the conclusion: “emergent interpretability is an axis-aligned property of a representation that can be destroyed by rotation without affecting discriminative power” — Bau and colleagues, 2017.

Interpretation should therefore target a defined behavior and dataset. Searching for a permanent dictionary of neuron labels is often the wrong objective. The rotation experiment shows that such a dictionary can be deleted without touching anything the network does.

A linear probe asks whether an attribute can be decoded by a simple linear boundary from frozen activations, and strong probe performance means the information is linearly accessible to that probe.

It does not show that the original network relies on the attribute, that the information is causal, or that the probe cannot exploit a shortcut correlated with the label.

The size of that gap has been measured. An experiment on BERT removed properties from the representation by causal intervention, then measured what the removal did to word prediction. Rank the properties by how well a probe decodes them. Rank the same properties by how much the model’s predictions actually depend on them. The two orders have no relationship to each other: “the probe accuracy does not correlate with task importance as measured by our method (Spearman correlation of 8.5, with a p-value of 0.871)”. That is Elazar and colleagues, in the 2021 amnesic probing paper. Their own summary is blunt. “Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.” A probe score is not a weak form of a causal claim. On this evidence it is not a form of one at all.

An independent group reached the same failure from theory. Kumar and colleagues proved in 2022 that “Even under the most favorable conditions for learning a probing classifier when a concept's relevant features in representation space alone can provide 100% accuracy, we prove that a probing classifier is likely to use non-concept features”. The probe takes a shortcut even when it does not need one.

There is a cheap test for the shortcut, and it dates to 2019. Hewitt and Liang pair each linguistic probing task with a control task. Control tasks are ones “which associate word types with random outputs” and can therefore “only be learned by the probe itself”. A trustworthy probe should be selective, “achieving high linguistic task accuracy and low control task accuracy”. They tested part-of-speech tagging and dependency edge prediction on ELMo representations. The verdict: “popular probes on ELMo representations are not selective”. The control also reorders the layers, since although “probes on the first layer of ELMo yield slightly better part-of-speech tagging accuracy than the second, probes on the second layer are substantially more selective”. A probe accuracy reported without a control is a measurement of the probe.

Choose a method from the claim, not from the most attractive visualization. Activation statistics reveal scale, sparsity and saturation. Feature visualization searches for inputs that increase selected activations. Representation similarity compares layers or checkpoints at a population level. Probes measure accessibility. Interventions test whether changing an internal value alters a target behavior. Three of those five have a published, measured failure mode.

For saliency maps the failure is not hypothetical. Adebayo and colleagues ran sanity checks on the popular methods in 2018 and reported that “reliance, solely, on visual assessment can be misleading”. Worse, they found that “some existing saliency methods are independent both of the model and of the data generating process”. Such a picture reads as an explanation. It is meanwhile insensitive to the very model it claims to explain. Their conclusion is operational. Methods that fail those checks “are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model”. They compare such output to edge detection, “a technique that requires neither training data nor model”.

Feature visualization has its own measured limit, and it predates the current interest by a decade. Nguyen and colleagues used evolutionary algorithms and gradient ascent to synthesise inputs that maximise class outputs. What came out, in their words: “Here we show a related result: it is easy to produce images that are completely unrecognizable to humans, but that state-of-the-art DNNs believe to be recognizable objects with 99.99% confidence (e.g. labeling with certainty that white noise static is a lion).” That was 2015. The optimiser did exactly what it was asked. What it returned is a picture of a unit’s sensitivity, at 99.99% confidence, and not a picture of the concept a reader will name when shown it.

At the other end of the table, an intervention-based method returns something with a denominator. Conmy and colleagues automated the activation-patching step of circuit finding in 2023 and validated it against circuits found by hand. Their ACDC rediscovered 5/5 component types in the GPT-2 Small Greater-Than circuit. And: “ACDC selected 68 of the 32,000 edges in GPT-2 Small, all of which were manually found by previous work.” 68 edges out of 32,000, each one checkable against prior manual work. A heat map has no such count and offers nothing to check it against.

The wording of an interpretation should match the method used. Each claim below names the evidence it needs. Where the field has already run that evidence, the result is attached.

Build claims from weaker to stronger evidence. Define one behavior, slice or failure to explain. Collect descriptive evidence across positive, negative and counterexample inputs. Test decodability with simple probes, under held-out controls and against baseline features. Intervene by ablating, replacing or patching values, watching for distribution shift. Then state the claim narrowly: what the method supports, and which alternatives remain.

The fourth step is the one that needs a warning attached to it. Even a clean intervention result bounds its own claim rather than generalising. Hase and colleagues tested in 2023 whether the localisation produced by causal tracing predicts where a fact can be edited, and found that it does not. Which layer was edited predicted editing success far better than the tracing result did. “Specifically, we show that localization conclusions from representation denoising (also known as Causal Tracing) do not provide any insight into which model MLP layer would be best to edit in order to override an existing stored fact with a new one.”

That result comes from the same community that produced causal tracing, and it lands on the same lesson the probe results do one level down. An intervention answers the question it actually ran. The fifth step exists because the next question needs its own experiment.