Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise

AI summary

Overview

Research area: Mechanistic interpretability of transformer language models — specifically the comparison between sparse dictionary methods (transcoders) and raw MLP neurons as units of analysis.

Technical level: Intermediate. The paper assumes familiarity with MLP blocks, sparse autoencoders/transcoders, and attribution graphs, but its core argument can be followed by anyone who knows what a neuron's input and output weights are.

Scope (one sentence): An empirical audit of 13 open-source transcoders across Gemma, Qwen, and Llama models, showing that a large share of the transcoder features actually used are near-copies of individual MLP neurons, and that the remaining features are unstable, unfaithful, and barely move from random initialization.

What This Paper Is About

MLP neurons are widely assumed to be polysemantic — each one fires on many unrelated concepts — which is why the field replaces them with learned sparse dictionaries such as transcoders for circuit analysis. Recent work (Arora et al., 2026; Ye et al., 2025) found that raw, thresholded neurons can already yield sparse and interpretable circuits, raising the question of what transcoders actually add. This paper answers that question by measuring, feature by feature, how much of a transcoder is simply rediscovering the MLP's own neurons.

Key Contributions

1. A definition and measurement of "neuron features." The authors define a transcoder feature as a neuron feature when both its encoder and decoder exceed a cosine similarity threshold to the same MLP neuron's effective encoder (W_up ± W_gate) and decoder (W_down). Thresholds separating the cluster from the bulk range from 0.6 to 0.9 across transcoders.

2. Three independent lines of evidence that neuron-aligned structure is the only consistent structure transcoders learn. Linearizations (Jacobians) of the transcoder match the original MLP only on neuron features; only neuron features reoccur across random seeds; and non-neuron feature weights remain close to their random initialization.

3. A neuron-based replacement model that beats transcoders on causal importance. After a small learned linear correction, top-k-pruned sign-split neurons achieve lower KL divergence than transcoders, though their FVU (fraction of variance unexplained) remains substantially higher.

4. A training-time intervention ("neuron initialization") that initializes encoder rows at random sign-split neuron directions with 10% Gaussian noise and a zero decoder, which raises encoder alignment above 0.5 cosine from 11% to 92% and decoder alignment from 12% to 93% at 256k width on Gemma 3 4B L22, at little cost to reconstruction.

Main Findings

• Neuron features dominate activations in late layers. They are a small fraction of the dictionary but account for over half of all feature activations in late layers across 13 open-source transcoders. Reported dictionary fractions rise from 5.9% to 8.9% to 15.6% to 37.1% from 1B to 27B, with the share in activations going from 57.9% to 82.8%. A separate footnote reports 3.6% → 7.2% → 10.7% → 17.7% from 1B to 27B, attributed to the rising fraction of neurons in the model at constant transcoder width.

• They are far more frequent than other features. Neuron features fire more than 10× more often on average than non-neuron features (Figure 6).

• They appear mid-model, not early. Prevalence climbs from near zero in early layers to over 50% through the middle of the model on Gemma 3 4B and Qwen 3 4B; early-layer share is at most 33% across transcoders.

• They dominate attribution-graph circuits. In the late third of layers, neuron features account for at least 40% of nodes in attribution graphs for every transcoder tested except one (Llama 3.2 1B mntss, around 38%). Pruning by influence keeps their share stable (56–61% on Gemma 3 4B from L0=10 to L0=100 features per token).

• They behave like their neurons. Thresholding a neuron's per-token post-activation predicts whether its feature fires with a median AUC of at least 0.977; one example reaches 0.9997 AUC (87.8th percentile). Neuron features fire roughly 3× less often than their matched neurons (median firing-rate ratio 0.34 on Gemma 3 4B L22). The median rank of the matched neuron is as low as 12 for the top-firing feature and climbs to about 250 for the 32nd highest-firing.

• Faithfulness concentrates in neuron features. The transcoder's Jacobian alignment with the MLP is about 0.3; restricting the sum to neuron-aligned features matches that, while non-aligned features reach only about 0.05. Top-k-pruned neurons, with no learned correction, match or exceed the full transcoder's Jacobian alignment across late layers.

• Non-neuron features are not stable across seeds. Given a feature is shared between two seed-differing runs (encoder and decoder cosine > 0.7), it is more than 50% likely to be a neuron feature for most training configurations. Scaling width does not fix this: even the 1M-feature dictionary only discovered 22 thousand stable features, comparable to the 20,480 neurons in the original model after accounting for sign.

• Non-neuron features stay near initialization. In the TopK pipeline (262k width, k=64, Gemma 3 4B L22, 15,000 steps), neuron features rotate to near-zero cosine with their initialization, while non-neuron features retain a median cosine similarity of +0.48 to init; many stay above 0.8. In the JumpReLU pipeline most alive features rotate significantly, but dead features remain uniformly distributed.

• Neurons win on causal replacement, lose on reconstruction. Untrained pruned neurons have FVU above 1 and high KL. Adding a trained linear map on the pruned MLP output drops KL below transcoders, while FVU stays substantially higher; extra reconstruction terms close some of the FVU gap without improving KL.

• Neurons match or beat transcoders on sparse probing. Across 113 binary classification tasks and eight transcoders spanning Gemma 2/3, Qwen 3, and Llama 3.1/3.2, neurons match or exceed the transcoder on every model.

• Transcoders do win on fine-grained bigrams. On 1,500 phrase bigrams sampled from FineWeb, transcoders win the comparison, but neurons still score a high mean AUC of 0.988, and the selected dictionary feature actually fires on the target bigram for only 75.6% of bigrams while neurons fire on close to 0%. Gemma 4 26B A4B, a mixture-of-experts model, matches bigrams in 20.1% of cases.

• Top neuron activations are as interpretable as features. On sort evals, neurons score comparably to transcoder features at the top of their activation distribution, but degrade noticeably under quantile sampling.

• Case studies reproduce prior results in the neuron basis. For linebreaking (Gemmascope L0-small, Gemma 3 4B, layer 20), 7 of 8 clean peaked line-position features track a single neuron's tuning curve. For arithmetic in Llama-3.1-8B layer 18, neuron features corresponding to the 28 localized addition neurons are the majority of features writing to the output subspace and the only ones with strong causal effects — ablating them drops addition accuracy from 94.3% to 30.6%. For the Russian word for "red" in Gemma 3 4B, three polysemantic layer-22 neurons are active; transcoder features mix 2–4 neurons and also discover a monosemantic "red" feature.

• Cross-layer transcoders look unfaithful. In two open-source cross-layer transcoders (Gemma 2 2B, Llama 3.2 1B), the extent of cross-layer decoder use is negatively correlated with alignment to the closest neuron, with Pearson ρ from −0.59 to −0.93 across layers and models.

Methodology in Plain English

The authors start from a Gated MLP, where each neuron has a gate vector and an up vector that are combined and then read out through a down vector. Because a neuron's behavior is approximately linear at large activations, they treat each neuron as two features — one per sign — with an "effective encoder" of W_up + W_gate or W_up − W_gate and a decoder of ±W_down. They then sparsify the MLP by keeping only the top-k activations, making it directly comparable to a transcoder of similar sparsity.

To find neuron features, they compute, for every transcoder feature, the cosine similarity of its encoder and decoder to the best-matching neuron's effective encoder and decoder. Plotting these two similarities produces a bimodal scatter: a tight high-similarity cluster plus a diffuse bulk. The cluster is the neuron features.

The authors then stress-test the split three ways. First, they compare each method's first-order behavior to the original MLP by measuring Jacobian alignment, a Frobenius cosine between the transcoder's per-feature rank-1 Jacobian contributions and the MLP's. Second, they train two transcoders that differ only in random seed and ask which features reappear. Third, they save the random initialization and measure how far each feature's encoder rotates during training.

For the neuron baseline they build replacement models: prune neurons to the top-k, optionally apply a learned linear map, add a learnable skip connection, and optionally add a per-neuron rank-1 term gated on activation, then measure FVU and KL. They also run sort evals, sparse probing, bigram detection, and attribution graphs on both bases, and inspect case studies from prior interpretability papers.

Why This Matters

The paper reframes a foundational assumption: if most of the transcoder features that actually fire are neurons in disguise, then the standard argument that neurons are "the wrong unit" is weaker than assumed, and much of the apparent advantage of dictionary learning comes from reconstruction rather than from model-relevant computation. It also gives practitioners a concrete diagnostic — check whether your dictionary features survive a reseed and align with the module's Jacobian — and a training recipe (neuron initialization) that makes dictionaries more neuron-aligned.

Real-world applications:

• Safety auditing and red-teaming: circuits for emotional states such as distress can be traced with neuron bases, enabling cheaper and more faithful audits of deployed models.
• Model debugging in production: a sparser, more faithful replacement model lets engineers attribute a specific wrong prediction to concrete components rather than to unstable learned features.
• Arithmetic and structured-reasoning analysis: the Llama-3.1-8B addition circuit result gives a template for localizing and causally verifying algorithmic capabilities.
• Architecture and training decisions: the recommendation against transcoders with less than 4× width expansion (and effectively against transcoders for MoE models) directly informs tooling budgets.

Industry relevance: attribution-graph tooling built on transcoders is used in frontier interpretability pipelines, so a finding that a large share of those features are duplicated neurons — and that the remainder is largely seed-dependent — affects how much trust to place in published circuit analyses and where to invest in better dictionaries.

Future Directions

• Attributing the non-neuron residue. The authors explicitly call for better methods of attributing the non-neuron structure the model uses, since that portion varies across seeds, misses the Jacobian, and barely leaves initialization.
• Cross-layer superposition. The authors remain "cautiously optimistic" about resolving superpositions across layers using supernode graphs, and want further behavior case studies with these tools.
• Mixture-of-experts architectures. The preliminary MoE results (including Gemma 4 26B A4B matching bigrams in 20.1% of cases) and the weak cross-layer transcoder faithfulness are flagged as undeveloped; since most open-source frontier models are MoEs, techniques that work there are described as necessary.
• Beyond sparse, single-token, high-magnitude activations. The authors note that low activations and multi-token patterns may be as or more interpretable but are out of scope, and that the architectural factors determining how many neuron features appear are not comprehensively explained.

Target Audience

Interpretability researchers who build or consume attribution graphs and sparse dictionaries; mechanistic-interpretability engineers choosing between SAEs, transcoders, and neuron baselines; and safety teams auditing frontier models. It is also valuable for anyone who has read claims that dictionary features are more interpretable than neurons and wants to know how much of that gap is real computation versus reconstruction. Readers without background in sparse dictionary methods will need to consult the cited transcoder and attribution-graph papers first.

Authors’ abstract

MLP neurons in transformer language models are widely understood to be polysemantic, motivating transcoders - sparse, learned approximations of MLP layers - as interpretable replacement models for mechanistic analysis. However, recent work shows that MLP neurons can be used to recover sparse, interpretable circuits (Arora et al. 2026, Ye et al. 2025), raising the question of whether transcoders are necessary for circuit analysis. We define neuron features as transcoder features whose weights are strongly aligned with individual MLP neurons, and show that they account for a substantial fraction of active features in late layers across 13 open-source transcoders. Only neuron features have a stable correspondence with something in the underlying MLP: other features vary more between training runs, do not match the MLP's Jacobian, and change less significantly from their random initialization. Our findings reframe MLP neurons as a stronger basis for interpretability than commonly assumed, and call for better methods of attributing the non-neuron structure that is used by the model.