Encoded Early, Used Late: Where Transformers Begin to Act on an Inferred Partner's Expertise

AI summary

Overview

Research area: Mechanistic interpretability of large language models — specifically, how transformer residual streams encode versus how they use an inferred social attribute.

Technical level: Intermediate. Readers should be comfortable with linear probing, activation patching, residual streams, and layer-by-layer analysis of transformer depth.

Scope: A single-model, single-corpus initial demonstration that a dialogue partner's expertise is linearly decodable early in a transformer's depth (peak 0.79 accuracy at layer 8) but only becomes causally active past the network midpoint (near-complete effect from layer 40), with the observation stated as a falsifiable hypothesis rather than a general law.

What This Paper Is About

Probing a transformer's intermediate layers can reveal that some attribute is "readable" in the residual stream, but readable does not mean used — skip connections copy information forward even when nothing downstream consumes it. Prior evidence for this decodable-versus-causal gap came from attributes stated directly in the input; this paper asks whether the same separation holds for an attribute the model must gradually infer across a conversation, namely how expert its dialogue partner is. The authors build a corpus of multi-turn research-planning dialogues between model-played personas at four expertise levels and measure both where expertise can be decoded and where it causally shapes the output.

Key Contributions

1. A controlled paradigm that separates where an inferred attribute is decodable from where it causally drives the output, using counterfactual patching with a norm-matched random control, a trivial-recovery calibration control, and a probe-free diagnostic.
2. ExpertCollab, a constructed and released corpus of 28 synthetic multi-turn dialogues in which expertise is signaled only through how each persona engages and is never stated (all 28 pass an LLM leakage detector screening for explicit credentials).
3. A structural finding: the inferred partner-expertise attribute is decodable early but causally inert at that depth, and becomes causal only past the network midpoint — unlike the statically specified control attribute, which stays decodable throughout.
4. A falsifiable hypothesis (Appendix F) linking the depth gap between peak decodability and causal onset to how much an attribute must be inferred rather than stated.

Main Findings

• Partner expertise decodes early and then fades. Pooled probe accuracy on the partner's expertise level peaks at 0.79 at layer 8 and decays to near chance by layer 40. The peak-decoding layer falls in the early layers, before the midpoint of the network.
• A stated attribute does not fade. The speaker's own expertise level, fixed by the system prompt, stays above 0.60 accuracy at every layer, showing the architecture can carry a stated attribute forward without loss.
• The early signal is not merely lexical. A surface-feature baseline using hedging rate, mean sentence length, question rate, type-token ratio, and vocabulary complexity reaches 0.32 accuracy — above the 0.25 chance level but 0.45 below the neural probe at 0.786 at the peak cell.
• Early representation is causally inert. With patch and readout at the same layer the normalized effect averages 1.01 (calibration). At the peak-decoding layer the effect is near zero, at 0.01, meaning the injected expertise difference leaves the fixed late-layer readout essentially unchanged.
• Late representation propagates almost completely. The effect rises through the middle layers, reaches 0.86 at layer 36, and exceeds 0.90 from layer 40 — a separation of more than an order of magnitude from the early-layer value.
• The effect is specific to the expertise direction. A norm-matched random vector control produces no comparable rise and far wider intervals, so the early-layer null is not a generic consequence of perturbing the stream.
• A probe-free diagnostic marks the same transition. Cosine dissimilarity between patched and unpatched final-layer activations is flat at the first layers, rises sharply from layer 8, and saturates near 0.40 by layer 32 — locating the transition without any trained classifier.
• Output behavior reflects the same split. Top-5 token overlap declines gradually from 0.97 at the first layers to about 0.67 by layer 32. Late-layer patches change about one in three of the five most probable next tokens, while peak-decoding-layer patches leave the output almost untouched.
• Recovery is direction symmetric and consistent across conditions. Patching toward a higher-expertise source (24 pairs) gives 0.653 with a 95% interval of [0.597, 0.713]; toward a lower-expertise source (24 pairs) gives 0.660 with [0.614, 0.704]. The 0.007 difference sits well inside both intervals.
• Domain variation is modest but present. Recovery by domain: graph-neural-network robustness 0.691, scaling laws 0.668, speculative decoding 0.659, soil microbiome 0.598. The single non-machine-learning domain shows the weakest recovery.
• The transition follows depth, not conversational position. Every one of the six speaker turns is near zero at early layers and near complete at late layers, indicating the causal structure tracks network depth rather than where the conversation has progressed.
• Errors are graded, not categorical. The row-normalized confusion matrix at the peak cell concentrates mass near the diagonal, with most confusions between neighboring expertise levels.
• The causal-onset boundary aligns with a known organizational boundary. The authors relate the gap between layer 8 and the causal onset near layer 36 to the midpoint transition reported by Lad et al. (2025).

Methodology in Plain English

The pipeline has four stages.

Generate. The authors build ExpertCollab: 28 synthetic research-planning dialogues of 12 turns each, between two model-played agents. Dialogues are generated by meta-llama/Llama-3.3-70B-Instruct-Turbo served through Together AI, with a 400-token cap per turn. The corpus crosses seven persona pairings with four research scenarios. The four expertise levels are a high-school student, a junior researcher, a researcher, and a professor (integer coded 0 through 3). Persona prompts specify engagement style, characteristic blind spots, and interaction tendencies, and explicitly forbid stating credentials, so expertise must be inferred from how an agent engages. Both speakers are played by one model, which removes cross-role style confounds but introduces a same-model limitation. An LLM leakage detector screens every transcript for explicit titles, credentials, or years of experience; all 28 dialogues passed.

Extract. Activations are read from meta-llama/Llama-3.3-70B-Instruct (80 layers, hidden size 8192) via NDIF and nnsight, sampling every fourth layer (layers 0, 4, 8 through 76, for 20 checkpoints). The key position is the "participant perspective": the chat history is rebuilt from the target speaker's point of view — its own prior turns as assistant messages, the partner's prior turns as user messages — and activations are taken at the final token, the generation boundary. This is the model's working state at the moment it would begin generating, not a passive reading of the transcript. Both speakers are pooled into 56 role-agnostic instances.

Probe. At each layer and turn, features are standardized and an L2-regularized multinomial logistic-regression probe is fit. With 8192 features and far fewer samples, the inverse regularization strength is set to 0.01 with the saga solver and up to 5000 iterations, fixed a priori and not tuned against the layer profiles. Cross-validation uses stratified k-fold with k = max(2, min(5, c_min)), which gives k = 5 for the pooled 4-class targets. Two pooled targets are trained: the partner's level and the speaker's own level. Reported intervals are 95% two-sided t-intervals over the five fold accuracies. With roughly 9 to 11 instances per class per fold, per-cell estimates carry substantial uncertainty and should be read as a layer profile rather than point estimates.

Patch. Counterfactual pairs share a scenario and hold one participant's persona fixed while varying the other's expertise across the four levels. For each pair, the source-minus-target activation difference is added to the target's residual stream at a given layer, the forward pass runs to completion, and the readout probe is fixed at the last layer so the patch must propagate causally to reach it. The normalized effect is E(L) = (p_patched − p_target) / (p_source − p_target): near one means full propagation, near zero means none. The main analysis uses 48 cross-level pairs. Cells where the source and target baselines differ by less than 0.05 are masked because the probe cannot resolve the two conditions there, removing 20.1% of cells. Layer and turn-band summaries and 95% confidence intervals use a cluster bootstrap that resamples pairs, with 1000 resamples.

Controls and probe-free checks. Trivial recovery sets the patch layer equal to the readout layer. Null restricts to same-level pairs where the difference is exactly zero. Random replaces the expertise difference with a norm-matched random vector of the same magnitude and a random direction. Two probe-free metrics are recorded on the same forward passes: top-5 token overlap between patched and unpatched runs, and cosine dissimilarity between patched and unpatched final-layer activations. KL and Jensen-Shannon divergences between next-token distributions were also computed but were numerically tiny, on the order of 10^-8, and are omitted in favor of the two interpretable metrics.

Why This Matters

Impact on research. The paper argues that decodability does not establish function: the most readable layer for this attribute is causally silent, and the layers that matter carry little decodable signal. The residual-stream property that makes probing attractive — preservation of information through skip connections — is precisely what makes decodability an unreliable guide to causal role. The practical methodological consequence is that probes should be paired with intervention before functional conclusions are drawn. The authors also link their causal-onset boundary to the midpoint transition reported by Lad et al. (2025), giving the result a structural rather than task-specific character, and state the inferred-versus-stated contrast as a testable hypothesis rather than a settled law.

Real-world applications (grounded in the behaviors and topics the paper discusses):

• Steering partner-conditioned behavior, such as sycophancy, which the paper notes is conditioned on an inferred user attribute. If such behavior must be steered in the network's second half rather than at the most-readable layer, intervention design changes accordingly.
• Implicit personalization in assistants, where a system adapts its explanations to a user's apparent expertise. The paper's result bounds where such adaptation is actually computed.
• Dialogue research-planning and collaboration tools, the exact setting of ExpertCollab, where an agent must infer a collaborator's level from how they write.
• Model auditing and interpretability tooling, since the result defines an intervention window for anyone trying to read out or control inferred relational attributes.

Industry relevance. The finding is directly relevant to teams building steering, monitoring, or interpretability tooling on frontier-scale models: it says that the layer selected by probe accuracy is likely the wrong layer to intervene on for inferred attributes, and it gives a concrete measurement protocol (calibration control, norm-matched random control, and a probe-free check) that can be reproduced on in-house models. The authors caution that the layer boundaries are likely architecture dependent, so the specific layers reported here should not be transplanted between models without re-measurement.

Future Directions

1. Cross-model replication. All results are for one model (Llama-3.3-70B Instruct) on one synthetic, same-model corpus of 28 base conversations. Whether the layer boundaries hold across architectures, scales, and training recipes is untested.
2. The explicitness test. Appendix F predicts that varying how explicitly an attribute is provided — stated outright, implied by a single cue, or inferable only from indirect behavior — should move the causal-onset depth monotonically, with more inference pushing the onset later. Controlled prompt manipulations on a single attribute would convert the qualitative observation into a quantitative relationship.
3. Breaking the same-model confound. Because the corpus is generated by the model later probed, the representation may partly reflect recognition of its own generation patterns. Corpora generated by a different model, or human dialogue, would isolate whether the effect depends on self-generated text.
4. Generalizing across inferred attributes. The paper studies one relational attribute, expertise. Whether intent, emotional state, or other inferred attributes show the same early-decode, late-causal structure is an open question.

Target Audience

Mechanistic interpretability and alignment researchers who work with activation probing, activation patching, or steering vectors and need to decide where in a network to intervene. Also useful for researchers studying dialogue, personalization, and sycophancy who require a causal, not merely correlational, account of how partner attributes shape model output, and for practitioners building monitoring or control tooling on top of open-weight models. The paper's explicit scope statement — one model, one synthetic same-model corpus of 28 conversations, treated as an initial demonstration — makes it a methodological template and a hypothesis-generating result rather than a settled scaling-law claim.

Authors’ abstract

A transformer can make an attribute linearly decodable in its residual stream at a depth where that attribute does not yet influence the output. This gap between where information is readable and where it is used has been shown for attributes stated directly in the input. We ask whether it also holds for an attribute the model must infer gradually over a conversation, namely how expert its dialogue partner is. Using ExpertCollab, a corpus of multi-turn research-planning dialogues between model-played personas at four expertise levels, we find that partner expertise is most decodable in the early layers and falls to near chance before the midpoint of the network. Counterfactual patching shows that injecting the expertise difference at the layer of peak decodability barely changes a fixed late-layer readout, whereas the same difference injected past the midpoint propagates almost completely, a separation of more than an order of magnitude. A content-matched random control and a probe-free diagnostic place the transition at the same early layer, and a statically specified control attribute stays decodable throughout. An inferred relational attribute is therefore represented well before it becomes causally active, which bounds where any attempt to read out or steer partner-conditioned behavior must intervene. We use one model on a synthetic corpus as an initial demonstration.