Cross-Prompt Reuse of Entity-Replacement Responses

AI summary

Overview

Research area: Mechanistic interpretability and activation-level control in large language models — specifically, whether an internal "response" recorded in one prompt can be reused in a different prompt about the same entity.

Technical level: Advanced. The paper assumes familiarity with hidden states, decoder-layer interfaces, activation patching, steering/function vectors, and log-probability scoring of complete answer strings.

Scope (one sentence): With model weights frozen, the paper quantifies how much of an entity-replacement response — the hidden-state or output-score change caused by swapping one entity for another — transfers between English factual questions about the same entity, and separates the part of that response shared by both questions from the part specific to the relation each question asks about.

What This Paper Is About

Prior work on steering vectors, function vectors and activation patching shows that recorded activation changes can be reused across prompts, and accounts of factual recall predict that mid-depth changes at an entity's tokens can be read out by another question's later layers. But a changed answer does not by itself tell you how much of the receiving question's response has been reproduced. This paper makes that reference measurable by replacing the entity in the receiving question itself (the "target") and comparing the target's actual response with the transferred one.

The goal is to answer two questions: how faithfully can a response recorded in one question reproduce the response in another, and does the transfer also reproduce the part of the response specific to the relation each question asks about?

Key Contributions

1. A measurable reference for response reuse. The authors define an entity-replacement response as the hidden-state or output-score change caused by replacing a named entity, and score reproduction as the share of the target's own signed answer-contrast change that a transfer recovers — rather than just checking whether the answer changed.

2. Two matched reuse routes compared under identical sources and weights. Hidden reuse adds the aligned source changes at 50% depth and lets the target's remaining layers read them out; direct reuse adds the source's own output-score changes. Because both routes use the same two source questions and the same similarity weights, the comparison isolates which read-out transforms the change.

3. A decomposition into shared and relation-selective components. Half the sum and half the difference of the two relation-swapped targets' contrast changes separate a shift both questions share from each question's shift toward its own relation's answers (Equation 4), letting the authors test whether relation-selective behavior is reproduced at all.

4. Depth, row-position and construction analyses. A pre-specified nine-interface depth scan, row-subset ablations, attention blocking, and a 90-construction single-vector testbed locate where and in what form the response remains reusable.

Main Findings

• Hidden reuse removes a substantial but incomplete share of no-change error. In the plain format (no chat template, no BOS token) on the original 35-block cohort, hidden reuse removes 45–51% of the no-change error on Qwen and Llama checkpoints but only 14–22% on Gemma. For Qwen2.5-0.5B specifically, mean error falls from 2.869 nats for zero prediction to 1.572 nats for hidden reuse, a 45.2% reduction.

• The BOS token drives a large format effect on Gemma. Plain-format Gemma-2-2B removes 22% and passes one of three pre-specified comparisons; restoring its default BOS token yields 68%, 65% and 58% on the original and two fresh cohorts, passing every comparison. The token also enlarges the observed changes, so shares are not directly comparable across formats.

• Chat format passes everything. In each checkpoint's chat template, all eighteen comparisons pass. Larger checkpoints, chat format and new entities show replication alongside format- or cohort-dependent failures; on the 31-block second fresh cohort, the small checkpoints pass none against zero prediction in plain format.

• Relation selectivity is not reliably reproduced on small plain-format checkpoints. There, other-relation hidden reuse mainly reproduces the shift both relations' questions share. On the original cohort, hidden reuse removes 49% of plain-format Llama-3.2-1B's no-change error, yet its selective-error reduction is −11.4%. Descriptive intervals for selective-error reduction never lie wholly above zero on the small plain-format checkpoints in any cohort. The selective component carries 9–38% of the observed variance, less on small checkpoints than at 7–9B.

• The hidden-over-direct advantage depends heavily on which relation the sources ask. With other-relation sources, direct reuse imports that relation's read-out and performs poorly (adding 12–28% to error at 7–9B in plain format, 63–78% in chat). Switching to target-relation sources narrows the gap from 0.55–5.05 to 0.20–0.90 nats pooled over the three cohorts, except for plain-format Gemma. The residual advantage is unconfirmed and not replacement-specific — a different replacement gains more from the hidden route than the correct one on every checkpoint in plain and plain+BOS formats, by 0.24–1.72 nats.

• Selective effects depend on the receiving question. A relation-neutral change ("Tell me about X." / "What do you know about X?") injected with equal weights into both relation-swapped targets is identical for the two questions, yet it lowers relation-selective error in 60–83 of 99 blocks, passing all six pre-specified tests. Identical random perturbations with the same row norms and the other replacement's neutral change lower selective error by at most 0.22 nats.

• Aligned changes beat single vectors. Span-aligned addition, which copies the changed entity span tokenwise, beats the best of 90 tuned single-vector constructions on five of six checkpoints (four under pooled correction across all 223 tests), with gains of 0.85, 0.66, 0.82, 1.38, 0.48 and 1.09 nats on Qwen 0.5B, Llama 1B, Gemma 2B, Qwen 7B, Llama 8B and Gemma 9B respectively (Llama 8B's interval includes zero). A broadcast last-entity-row vector has larger mean error than zero prediction on all six checkpoints.

• Reuse declines with depth. Span-aligned addition's share of error removed falls from 100% at the decoder input (d=0) to 46–70% at d=1/2, continuing to decline toward the last layer. Transferred entity rows remove somewhat less error than the target's own entity-row patch up to 50% depth.

• Answer-prefix rows matter less than expected. At the answer cue, hidden reuse reduces full-vocabulary KL divergence from the edited target by 88–97% relative to the unedited target, versus 44–87% for other-relation direct reuse. Omitting answer-prefix rows keeps most of the span-aligned share at d=1/2, so one question-plus-cue capture carries most of the gain under candidate scoring. In greedy generation, that capture reproduces the edited answer string for only 5–23% of targets within 16 tokens.

• Attention-blocking results differ by model family. Blocking attention from answer positions to entity rows above the interface retains only 3–15% of the error reduction on Qwen and Llama but more than half on Gemma; the pre-specified test that this blocking hurts more than blocking other rows passes only on Llama.

Methodology in Plain English

The researchers take pairs of factual questions from ZsRE, a zero-shot relation extraction dataset that supplies published questions, answers and exact relation labels. Each experimental "block" crosses two entity replacements with two relations, giving two source questions and one target question per replacement.

The manipulation is simple: take a question naming an entity (say, a question about SLIB), swap the entity for another (JDeveloper), and record how the model's internal states change. That difference is the "response." At the 50% depth mark — halfway through the network's layers — the researchers try to inject that same difference into a different question about JDeveloper, then let the target's own remaining layers process it. That is hidden reuse. The alternative, direct reuse, skips the target's computation and simply adds the source's output-score changes.

To measure fidelity, they score complete published answer strings token by token after known prefixes, sum probabilities within each answer group, and compare the log probability mass of the target relation's answers against the other relation's. Because the two target questions ask different relations, a shift both questions share cancels in half the difference of their changes, while each question's movement toward its own answers survives in half the sum. Zero prediction (predicting no change) is the baseline; the share of that baseline's error removed is the headline number.

Controls include using a different replacement's response, using relation-neutral sources, using randomly perturbed vectors matched in row norm, and comparing against tuned single-vector constructions (steering vectors, contrastive activation addition, entity-state replacement). Statistics are exact one-sided sign tests over blocks with Holm correction within pre-specified families, plus bootstrap and label-resampled intervals. Pre-specified comparisons were fixed in written protocols before outcomes were measured; the paper carefully labels which analyses are confirmatory, descriptive or post hoc.

Why This Matters

Impact on research. The paper supplies a quantitative reference that binary answer-agreement metrics cannot: how much of a replacement's signed complete-answer change transfers, and how much of that is relation-selective. It shows that overall fidelity and relation selectivity can diverge sharply — a model can reproduce most of a response while reproducing almost none of the part that distinguishes the two relations. It also reframes how to read cross-prompt transfer results: with read-out-matched (target-relation) sources, the apparent advantage of injecting inside the network over adding at the output largely narrows and is not specific to the correct replacement.

Real-world applications (as suggested by the work's framing):

• Auditing factual recall in deployed models by measuring what internal edits propagate to which questions.
• Evaluating knowledge-editing methods beyond paraphrase generalization, using signed contrast reproduction as a finer metric.
• Designing steering and activation-addition interventions with realistic expectations about what survives a change of prompt.
• Building interpretability tooling that reports transfer fidelity, not just final-answer agreement.

Industry relevance. Practitioners who use steering vectors or activation addition in production — for style control, safety steering, or knowledge editing — need to know that construction choice matters: span-aligned addition beat the best of 90 tuned single vectors on five of six checkpoints, and a broadcast last-row vector was worse than predicting no change at all on every checkpoint. The format sensitivity (Gemma's BOS token swinging shares from 22% to 68%) is also directly relevant to anyone benchmarking interventions across tokenizer and template choices.

Future Directions

• Disentangling relation from wording. The authors note that relation labels and question wording vary together and that some labels express overlapping predicates, so the two are not isolated. A design that varies wording while holding relation fixed, or vice versa, is the obvious next step.

• Resolving the unconfirmed residual advantage. The advantage of hidden over direct reuse with target-relation sources (0.20–0.90 nats pooled) is post hoc, unconfirmed, and not replacement-specific; a pre-specified test on the 31-block second cohort failed on all six checkpoints. Whether any real advantage exists at matched read-out remains open.

• Explaining format and cohort dependence. The BOS effect and the closed-format differences between Qwen, Llama and Gemma are not resolved; the authors note that chat format jointly changes template, BOS token, answer instruction and scored terminator, and that restoring BOS alone reproduces Gemma's chat-format improvement but not Llama's.

• Moving from candidate scoring to generation. The gap between candidate scoring (88–97% KL reduction at the cue) and greedy generation (5–23% exact answer reproduction within 16 tokens) is not attributed; separating autoregressive error accumulation from construction and capture differences is left open.

Target Audience

Researchers in mechanistic interpretability, activation steering, and knowledge editing who work with hidden-state interventions and want a quantitative fidelity benchmark rather than answer-level agreement. It is also useful for practitioners deploying steering vectors or function vectors who need evidence on how much of an intervention survives a prompt change, and for evaluation specialists interested in signed contrast metrics that separate shared from relation-specific behavior. Readers should be comfortable with transformer internals, log-probability scoring and nonparametric statistics; the paper is not an introductory treatment.

Authors’ abstract

With model weights fixed, we measure how much of an entity-replacement response transfers between English factual questions, beyond whether answers change. We add the hidden-state change from two source questions to a target question about the same entity, comparing the predicted log-probability shift between two relations' published answers, given known answer prefixes, with the measured one. Hidden reuse injects the aligned change at 50% depth before the target's remaining layers; direct reuse adds source output-score changes. With other-relation sources, hidden reuse removes 45% and 49% of no-change error on 0.5-1B Qwen and Llama. Gemma-2-2B removes 22% and passes one of three comparisons in plain text without its default beginning-of-sequence (BOS) token. A later protocol restoring it yields 58-68%, passing every comparison on three cohorts. Larger checkpoints, chat format and new entities show replication and format- or cohort-dependent failures. On small plain-format checkpoints, other-relation hidden reuse mainly reproduces the shift both relations' questions share, without reliably lowering relation-selective error. With target-relation sources, hidden reuse's post hoc advantage over direct reuse is 0.2-0.9 nats outside chat format, unconfirmed and not replacement-specific. The same relation-neutral change at both questions' entity tokens lowers relation-selective error more than random or other-replacement changes, indicating receiving-question dependence.