To Copy or Not to Copy: Copying Is Easier to Induce Than Recall

AI summary

Overview

Research area: Natural Language Processing / mechanistic interpretability, specifically the study of how large language models arbitrate between parametric knowledge (facts stored in weights) and contextual information in the prompt, in Retrieval-Augmented Generation (RAG) settings.

Technical level: Intermediate. The paper assumes familiarity with transformer internals (layers, hidden states, attention heads, MLP blocks), steering vectors, and standard QA metrics such as Exact Match (EM), F1, and perplexity (PPL).

Scope: A mechanistic study that builds an "arbitration vector" from model activations and injects it at selected layers and token positions to causally toggle three instruction-tuned models between copying from context and recalling from memory, evaluated on two open-domain QA benchmarks.

What This Paper Is About

When a language model is given retrieved context, it must decide whether to use the information in that context or rely on facts stored in its parameters. Prior work has observed traces of this "arbitration" inside models, but rarely demonstrated that the choice can be causally controlled. This paper asks whether a targeted intervention on internal representations can force the model in either direction: from copying to recalling (Copy → Recall) and from recalling to copying (Recall → Copy). The answer is yes, but the two directions are not symmetric.

Key Contributions

1. The authors identify a general "arbitration vector" spanning 27 relations that isolates a control signal governing the trade-off between parametric recall and contextual copying.
2. They show that scaling this vector (α = +30.0, -3.0) steers behavior between Copy → Recall and Recall → Copy, and that this mechanism is spatially localized.
3. They reveal an asymmetry: Recall → Copy is robust and requires smaller scaling, while Copy → Recall requires larger scaling and succeeds only at the object token, with significant perplexity spikes.
4. They create a curated arbitration dataset designed to disentangle parametric knowledge use from reliance on contextual information.

Main Findings

• Behavioral shifts generalize across architectures and datasets. Experiments on three architectures — Gemma2-2B (decoder-only), T5Gemma-2B (encoder-decoder using a prefix language modeling objective), and OLMo-2-0425-1B (1B-parameter decoder-only) — and two open-domain QA benchmarks, PopQA and EntityQuestions (PEQ), show consistent behavior shifts.

• Copy → Recall requires large positive scaling at early layers. Injecting α = +30.0 at early layers (L:6-6 for Gemma2-2B, L:3-2 for T5Gemma-2B, L:5-2 for OLMo-2-0425-1B) restores internal knowledge. Under authoritative misleading context, unsteered true-object EM is near zero. Reported peaks are PopQA 66%, 74%, 25% and PEQ 53%, 74%, 36% for Gemma2-2B, T5Gemma-2B, and OLMo-2-0425-1B respectively (the body text states Gemma2-2B recovers to 66-42%, T5Gemma-2B to 73-74%, and OLMo-2-0425-1B to 25-36%).

• Recall → Copy requires smaller negative scaling at middle-to-late layers. Injecting α = -3.0 (layers 12, 10, 5 for the three models) shifts the model toward copying the counterfactual. Reported peaks are PopQA 34%, 46%, 15% and PEQ 38%, 39%, 17% for the same three models; baseline EM scores were reported as 34-38% in Gemma2-2B, 54-39% in T5Gemma-2B, and 15-17% in OLMo-2-0425-1B.

• Copying is easier to induce than recall. Forcing copying is described as a low-resistance transition needing minimal scaling, causing minimal perplexity increase, and succeeding from any control point (object, subject, or last token). Forcing recall is described as high-resistance: it requires aggressive suppression, larger scaling, and sharp perplexity spikes, and succeeds only at the object token.

• Position matters for Copy → Recall. Successful recall only occurs at the counterfactual object, where the intervention "jams" attention already locked onto the context. It fails at the subject (which reinforces the attention link to the counterfactual) and at the last token (too late, since the decision has already been made). A significant drop in copying ability at subject and last token did not translate into restored internal knowledge.

• Operating range trade-off. At α = ±100.0 the model shows massive perplexity spikes and generation quality collapses, while F1 scores often reach high values. Moderate values (α = +30.0, -3.0) are the optimal stable range. At α = ±1.0 the intervention is too weak to steer output.

• Mechanistic picture for Copy → Recall. In the Gemma analysis (Context-First, 1,434 PopQA samples), injection at the object position immediately affects object-focused attention at the injection layer and reduces it in later layers, while subject-focused attention increases. MLP effects are less discernible, showing a slight but statistically weak increase after the subject-focused attention spike.

• Model scale may matter. OLMo-2-0425-1B shows the same directional behavior but substantially weaker effects than the two 2B models, suggesting the property may emerge with scale rather than being universal.

Methodology in Plain English

The researchers first built a dataset to isolate the two behaviors. Using the ParaRel dataset, they sampled 150 entities per relation across 27 Wikipedia properties, added historical Wikipedia pageview statistics (2000–2025) stratified into "High" and "Low" popularity tiers, and generated contexts with the OpenAI API (gpt-5-mini model) constrained to 12–16 words. Each instance has a counterfactual object replacing the true object, and contexts come in two kinds: Irrelevant Context (IC), where the counterfactual sits in a sentence unrelated to the subject, and Relevant Context (RC), where it forms a plausible but false relation-matching claim. In total, 7,642 data points were collected.

From this data they computed a steering vector per layer and extraction location. They pooled hidden states at three candidate locations (subject, counterfactual object, and last token), computed average centroids for IC and RC conditions across the whole dataset, and took the difference between the two centroids as the arbitration vector. At inference, they add this vector — scaled by α — to the hidden states at chosen layers and token spans.

The intervention has two directions: positive scaling (parametric steering, aiming to suppress context) and negative scaling (contextual steering, aiming to accept external evidence over memory). Scaling factors tested were {+1.0, +3.0, +30.0, +100.0} and {-1.0, -3.0, -30.0, -100.0}. Evaluation used PopQA and PEQ with synthetic templates — generic "Archive Templates" to force recall and relation-specific templates to force copying — with greedy decoding to avoid sampling variance. Overall-trend analysis sampled 2,000 examples per dataset; behavioral sweeps used a relation-stratified subset of 300 examples per dataset; mechanistic analyses were restricted to samples where subject and object were single tokens. Metrics were Exact Match and token-level F1 against both the true object and the counterfactual, plus perplexity as a fluency check.

Why This Matters

The work converts correlational observations about context-vs-memory arbitration into causal claims: the model's choice can be pushed in either direction by editing internal activations, with no fine-tuning required. That makes it a contribution to mechanistic interpretability and a practical handle for anyone who needs to understand or control RAG behavior.

Real-world applications that follow from the findings (the paper itself does not enumerate deployed applications):

• Reducing hallucination/context conflict in RAG systems. Understanding why a model ignores authoritative context, and where in the network the decision is made, informs debugging of retrieval-augmented pipelines.
• Making models defer to retrieved evidence. The Recall → Copy direction is cheap and robust, which is relevant when a system must prefer fresh external facts over stale parametric memory.
• Prioritizing parametric knowledge when retrieval is wrong. The Copy → Recall direction offers a mechanism for suppressing misleading retrieved claims, though the paper notes its fragility and cost.
• Interpretability tooling for model auditing. Token-level interventions provide a finer-grained probe than high-level control of information source.

Industry relevance is mainly for teams building or auditing retrieval-augmented generation systems, and for interpretability groups wanting causal, token-localized control methods. The paper notes limitations here: the setup inserts context directly into the prompt and does not model retriever failures, ranking, passage selection, or realistic retrieved contexts, so interaction with a full end-to-end RAG pipeline was not investigated.

Future Directions

• Prompt dependence. The instruction-tuned models were prompted explicitly to copy a minimal span if present and otherwise fall back on internal knowledge. Extended experiments could test how much the extracted arbitration vector changes with the prompt and whether prompt-robust vectors can be extracted.
• Scale and model diversity. Only three architectures across two parameter scales and three model families were studied. How findings scale to larger models, different instruction-tuning recipes, long-context variants, or models trained explicitly for RAG remains open.
• Statistical sample size. Because of the combinatorial explosion of layer, α, counterfactual-type, and prompt-topology combinations, 300 samples per benchmark were evaluated using relation-stratified sampling.
• Modeling the retriever. The setup emulates retrieval by inserting context directly, without modeling retriever failures, ranking, passage selection, or the distribution of retrieval errors, and without evaluation on realistic retrieved contexts.

Target Audience

Researchers in mechanistic interpretability and LLM internals; engineers building or debugging retrieval-augmented generation pipelines; and graduate students or practitioners with intermediate familiarity with transformer architecture, steering vectors, and standard QA evaluation metrics. Readers looking for production-ready RAG recipes will find the limitations section directly relevant, since the paper studies synthetic contexts rather than a full retrieval pipeline.

Authors’ abstract

Language models used in retrieval-augmented settings must arbitrate between parametric knowledge stored in their weights and contextual information in the prompt. This work presents a mechanistic study of that choice by extracting an \emph{arbitration vector} from model activations on a curated dataset designed to disentangle (i) irrelevant contexts that elicit parametric recall and (ii) relevant but false contexts that elicit copying. The vector is computed as the residual-stream centroid difference between these regimes across 27 relations, and is injected as an additive intervention at selected layers and token spans to steer behavior in two directions: Copy$\rightarrow$Recall (suppressing context use) and Recall$\rightarrow$Copy (inducing the model to copy any token from the context). Experiments on three architectures (decoder-only and encoder/decoder) and two open-domain QA benchmarks show consistent behavior shifts under moderate scaling while monitoring accuracy and fluency. Mechanistic analyses of attention routing, MLP contributions, and layer-wise probability trajectories reveal an asymmetry: inducing copying is an easy ``reactivation'' process that can be triggered at different locations in the input, while restoring recall is a ``suppression'' process that is more fragile and strongly tied to object-token interventions.