Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models

AI summary

Overview

• Research area: Natural Language Processing / mechanistic interpretability of large language models, focused on prompt-order sensitivity in multiple-choice question answering (MCQA).
• Technical level: Intermediate. The paper combines behavioral benchmarking with mechanistic interpretability tools (attention analysis, gradient attribution, activation patching), so some familiarity with transformer architectures is helpful.
• Scope: A diagnostic study of why decoder-only LLMs perform far worse when the context is placed after, rather than before, the question and options in MCQA prompts, identifying causal attention masking as the mechanism.

What This Paper Is About

Large language models are known to be sensitive to how a prompt is arranged, but prior work has mostly described what changes rather than why. This paper studies a concrete and striking case in multiple-choice question answering: putting the context first (CQO: context-question-options) beats the reverse ordering (QOC: question-options-context) by a large margin. The goal is to identify the underlying architectural mechanism and to test whether targeted interventions can close that gap.

Key Contributions

1. A systematic documentation of the CQO-QOC gap. The authors measure average accuracy across 21 decoder-only LLMs on 4 datasets and report that CQO outperforms QOC by over 14 percentage points consistently (average gap of +14.7% in the full per-model table).
2. Formulation and elimination of three competing hypotheses. They test (1) biased training data, (2) failure to recall options ("lost-in-the-middle"), and (3) causal attention, ruling out the first two and confirming the third.
3. Direct causal evidence for causal masking. Architecture comparisons show decoder-only models have a 14.72% gap, encoder-decoder models (Flan-T5) 2.30%, and encoder-only models (BERT, RoBERTa, ALBERT) a near-zero gap of 0.02%.
4. Targeted interventions that degrade CQO or repair QOC. Attention pruning drops CQO from 69.26% to 42.46%; activation patching raises QOC by 6.0 points; repeating options after the context (QOCO) raises QOC by 8.2 points; chain-of-thought shrinks the gap from 14.72 to 7.47.

Main Findings

• The core ordering effect: Placing context before the question and options (CQO) consistently outperforms the reverse order (QOC) by over 14 percentage points across a wide range of models and datasets (4 benchmarks, 21 decoder-only LLMs of 0.5B to 9B parameters from the LLaMA 3, Qwen 2.5/3, and Gemma 2 families).

• Instruction tuning is not the cause: Comparing nine matched base-instruct model pairs, the CQO-QOC gaps are nearly identical (Base: 14.70%, Instruct: 14.12%), ruling out training-distribution bias as the primary driver.

• In-context learning barely helps: Training the model on the QOC format through few-shot demonstrations (up to 5-shot) improves QOC accuracy by only 3.1% (from 54.54 to 57.63 on average), still far below the CQO baseline of 69.26%.

• Option recall is not the problem: Option recall accuracy is high in both orderings (CQO 93.5%, QOC 94.7%), and QOC is sometimes higher, which rules out "lost-in-the-middle" memory failure as the cause.

• Causal masking creates an information bottleneck: In QOC, causal masking prevents option tokens from attending to the context, which appears after them. Removing the context entirely (QO) barely changes QOC performance, indicating the context is effectively invisible.

• Attention and attribution evidence: Option tokens receive exactly zero attention from context tokens in QOC by construction of the causal mask. Context attribution is 0.797 in CQO but only 0.335 in QOC, an average ratio of 2.38x across datasets. Layer-wise, attention to options increases with depth in QOC but decreases in CQO.

• The final decoding step cannot fully repair this: Because causal masking prevents option hidden states from being context-conditioned in the first place, and later tokens cannot retroactively update earlier states, evidence-option comparison must happen at a single final step. The authors note the paper's theoretical account establishes that option hidden states under QOC are structurally independent of the context (I(h_O^QOC; C | Q, O) = 0).

• Longer contexts widen the gap: Datasets with longer average context length show larger gaps: LogiQA (~70 words) 6.2%, SciQ (~70 words) 7.3%, RACE-M (~195 words) 24.8%, RACE-H (~305 words) 20.8%.

• Answer position matters: Earlier answer positions suffer larger drops: option A 22.4%, B 20.5%, C 19.2%, D 9.9%, with option D being the most robust because later options sit near the context.

• Interventions confirm the mechanism: Attention pruning in CQO (blocking option-to-context attention) drops accuracy from 69.26% to 42.46% (-26.8), consistent across families (Qwen 28.1%, LLaMA 25.3%, Gemma 26.9%). Activation patching option hidden states (layers 12-23 for 24-layer models) raises QOC by 6.0 points. Repeating options after the context (QOCO) raises QOC by 8.2 points.

• Chain-of-thought partially closes the gap: Replacing greedy answer scoring with "Let's think step by step" shrinks the CQO-QOC gap from 14.72 to 7.47 on average. The authors report that for a larger closed-source model (Gemini), CoT reduces the gap by 83.4%, though this appears in the appendix of the provided text without the underlying figures.

Methodology in Plain English

The researchers first establish the effect by evaluating the same MCQA prompts in two orders, CQO and QOC, and computing the performance gap (Δ = Acc_CQO − Acc_QOC) across 21 decoder-only models and four reading comprehension benchmarks (LogiQA, SciQ, RACE-M, RACE-H).

They then test three explanations one at a time. To test whether CQO is simply more familiar from training data, they compare matched base and instruction-tuned pairs and try few-shot in-context learning in the QOC format. To test whether the model forgets the options, they ask the model to recall each option verbatim after seeing the prompt. To test whether causal attention is responsible, they compare architectures: decoder-only models use causal attention, encoder-only models (BERT, RoBERTa, ALBERT) use bidirectional attention, and encoder-decoder models (Flan-T5) use a bidirectional encoder.

They then dig into the model internals. They track how attention to option tokens evolves across layers, and use Gradient × Input attribution to measure how much context tokens contribute to the prediction. They also run causal interventions: pruning attention edges between option and context tokens to damage CQO, and patching option hidden states from a CQO run into a QOC run to repair QOC. Finally, they test prompt-level fixes: repeating the options after the context, and chain-of-thought prompting, which generates reasoning tokens that sit after the full prompt and can attend to both the options and the context.

Evaluation uses protocols matched to each architecture: constrained likelihood over the four option tokens for decoder-only models, masked token prediction in a cloze template for encoder-only models, and decoder probability scoring for encoder-decoder models. Inference used greedy decoding, single runs, maximum 16 output tokens, bfloat16 precision, and NVIDIA A6000 GPUs.

Why This Matters

• Impact on research: The paper moves prompt-sensitivity research from behavioral description to a mechanistic account, showing that a well-known phenomenon is a direct consequence of the causal attention mask rather than data or memory. It also connects the MCQA literature to mechanistic interpretability tooling (activation patching, attention ablation, gradient attribution) and gives a concrete structural claim that future work can build on or challenge.

• Real-world applications:

  • Retrieval-augmented generation (RAG): Systems that append retrieved documents after a question may be crippled by the same bottleneck; the paper's QOCO (repeat options) and CoT results suggest placement and repetition strategies that avoid it.
  • Evaluation and benchmarking: Since option order and context position shift accuracy by double digits, benchmark designers must control or report prompt ordering to avoid measuring format sensitivity instead of model capability.
  • Prompt engineering and product design: Practitioners can apply the paper's practical remedies (context-before-question, CoT, repeating key options after context) without any model changes.
  • Model selection and deployment: The architecture comparison gives a principled way to reason about when bidirectional encoders or encoder-decoder models may be a better fit than decoder-only models for context-heavy scoring tasks.

• Industry relevance: The mitigations are all inference-time and require no retraining, which makes them immediately usable in deployed systems. The paper is also diagnostic in scope and explicitly does not propose a practical training-time fix that closes the gap without runtime overhead.

Future Directions

1. A training-time fix. The authors state in their Limitations section that they have not yet proposed a practical way to close the gap without runtime overhead, and leave this for future work.
2. A stronger theoretical account. The single-step bottleneck derivation in Appendix F is described by the authors as deliberately minimal; extending it beyond the result that option hidden states under QOC are independent of context is an open direction.
3. Extending architecture coverage. The main text reports results on decoder-only, encoder-decoder, and encoder-only models, and the authors note that additional architectures (state-space models, hybrid linear attention, and diffusion language models) are reported in Appendix D, which is truncated in the provided text.
4. Understanding residual gaps. After chain-of-thought prompting, a residual CQO-QOC gap of 7.47 remains, which the authors attribute to option tokens still being initially encoded without context and only a few reasoning steps being available to recover alignment; how to reduce this further is unresolved.

Target Audience

• Mechanistic interpretability researchers interested in causal attention, activation patching, and attention ablation as tools for explaining LLM behavior.
• NLP practitioners building RAG, QA, or MCQA systems who need concrete, inference-time guidance about prompt component ordering.
• Benchmark and evaluation designers who want to understand how much of a model's reported accuracy depends on superficial prompt structure.
• Model architects and researchers comparing decoder-only, encoder-decoder, and encoder-only designs for context-dependent tasks.

Note: The provided paper content is truncated (Appendix C.6 and Appendix D/E/F are partially or fully cut off), so details such as the full state-space, hybrid linear attention, and diffusion language model results, and the appendix-level Gemini figures, are not available here and are not reported in this summary.

Authors’ abstract

Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and datasets. Through systematic architectural analysis, we identify causal attention as the core mechanism: in QOC prompts, the causal mask prevents option tokens from attending to context, creating an information bottleneck where context becomes invisible to options.