Learning When to Trust via Selective Context Preference Optimization

AI summary

Overview

Research area: Natural Language Processing, specifically large language model robustness, instruction tuning, and preference-based alignment (Direct Preference Optimization).

Technical level: Intermediate. The paper is readable without deep mathematical background, but familiarity with LLM evaluation protocols and preference optimization will help.

Scope (one sentence): The paper introduces the idea of "selective trust" in language models, releases a 1,000-item, four-condition benchmark called MIST to measure it, and proposes SCOPE, a preference-optimization method that reduces susceptibility to misleading context without teaching models to ignore context altogether.

What This Paper Is About

Language models increasingly condition their answers on external text such as retrieved documents, search snippets, or user-provided hints. A single plausible but wrong signal can flip an answer the model originally got right, and the obvious fix, training the model to resist such signals, creates a new problem: a model that ignores all context looks robust but is useless when the context is actually worth trusting. This paper reframes the problem as selective trust and builds both a measurement instrument (the MIST benchmark) and a training method (SCOPE) that targets rejecting misleading signals while still benefiting from correct context and staying unmoved by irrelevant context.

Key Contributions

1. MIST (Misleading Signal Testbed). A human-annotated benchmark of 1,000 source items, each rendered under four matched conditions (clean, misleading, correct-context, irrelevant-context), producing 4,000 matched condition rows. The benchmark supports a paired metric, SC2W (signal-induced correct-to-wrong rate), which conditions on items a model already solves when clean and measures how often a misleading signal flips them to wrong.

2. Evidence that susceptibility is universal, and that reducing susceptibility is not the same as selective trust. The authors report nonzero SC2W for GPT-5.5 and widespread susceptibility across frontier API models and open-weight models, and show that prompt-defense, supervised fine-tuning (SFT), and misleading-only DPO raise misleading-context accuracy while eroding clean or correct-context accuracy.

3. SCOPE (Selective Context Preference Optimization). A signal-counterfactual preference framework that mines items a base model answers correctly on its own but gets wrong once a misleading signal is added, builds one response pair per item reused across all four contexts, and balances those pairs equally across the misleading, clean, correct-context, and irrelevant-context conditions inside a standard DPO objective.

4. Zero-shot transfer to three external benchmarks. Training only on the item-disjoint matched pool, the method transfers without training on target datasets to GSM-IC, GSM-Plus, and Sharma-style sycophancy examples.

Main Findings

• Susceptibility is widespread. On MIST, replacing a clean context with a plausible misleading one lowers accuracy for every model evaluated, frontier proprietary models included. Across 23 frontier and open-weight models, a single misleading signal costs 17.1 points on average.

• Even the strongest reference model is susceptible. GPT-5.5 scores 96.0 clean accuracy, 86.1 misleading-context accuracy, 98.5 correct-context accuracy, 96.1 irrelevant-context accuracy, 94.2 overall accuracy, and SC2W of 10.5. Gemini 3.1 Pro shows 12.9 SC2W, Claude Opus 4.8 shows 12.0, Claude Opus 4.7 shows 24.6, Claude Sonnet 4.6 shows 16.9, and Claude Haiku 4.5 shows 19.2.

• Weaker and smaller models are far more susceptible. Qwen2.5-Instruct-1.5B shows 51.7 SC2W and Qwen2.5-Instruct-0.5B shows 50.1 SC2W; SmolLM2-Instruct-1.7B shows 48.3.

• SCOPE roughly halves SC2W on Qwen3-4B. Base Qwen3-4B goes from SC2W 35.0 to 16.3, while clean accuracy moves 94.5 to 95.0, misleading accuracy 62.5 to 80.7, correct-context accuracy stays at 98.1, irrelevant accuracy 92.6 to 94.3, and overall accuracy 86.9 to 92.0.

• SCOPE also improves Llama-3.2-3B. Base SC2W 31.5 drops to 20.6, with clean accuracy 69.5 to 72.0, misleading accuracy 54.4 to 63.1, correct-context accuracy 78.5 to 80.0, irrelevant accuracy 69.3 to 71.6, and overall accuracy 67.9 to 71.7.

• Competing defenses buy resistance by damaging controls. On Llama-3.2-3B, Standard-DPO raises misleading-context accuracy to 63.3 but collapses correct-context accuracy to 56.4 from a base of 78.5, and OPSD is reported as negative overall (overall accuracy 63.0, SC2W 34.1). Prompt-defense and SFT also lower clean accuracy on Qwen3-4B (92.8 and 93.4 versus the base 94.5).

• The gains come without losing controls. Misleading accuracy rises by 18.2 points on Qwen3-4B and 8.7 on Llama-3.2-3B, with no control condition falling on either family.

• Matched pairing is load-bearing. Ablating the Qwen3-4B construction shows that unmatched or random pairs "substantially degrade" misleading accuracy, overall accuracy, and SC2W, and that removing the clean, correct-context, or irrelevant-context pairs weakens control preservation.

• Transfer is zero-shot. On GSM-IC, GSM-Plus, and Sharma-style sycophancy examples (300 items per dataset, GSM-IC and GSM-Plus scored by deterministic numeric exact match, Sharma scored with a fixed Qwen2.5-14B judge), SCOPE is best or tied on all four higher-is-better metrics for both families, with no external example used for training, model selection, or hyperparameter tuning. Standard-DPO attains the lowest Sharma Bias but loses accuracy on several tasks.

• The improvement holds across slices. SC2W reductions appear across existing-benchmark items, multiple-choice and boolean answers, and authoritative-looking signals such as answer keys, retrieved documents, and lecture notes, with the largest reductions on the most misleading authoritative-signal slices.

• Automatic scoring agrees with human judgment. In a reference-assisted audit, five of the six main metrics have Spearman correlations of 0.97 or higher, and SC2W remains positive at ρ = 0.821.

Methodology in Plain English

The researchers first built a testbed that isolates the effect of context. They took 1,000 source items—800 adapted from public QA, math, and reasoning benchmarks (StrategyQA, CSQA, ARC, GSM8K, MMLU, OBQA, MATH, AQuA, SVAMP) and 200 newly written by STEM-trained annotators—and rendered each one four ways that differ only in the text wrapped around the same question: no added context (clean), context pointing to a plausible wrong answer (misleading), context pointing to the gold answer (correct), and related but non-answer-bearing text (irrelevant). Because the question, answer space, answer type, gold answer, and plausible wrong answer stay fixed, any accuracy change can be attributed to the signal rather than to task difficulty. Annotation ran in two passes with independent verification for answer validity, wrong-answer plausibility, naturalness, ambiguity, answer leakage, and formatting, and everything was frozen before any model was run.

To score models they used deterministic, rule-based exact-match parsing of the final answer for multiple-choice, numeric, and boolean formats—no LLM judge on the main benchmark. Alongside per-condition accuracy they computed SC2W, which looks only at items a model got right when clean and reports how often a misleading signal flips them to wrong. An LLM judge was used only for the separate free-form external bias evaluation.

The training method reuses a standard DPO objective and changes only the data. For each mined item where the base model was clean-correct but misleading-wrong, they formed one pair: a correct full-completion response (chosen) versus the base model's own naturally terminated wrong response (rejected). That same pair was then attached to all four matched context prompts, so the same underlying problem appears under misleading, clean, correct, and irrelevant context. Rather than training on misleading pairs alone, which teaches blanket distrust, they weighted the four conditions equally: (λm, λc, λp, ρ) = (0.25, 0.25, 0.50, 0.50), giving 25 percent sampling mass to each of the four condition datasets, with the correct and irrelevant components combined. Weights were fixed before evaluation and not tuned on the MIST test set. The training pool shares no items with MIST.

They compared against four mitigations on the two trainable bases (Qwen3-4B and Llama-3.2-3B): prompt-defense as an inference-only warning, SFT using the same four-condition prompts and chosen responses without rejected responses, Standard-DPO trained on misleading-only pairs, and on-policy self-distillation (OPSD).

Why This Matters

Impact on research. The paper argues that robustness to misleading context is the wrong target for evaluation. Under single-condition evaluation, a model that distrusts every signal is indistinguishable from one that judges signals on their merits, and only the latter is useful. By releasing a matched-condition benchmark and a paired metric that conditions on clean-correct items, the work gives the field a way to separate signal-induced failure from ordinary inability, and it shows that changing what enters a standard DPO loss—not the loss itself—can move that metric.

Real-world applications (drawn from the signal channels and topics the paper covers):

• Retrieval-augmented generation and search. Signals such as retrieved document sentences and search-result snippets are among the nine supervised misleading channels; models must benefit from relevant retrieved text while resisting a confidently wrong snippet.
• Education and tutoring. Teaching-assistant comments and lecture notes are explicit signal channels in the benchmark, and tutoring systems need to accept correct guidance without parroting a wrong hint.
• Workplace policy and enterprise document reasoning. The human-authored subset includes workplace-policy interpretation, planning under constraints, multi-note information retrieval, and consumer or finance reasoning.
• Answer-key and calculator-style intermediates. Authoritative-looking notes and intermediate calculation notes are among the channels; assistants embedded in workflows must not adopt a plausible but wrong stated result.

Industry relevance. The failure mode is directly relevant to systems that pipe external text into a model's prompt, including RAG pipelines, tool-using assistants, and agentic workflows. The paper is explicit that this is not a security setting: the target is not refusal, but correctness across misleading and non-misleading contexts—so a method that simply hardens a model against all context would degrade the product where context is genuinely useful. The method's design also keeps the optimizer untouched and changes only the preference data, which lowers the barrier to adoption for teams already running DPO. The authors also note that conditioning SC2W on clean-correct behavior helps mitigate the fact that most items are adapted from public benchmarks and contamination cannot be fully excluded.

Future Directions

• Broader architectures and scales. The authors state that their two-family mitigation leaves broader architectures and scales untested; whether SCOPE-style balanced preference data works as well beyond Qwen3-4B and Llama-3.2-3B is an open question.
• From controlled rates to deployment prevalence. The benchmark is described as a controlled, text-only diagnostic, so its rates measure susceptibility rather than how often these failures occur in real deployments; a natural next step is measuring and reducing signal-induced flips in live pipelines.
• Chain-of-thought faithfulness. The paper notes that robustness under matched prompts is not evidence of chain-of-thought faithfulness; whether models that are robust to misleading signals also produce reasoning that faithfully reflects their decisions is left open.
• Contamination and benchmark expansion. Because 800 items are adapted from public benchmarks, contamination cannot be fully excluded. Whether expanding the human-authored portion, or building matched-condition sets in new languages and modalities, changes the measured susceptibility is not addressed.

Target Audience

This paper is most useful to researchers and practitioners working on LLM robustness, evaluation design, and preference-based alignment—especially those who build benchmarks or train models with DPO. It also benefits applied teams deploying retrieval-augmented or tool-using systems who need a concrete way to measure whether their model follows context when it should and resists it when it shouldn't. Readers looking for deployment-scale failure statistics or for results on architectures beyond the two training families evaluated here will find those explicitly outside the paper's stated scope.

Authors’ abstract

Language models increasingly condition their answers on external signals, and a single misleading one can turn a correct answer wrong. The obvious remedy, training models to resist such signals, hides a failure mode: a model that ignores all context looks robust yet is useless when the context is worth trusting. We recast the problem as selective trust and introduce MIST, a human-annotated benchmark that renders each reasoning item under four matched conditions (clean, misleading, correct-context, and irrelevant-context), together with SC2W, a paired metric counting how often a misleading signal flips a clean-correct answer to wrong. Across a comprehensive benchmark study, we observe that such a susceptibility is universal. We then propose SCOPE, which mines clean-correct/misleading-wrong failures and optimizes a standard Direct Preference Optimization (DPO) objective over matched preference pairs balanced equally across all four conditions, rather than over misleading items alone. Our approach substantially reduces SC2W on popular open-sourced models while preserving accuracy when the added context is clean, correct, or irrelevant. With this work, we argue that models should be judged on selective trust, not on resistance alone.