LLMs Show Surface-Form Brittleness Under Paraphrase Stress Tests

AI summary

Overview

Research area: Natural Language Processing — specifically LLM evaluation, benchmark validity, and robustness to surface-form perturbations.

Technical level: Intermediate. The paper is short and clearly written, but it assumes familiarity with multiple-choice QA benchmarks, instruction-tuned decoder-only models, decoding parameters, and 4-bit quantization.

Scope (one sentence): The paper measures how much accuracy drops when two 7B instruction-tuned models are re-evaluated on automatically paraphrased versions of ARC-Easy and ARC-Challenge questions, using a paraphrase-cleaning pipeline and cross-family model pairings.

What This Paper Is About

Static benchmark scores for LLMs can be inflated when test items, or near duplicates of them, appear in pretraining data, and models can also succeed for the wrong reason by latching onto superficial phrasing. This paper asks a simple question: if the meaning of a benchmark question is preserved but its wording is changed, does model accuracy hold up? To answer it, the author re-runs two open 7B instruction-tuned models on paraphrased ARC questions and reports the accuracy gap between original and paraphrased conditions.

Key Contributions

1. A paraphrase stress-test protocol for measuring generalization: each multiple-choice item is evaluated once on the original question and once on a semantically equivalent paraphrase, and the paired accuracy gap (Delta) is reported.
2. A paraphrase-cleaning pipeline designed to preserve semantics while stripping formatting artifacts, with automatic fidelity checks, retries (up to K = 3), and fallback to the original stem if a rewrite fails validation.
3. A fully specified experimental setup — ARC-Easy and ARC-Challenge validation splits, Mistral-7B-Instruct and Qwen2.5-7B-Instruct in cross-paired answerer/paraphraser roles, 4-bit inference on a single A100 — with released code for reproducibility.
4. Empirical evidence of consistent accuracy drops under paraphrasing, consistent with residual memorization or brittle surface-form shortcuts.

Main Findings

• Paraphrasing consistently lowers accuracy across both ARC splits and both cross-pairings. Drops range from 0.06 to 0.10, confirming that meaning-preserving surface-form changes measurably degrade performance.
• The largest drop occurs for Mistral-7B-Instruct answering ARC-Easy. Delta = 0.10 (0.84 original vs. 0.74 paraphrased), suggesting even "easier" items are fragile under rewording.
• ARC-Challenge drops are more moderate. Both models show Delta = 0.06 to 0.07 on ARC-Challenge.
• Qwen2.5-7B-Instruct has the higher baseline when answering. It reaches 0.90 on ARC-Easy and 0.89 on ARC-Challenge, versus 0.84 and 0.75 respectively for Mistral-7B-Instruct.
• Brittleness is not confined to one model family. Despite different baseline accuracies, both answerers show paraphrase sensitivity in the same 0.06 to 0.10 band.
• Errors move in both directions. Two categories of flips are observed: original-to-incorrect changes, where a paraphrase induces an error despite a correct original response, and incorrect-to-correct changes, where paraphrasing helps the model recover the right answer. The former dominates, but the latter occurs in a nontrivial minority of cases, showing paraphrasing is not uniform noise.
• No explicit contamination audit was performed. The paper states this directly and notes that methods such as n-gram overlap checks or retrieval-based similarity could strengthen causal attribution in future work.

Full table as reported (Answerer / Paraphraser / Dataset / n / Acc orig / Acc para / Delta):

Methodology in Plain English

The author takes multiple-choice science questions from ARC. Only the question stem is rewritten; the answer options are kept verbatim so the answer key still maps to the same letters. Each item is then answered twice by the same model — once with the original wording and once with the paraphrase — and the two accuracies are compared.

To avoid a model editing its own homework, the paraphraser is always a different model family from the answerer, and both cross-pairings are reported: Qwen answers while Mistral paraphrases, and Mistral answers while Qwen paraphrases. The paraphrasing model is instructed to rewrite the stem only, preserve all details verbatim (numbers, units, entities), and output nothing but the rewritten stem.

Every paraphrase passes through automatic checks: it must be non-empty after cleaning, must differ from the original stem case-insensitively, and must be at least max(10, 0.6 × original length) characters. If a check fails, the rewrite is retried up to three times; if it still fails, the original stem is used so the item count stays fixed.

The answerer is forced into a strict format: a single-letter decision, or a one-object JSON with an answer letter and a one-to-two sentence explanation when force_letter is enabled. A strict parser prefers the JSON key, then looks for a leading "Answer: LETTER" pattern, and rejects spurious matches such as grabbing the first uppercase letter it encounters (for example, the "A" in "Answer: C). Scoring is exact letter agreement with the gold key.

For fairness, answering is fully deterministic (temperature 0.0, top_p 1.0, no sampling, 32 max new tokens), while paraphrasing uses light sampling (temperature 0.7, top_p 0.95, 128 max new tokens) to avoid trivial restatements. Everything runs on a single NVIDIA A100-SXM4-80GB with CUDA 12.4 and driver 550.54.15, using 4-bit NF4 quantization via bitsandbytes with double quantization and bfloat16 compute. Randomness is fixed with random.seed(1337) and torch.manual_seed(1337), and the dataset subsets (300 items for ARC-Easy, 299 for ARC-Challenge) are selected deterministically with a fixed random seed. No few-shot examples, chain-of-thought, tools, or retrieval are used.

Why This Matters

If a model's score can drop by 6 to 10 points simply because a question was reworded, then a headline benchmark number may say less about reasoning ability than it appears to. The paper argues that strong scores may partly reflect memorization of familiar strings or reliance on brittle surface patterns, and it makes the case for paraphrase-aware evaluation as a routine part of the LLM lifecycle.

Real-world applications:

• Benchmark design and leaderboards: benchmark authors can add paraphrased variants to reduce the value of memorized or leaked items, and to separate genuine generalization from surface matching.
• Model selection for deployment: teams choosing between models can weigh paraphrase stability, not just headline accuracy — a model that holds steady under rewording may be a safer pick than a slightly higher-scoring but brittle one.
• Prompt and template engineering: because instruction templates can inadvertently favor one phrasing style, understanding format sensitivity helps teams write more robust prompts and output constraints.
• Evaluation of high-stakes QA systems: the same stress-test logic applies to any setting where a user's wording will differ from the wording the system was tested on.

Industry relevance: paraphrase robustness is directly relevant to production systems where real user inputs never match the phrasing of test items. The paper's finding that brittleness is a shared vulnerability across two different model families suggests this is a general property of current instruction-tuned models rather than a quirk of one vendor, which matters for anyone treating public benchmark scores as a proxy for deployed reliability.

Future Directions

1. Explicit contamination auditing. The paper does not perform one, and suggests n-gram overlap checks or retrieval-based similarity to establish whether high original-phrasing accuracy reflects memorization or reasoning.
2. Higher-fidelity paraphrasing. Because an LLM generates the paraphrases, semantic drift occurs and is a limiting factor; the paper suggests human-in-the-loop filtering, specialized paraphrasing models, or multi-pass verification to ensure measured drops reflect answerer brittleness rather than paraphrase artifacts.
3. Disentangling answerer and paraphraser effects. The cross-family design prevents stylistic self-imitation but means results reflect an interaction of two models; future protocols could evaluate answerers against a fixed, high-fidelity paraphrase set instead.
4. Extending beyond ARC. ARC targets grade-school science; the paper flags open-domain QA, multi-turn dialogue, and reasoning-intensive benchmarks as untested, and also notes that rigid letter-only formatting may itself cause some errors independent of the model's underlying knowledge.

Target Audience

This paper suits LLM evaluation researchers and practitioners — especially those building benchmarks or leaderboards — along with applied machine learning engineers who need to judge whether a benchmark score will hold up under real user phrasing. It is also a good entry point for students or newcomers to evaluation methodology, since the protocol is simple, the full pipeline is specified, and the code is released. Readers looking for large-scale results or statistical rigor should note the small scale: two 7B models, one benchmark family, fixed subsets of 300 and 299 items, and no significance testing or confidence intervals reported.

Authors’ abstract

Benchmark scores for Large Language Models (LLMs) can be inflated by memorization of test items or near duplicates. We present a simple, protocol that probes generalization by re-evaluating models on paraphrased versions of benchmark questions. Using Mistral-7B-Instruct and Qwen2.5-7B-Instruct, we measure the accuracy gap between original and paraphrased items on ARC-Easy and ARC-Challenge. Our pipeline controls decoding, enforces multiple-choice output format, and includes a robust paraphrase-cleaning step to preserve semantics. We find that paraphrasing induces a non-trivial accuracy drop (original vs. paraphrased), consistent with prior concerns about contamination and brittle surface-form shortcuts.