Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage

AI summary

Overview

Research area: NLP evaluation methodology, specifically truthfulness benchmarks and benchmark contamination/shortcut learning. Technical level: Intermediate. This paper audits and repairs binary-choice truthfulness benchmarks that can be gamed by surface-level stylistic cues rather than actual reasoning about truth.

What This Paper Is About

Binary-choice benchmarks like TruthfulQA present a model with a correct answer and an incorrect answer and ask it to pick the true one. If the two answers differ systematically in superficial ways — length, hedging, or whether they contain negation — a model can score above chance without understanding anything. This paper shows that this leakage is present, measurable, and fixable, and it releases a cleaned version of TruthfulQA plus a general-purpose tool for repairing other datasets.

Key Contributions

1. Detection of serious leakage in TruthfulQA. A deliberately simple six-feature logistic classifier (Surface6) that never sees the question separates correct from incorrect answers at 0.689 accuracy and 0.715 AUC (grouped cross-validation), meaningfully above chance.
2. Audit-Prune, a general cleaning mechanism. A greedy, classifier-guided procedure that iteratively removes the pairs most reinforcing the surface asymmetry, then re-adds any pair that can be restored without breaching a target leakage threshold.
3. A cleaned dataset release, TruthfulQA-476. A 476-pair subset of binary-choice TruthfulQA with audit AUC reduced to 0.528 (p = 0.048), while preserving model-ranking agreement with the full benchmark (Spearman ρ = 0.915).
4. Cross-dataset extension. The same Surface6 audit is applied to 13 additional benchmarks, showing leakage is a general phenomenon rather than a TruthfulQA-specific quirk.

Main Findings

• Negation is the dominant leak. Dropping the negation-count feature lowers audit AUC from 0.715 to 0.641 — by far the largest per-feature effect. hedge_rate and word_count follow.
• Leakage exceeds leaderboard scores. The question-blind six-feature probe at 68.9% accuracy beats the top model on the llm-stats TruthfulQA leaderboard (66.9%) as of May 2026, though these measure different things.
• HaluEval QA is far leakier than TruthfulQA. Its audit AUC is 0.973. Hallucinated answers average 11.2 words versus 2.3 for accurate ones — a pure length artifact, inherited by HaluBench, which reuses its examples verbatim.
• Leakage is broad across benchmark families. MedHallu (0.821), MultiNLI (0.641), SNLI (0.591), FEVER 1.0, SelfCheckGPT, and MultiRC all show above-chance separation; BoolQ (0.525) and PIQA (0.509) sit near the detectability floor.
• Fixed-prefix removal fails where Audit-Prune succeeds. A baseline that scores all pairs once and removes in a fixed order cannot reach an AUC threshold of 0.58; Audit-Prune reaches 0.51 while keeping 401 pairs.
• Audit-Prune beats AFLite at matched sizes. At N = 476, AFLite's retained pairs still separate at AUC 0.603 versus 0.528 for Audit-Prune, with comparable ranking fidelity.
• Pruning is spread across topics. TruthfulQA-476 retains 60.3% of pairs overall (56.2% Adversarial, 64.9% Non-Adversarial), and every one of the 37 categories keeps at least one pair.
• Cleaned training data generalizes better under adversarial surface inversion. On a surface-flipped test set, classifiers trained on full TruthfulQA collapse (e.g., Llama-3.2-3B at 0.267 accuracy), while those trained on TruthfulQA-476 recover substantially (0.450), with McNemar p < 0.001.
• IRT-selected anchors do not solve the problem. tinyTruthfulQA's 95 IRT-selected anchors still leak at AUC 0.7018, essentially matching a random 100-pair subset.

Methodology in Plain English

The researchers start by defining six simple, interpretable properties computable from any answer string alone: negation count, whether negation leads the sentence, hedging rate, word count, average token length, and type-token ratio. They call this family Surface6. None of these require an embedding model, a neural network, or access to the question — only the answer text.

They then train an ordinary logistic regression on these six numbers to predict whether an answer is correct. Using grouped cross-validation (keeping both members of a question pair in the same fold), they measure how well this question-blind classifier separates true from false answers. They repeat this same audit across 14 benchmarks.

For cleaning, they formalize the goal as: find the largest subset of pairs whose grouped-CV audit AUC stays at or below a chosen threshold θ. Since exhaustive search is combinatorially infeasible, they use a greedy approach. Each pair gets a score combining (a) how strongly its within-pair feature differences point in the same direction as the dataset-wide class-mean difference, and (b) how heavily the logistic classifier relies on each feature. The highest-scoring pair is removed, the model and statistics are refit, and the process repeats until the threshold is met. A final add-back pass restores any removed pair whose reinsertion does not re-breach the threshold. They select θ = 0.53 for the released dataset.

To validate that cleaning does not destroy the benchmark's usefulness, they compare per-model accuracy rankings on the subset against the full benchmark across 14 open-weights models, and they construct two new 131-pair test sets — one adversarially surface-flipped, one naturally written — to check whether models trained on cleaned data actually learn truthfulness rather than style.

Why This Matters

Benchmark scores are used as evidence of model capability, including in safety claims. If a benchmark can be passed by counting negations, then published comparisons, leaderboard rankings, and procurement decisions may partly reflect stylistic artifacts rather than the reasoning they claim to measure. Training on such benchmarks could even improve scores without improving capability.

Real-world applications:

• AI safety and capability evaluation pipelines that currently rely on TruthfulQA as a standard truthfulness probe (lm-evaluation-harness, OpenCompass, and public leaderboards).
• Benchmark dataset construction, where pre-release surface audits could prevent leaked items and reduce the need for retroactive cleaning.
• Model procurement and regulatory review, where third-party claims of truthfulness need to survive adversarial scrutiny rather than surface-form heuristics.
• Hallucination detection research, particularly medical and QA settings where HaluEval-style length artifacts can inflate reported detector performance.

Industry relevance is direct: evaluators, benchmark maintainers, and safety teams at labs and audit firms all rely on datasets that this paper shows are partly gameable, and the released cleaned subset and tooling offer an immediate drop-in mitigation.

Future Directions

• Richer feature families. The authors note Surface6 is intentionally narrow; embedding-based or syntactic features may expose leakage this hand-designed set cannot see.
• Full-input audits. The probe only reads answers, not questions or context. Artifacts that appear only with the full prompt could go undetected, and the authors cite prior work warning that partial-input baselines miss these.
• Optimal subset selection. Finding the largest subset under a non-linear AUC constraint is an open combinatorial problem; Audit-Prune is greedy and not globally optimal.
• Auditing and repairing more benchmarks. Thirteen datasets were audited but only TruthfulQA was repaired and tested for downstream model impact; the pipeline could be extended to the leakier datasets such as HaluEval QA and MedHallu.
• Multilingual auditing. The released negation and hedge lexicons are English-only, so applying the audit in other languages requires language-specific feature sets.

Target Audience

NLP evaluation researchers and benchmark maintainers, AI safety and red-teaming practitioners, ML engineers selecting models or building evaluation harnesses, and anyone who reports or consumes LLM leaderboard scores. The paper is also accessible to graduate students entering the shortcut-learning and dataset-bias literature, since the core method relies on logistic regression and interpretable features rather than deep architecture knowledge.

Authors’ abstract

Binary-choice truth benchmarks ask models to choose between a correct and an incorrect answer, but if the two answers differ systematically in surface-level features, models can exceed chance without performing the intended reasoning. We show that this failure mode is detectable and can be exploited by downstream classifiers. In TruthfulQA, a simple six-feature logistic classifier achieves substantial accuracy in separating correct from incorrect answers. We further show that similar surface-level artifacts are present in additional benchmarks. To counteract this, we developed a general mechanism to clean them by removing the most leakage-reinforcing pairs. We release a version of TruthfulQA with surface-feature leakage reduced close to chance and provide a mechanism, Audit-Prune, so that the datasets can be cleaned before release.