BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks

AI summary

Overview

Research area: Natural language processing evaluation — specifically the quality of multiple-choice question answering (MCQA) benchmarks, LLM-as-a-judge tooling, and the transfer of assessment-design principles from education research into NLP.

Technical level: Intermediate. The paper assumes familiarity with benchmark evaluation and LLM prompting, but its central ideas (contamination, shortcuts, writing errors) are explained in plain terms and illustrated with examples.

Scope: The paper introduces BenchMarker, an LLM-judge toolkit that scores individual multiple-choice questions for contamination, shortcuts, and writing errors, validates it against 8042 human judgments, and uses it to audit 12 MCQA benchmarks and five revised benchmark versions.

What This Paper Is About

Multiple-choice question answering is a standard way to evaluate language models, but the benchmarks themselves are rarely checked for quality problems that have nothing to do with the skills being tested. The authors build a toolkit, inspired by how educators write and vet exam questions, that flags three specific flaws in individual MCQs: items that appear verbatim online (contamination), items solvable from the answer choices alone using shallow cues (shortcuts), and items that break any of 19 rules from an established education rubric (writing errors). They then use the toolkit to measure how common those flaws are, how they affect model accuracy and rankings, and whether existing "fixed" versions of benchmarks actually resolve them.

Key Contributions

1. BenchMarker, a toolkit using LLM judges to predict contamination, shortcuts, and writing errors in MCQs, wrapped in the InspectAI library with judge logs, standard prompts, and a UI to track runs.
2. A rigorous audit of 12 NLP benchmarks, showing that MCQA flaws are pervasive and that they measurably affect LLM accuracy and model rankings — particularly writing errors in crowdsourced and automatically generated MCQs.
3. An in-depth analysis of common writing flaws in both NLP and education MCQs, plus evidence that prior benchmark repair efforts do not fully address the problems they target and can introduce new ones.
4. A validation dataset of 8042 human judgments for testing how reliably LLM judges detect MCQ flaws.

Main Findings

• Flaws are pervasive across benchmarks. Running BenchMarker on up to 1000 sampled MCQs from the test sets of 12 benchmarks, the authors detect flaws in every dataset: 47% of TruthfulQA items appear online, 23% of SocialIQA items have shortcuts, and on average each HellaSwag item violates 44% of the 19 writing rules. The abstract states that 100% of HellaSwag violates multiple writing rules. The authors suspect 7 of the 12 datasets have over 90% of items with 2 or more writing violations, which education research treats as "unacceptable" (Tarrant et al., 2006).

• MCQs written by educators fare better on writing quality but are more contaminated. Grouping items by origin, MCQs from educator-written student exams contain far fewer writing flaws than those written automatically or by crowdworkers. However, annotation found many educator-written items online as study aids such as flashcards and tutor sites, so the authors suggest filtering sites linked to test preparation. Release date does not track contamination — older MCQs (HellaSwag) have a contamination score of 0 while more recent ones (TruthfulQA) can be higher.

• Choices-only accuracy overestimates shortcut prevalence. ARC, TruthfulQA, and OBQA show high choices-only accuracy, but often because models can reconstruct the original question rather than exploit a shallow cue. After filtering those cases, shortcut prevalence drops sharply — for example from 83% to 6% on TruthfulQA. The authors argue that citing choices-only success alone, without asking why models succeed, overstates dataset flaws.

• Contamination inflates accuracy; writing errors lower it. Across micro averages, LLM accuracy is 0.81 on contaminated splits versus 0.76 on uncontaminated splits. For splits with 2 or more writing errors, accuracy is 0.75 versus 0.83 for splits without them. Splits with shortcuts show similar or mixed accuracy after correcting for strategy (0.77 versus 0.77 micro). The authors conclude writing errors are the most critical flaw to fix, given both their accuracy impact and their prevalence.

• Writing errors shift model rankings; contamination and shortcuts do not. Using the full splits versus no-flaw splits across the 10 evaluated LLMs, ranks are identical for contamination and shortcuts (Spearman's ρ of 1.000), but writing errors shift ranks by up to two positions, exceeding what random sampling produces (confirmed via permutation tests at α = 0.01).

• Five writing flaws dominate. In both student-based and non-student-based MCQs, the most common BenchMarker-flagged flaws are ambiguous question stems, indirectly asking questions, unclear language, grammatical inconsistency in questions and choices, and implausible distractors.

• Benchmark repairs fix their targets but add new flaws. GoldenSwag and MMLU-Redux fix labels and raise accuracy, while MMLU-Pro lowers accuracy through LLM-written distractors. But GoldenSwag reduces multiple-correct-answer items (18% to 4%) and choice-length differences (52% to 43%) while adding grammar inconsistencies (68% to 79%). MMLU-Pro adds less plausible distractors (7% to 17%) and distractors that are correct when they should be incorrect (10% to 22%).

• Simple prompting does not fix writing errors. Prompting GPT-5.2, Claude Sonnet, and Qwen 235B to rewrite flawed TruthfulQA items given the detected errors reduces but does not eliminate problems — the proportion of items with 2 or more writing errors goes from 0.97 to 0.64 for GPT-5.2 and Claude Sonnet, and to 0.68 for Qwen 235B, while new errors appear in 0.56, 0.68, and 0.62 of items respectively.

• LLM judges agree with humans well enough to be useful. On shortcuts, GPT-5 reaches accuracy 0.82, F1 0.75, and Cohen's κ 0.61. On in-domain (NLP) writing errors, Gemini 2.5 Pro reaches 0.82 accuracy with κ 0.53, and GPT-5 reaches 0.81 with κ 0.50, both above the SAQUET baseline's in-domain 0.78 accuracy and 0.28 κ. On out-of-domain student exams, SAQUET reaches 0.93 accuracy with κ 0.48. For contamination, Google plus GPT-5 is the best combination at 0.71 accuracy, 0.68 F1, and 0.44 κ, with 0.86 precision.

• Recommended judges balance quality and cost. Gemini 2.5 Flash is only 0.05 below Gemini 2.5 Pro in Cohen's κ while costing roughly one quarter as much; Cohere Command-R is the best open-weight option for writing-flaw detection.

Methodology in Plain English

The authors start from education research rather than from NLP conventions. Educators already have decades of practice detecting bad test questions, so the team translates three of those concerns into automatable checks.

For contamination, they send the question stem and the gold answer to a web search API, then ask an LLM to judge whether the item appears exactly or nearly exactly in the returned pages. They deliberately do not flag items where the web only contains the underlying knowledge, since that would not permit exact memorization of the answer.

For shortcuts, they show models only the answer choices and have them both pick an answer and guess what the original question was. Three strong models — GPT-5, Gemini 2.5 Pro, and Claude 4.5 Sonnet — vote on this. An item is flagged only if the models can answer from choices alone and the question they infer does not match the real one. This separates harmful shortcuts (like spotting the "odd one out") from benign cases where the model effectively recovers the intended question.

For writing errors, they adopt the 19-rule Item-Writing Flaws rubric from Tarrant et al. (2006), which avoids subjective criteria. Each rule gets its own prompt containing the flaw's name, definition, and six examples — three flawed MCQs and three clean ones.

To check whether these LLM judges are trustworthy, the team built a validation set of 8042 human judgments: 229 MCQs labeled for contamination, 271 for shortcuts, 3419 for writing errors in NLP data, and 4123 human-written MCQs from higher education exams with existing writing-error labels. Two authors with annotation experience labeled items, with the second labeling 50 random items per metric (950 labels total for the 19 writing rules), achieving over 80% agreement. They also compared 23 LLMs across seven model families, six web search APIs, three trivial baselines (random, always-flawed, always-not-flawed), and the existing SAQUET toolkit.

Finally, they applied the validated tool to audit 12 benchmarks, measured how each flaw type relates to accuracy across 10 LLMs, tested whether removing flawed items changes model rankings versus random subsampling, and audited three published benchmark revisions plus a prompted-rewriting experiment.

Why This Matters

Impact on research. MCQA scores are used to claim progress, compare models, and select checkpoints for further training. If items are contaminated, solvable without the question, or grammatically broken, those scores partly reflect memorization, guessing, or confusion rather than the skills the benchmark claims to measure. The paper shows that removing writing errors alone changes model rankings by up to two positions, which means published comparisons could be affected by artifacts unrelated to model capability. It also gives researchers a repeatable, automated way to check whether their own benchmark fixes work as intended.

Real-world applications.

• Benchmark maintenance: dataset maintainers can run BenchMarker before release to catch items that appear online or violate writing rules, and can re-run it after edits to confirm no new flaws were introduced.
• Model evaluation and leaderboards: anyone reporting MCQA numbers can report no-flaw splits alongside full splits, showing whether conclusions hold up when flawed items are excluded.
• Educational assessment: the toolkit's writing-error judge performs competitively on educator-written exams, so it could help instructors screen their own test banks.
• Retrieval-augmented and web-connected systems: the contamination check reflects the realistic risk that a model can simply look up an answer rather than reason about it.

Industry relevance. Teams that ship models based on benchmark results have a direct stake in benchmark validity, because rankings drive deployment and training decisions. The paper's cost guidance matters too: Gemini 2.5 Flash is only 0.05 below Gemini 2.5 Pro in Cohen's κ at roughly a quarter of the cost, and Cohere Command-R is recommended for those without closed-source model access — making the toolkit practical for organizations of different sizes.

Future Directions

• Flaws outside individual items. BenchMarker deliberately detects flaws within MCQs only. Global concerns such as benchmark saturation and dataset diversity are left as future work, so a toolkit that scores those dimensions does not yet exist.
• Better automatic repair. Prompting GPT-5.2, Claude Sonnet, and Qwen 235B to fix detected writing errors reduced but never eliminated them, and introduced new errors in a majority of items. Novel rewriting approaches beyond simple prompting are needed.
• Multi-objective correction. Because fixing one flaw can add another — MMLU-Pro lowered accuracy but produced implausible and sometimes-correct distractors, and GoldenSwag removed multiple-answer items while adding grammar inconsistencies — correction should be iterative, with automated scoring runs tracking progress.
• Cross-field collaboration. NLP and education share the same salient writing flaws, especially unclear wording and distractor quality, and both fields are building solutions — text simplification on one side, LLM distractor generation on the other — suggesting joint tooling could advance both.
• Cheaper and open judge tuning. Open-weight models trail closed-source judges but remain competitive (Command R, Qwen 32B), so the authors suggest future work on tuning smaller models to close the gap for efficiency and reproducibility.

Target Audience

This paper is most useful to NLP researchers and engineers who build, maintain, or report on multiple-choice benchmarks; evaluation and leaderboard maintainers who need quality-control tooling; and industry practitioners who make training or deployment decisions based on MCQA scores. It is also relevant to educational assessment researchers interested in how LLM judges perform on educator-written exams, and to anyone studying LLM-as-a-judge reliability, since the validation set of 8042 human judgments and the cross-model comparison across 23 LLMs are directly applicable to that line of work.

Authors’ abstract

Multiple-choice question answering (MCQA) is standard in NLP, but benchmarks lack rigorous quality control. We present BenchMarker, an education-inspired toolkit using LLM judges to flag three common MCQ flaws: 1) contamination: items appearing exactly online; 2) shortcuts: cues in the choices that enable guessing; and 3) writing errors: structural/grammatical issues based on a 19-rule education rubric. We validate BenchMarker with human annotations, then run the tool to audit 12 benchmarks, revealing: 1) flaws persist in MCQA benchmarks, especially automatically-made and crowdsourced data - we detect 47% of TruthfulQA appears online and 100% of HellaSwag violates multiple writing rules; 2) contaminated MCQs tend to inflate accuracy, while writing errors tend to lower it and change rankings beyond random; and 3) prior benchmark repairs address their targeted issues (i.e., lowering accuracy with LLM-written distractors), but inadvertently add new flaws (i.e. implausible distractors, many correct answers). Overall, flaws in MCQs degrade NLP evaluation, but education research offers a path forward. We release BenchMarker to bridge the fields and improve MCQA benchmark design.