Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks

AI summary

Overview

Research area: Natural Language Processing — benchmark validity, evaluation methodology, and shortcut learning in multiple-choice question answering, with a focus on legal domain benchmarks and Ukrainian-language resources.

Technical level: Intermediate. The statistical machinery (binomial tail tests, position-bias correction, correlation analysis) is explained in the paper, but readers benefit from familiarity with LLM evaluation practice.

Scope: The paper measures how much of a legal multiple-choice benchmark can be answered from the answer options alone, with the question hidden, and tests whether filtering out such items repairs the benchmark.

Note on sourcing: the paper text supplied here ends mid-way through Section 6.3, so results attributed to Sections 7 and 8 (the LEXam comparison and the distractor-rewriting variant) are taken from the abstract, introduction, and contribution list.

What This Paper Is About

Multiple-choice benchmarks are scored on whether a model picks the correct option, but that score is only meaningful if the question was actually needed to answer. The paper measures that gap on a real, state-published Ukrainian judicial examination bank, UA-JudgeExam: 11,990 four-option items with official keys. Its core question is whether the obvious fix — deleting the items a model can answer without the question — produces a benchmark that is clean for other models too.

Key Contributions

1. Filtering does not transfer. Dropping the items that the gating model could guess retains 67.8% of the bank and drives that same model to 0.204 blind on what survives, below chance — yet GPT-5.6, which took no part in the selection, still answers 0.515 of those items with the question hidden. Rewriting distractors instead overshoots to 0.168, below chance and equally exploitable.

2. A measurement protocol with an estimator that separates habit from content. The blind condition is run over option-order permutations, with the selecting model held separate from the reporting ones. Forcing the key into each slot in turn makes the four accuracies sum to one whenever choice is independent of content, so their mean is chance by construction and any excess is content.

3. A benchmark that does not leak, and an explanation of why. The same probe on LEXam returns chance; every one of its options points into the stem and none is longer than 33 characters. Option format, not subject matter, decides whether the problem can arise.

4. A released resource. UA-JudgeExam, with 11,990 items carrying official state-issued keys, extraction verified by an independent second path, plus the gated 8,128-item subset and every prediction behind the tables.

Main Findings

• Blind solvability is large and concentrated. Shown the options and no question, Claude Haiku 4.5 scores 0.383 against a chance rate of 0.250. The leak is not diffuse: 3,517 items (29.3%) are never answered blind across eight option orders, while 1,419 (11.8%) are answered blind all eight times — an event with probability 1.5×10⁻⁵ under chance, where chance predicts 0.2 items.

• The leak is plausibility, not quotation. Lexical search over 280,059 current editions of Ukrainian legislation (2.356 billion characters) recovers only 0.128 overall on the full bank and 0.172 on the 8,024 items with four substantive options — below chance in both cases. Blind accuracy does not concentrate on items whose key is verbatim statutory text: on the 199-item pilot, Haiku scores 0.433 on verbatim keys against 0.347 on the rest, and Sonnet 4.6 splits the other way at 0.366 against 0.414.

• Surface features carry almost nothing. The best trivial strategy is "most words" at 0.301; always answering A gives 0.289 and the longest option 0.278. The correct option is on average 2.6 characters longer than a distractor. The key sits at A in 28.9% of items, at B and C in 24.8% each, and at D in 21.5%.

• A gate built on one model leaves the strongest models reading the options as before. The gate retains 8,128 items (67.8%; retention by specialisation runs from 74.3% administrative down to 61.4% general) on which Haiku 4.5 itself scores 0.204 blind. On those same items GPT-5.6 scores 0.515 and Sonnet 4.6 scores 0.320. Corrected for each model's own answer-position habit, their excesses are +0.265 and +0.081 — the only two excesses in the twelve-model held-out set.

• Position bias explains nearly everything else. Forced-placement measurement shows the models are orthogonal: Sonnet 4.6 carries the largest slot preference (it picks the last slot 0.450 of the time and the first 0.077, giving a spread of 0.528 across key positions), while GPT-5.6 has almost none (spread 0.085) and leaks most. Nova Pro, which has a real preference, sums to 1.085 with a position-free estimate of 0.271 that does not separate from chance. Ten of twelve held-out models land within 0.015 of what their positional habit alone would earn.

• Without the habit correction the ranking misleads. Llama 3.1 8B scores 0.292 blind — above every model but GPT-5.6 and Sonnet 4.6 — purely because it answers A to 92% of items; its excess over its own habit is +0.005.

• The gate did find something real, for almost everyone. On the items it rejected, eleven of twelve held-out models score 0.518–0.789, and the accepted-versus-rejected gap is positive for all twelve. The selection also carries a signature of the model that made it: agreement with Haiku's blind pick runs 0.42–0.52 for ten of the twelve where independence would give 0.25, and that agreement correlates with the accepted-to-rejected gap at r = 0.892 over twelve models (r = 0.577 over the remaining eleven once Llama 3.1 8B is dropped).

• A 400-item sample hides both problems. At that size intervals are wide enough to cover chance and nine models read as "statistically at chance" — concealing both that nine of them extract nothing once habit is subtracted and that what they do extract is shaped by the gate's own model.

• Leakage tracks capability. Ranking the eleven held-out models by full-condition accuracy reproduces their blind ranking at Pearson r = 0.916 (95% CI [0.703, 0.978]); against an independent capability measure on disjoint items it is r = 0.813 (95% CI [0.415, 0.950]). Blind accuracy is a fixed fraction of full accuracy, 0.384–0.538 with mean 0.463, but for nine of the eleven that fraction describes where habit lands, not partial competence. The eleven models span full-condition accuracy from 0.436 to 0.958.

• Prompt phrasing and option labels are not doing the work. Across five prompt phrasings the within-model spread is 0.033–0.061, and the control model stays at chance under all five. Full-condition wording moves results by 0.005–0.037, and Cyrillic or numeric labels shift results by at most 0.027.

• Inference-time reasoning does not create the effect. Sonnet 4.6's blind score is 0.343 whether answering directly or with extended thinking on, and 0.388 with a step-by-step prompt (p = 0.139); Nova Pro goes from 0.255 to 0.228 (p = 0.284); DeepSeek R1 moves from 0.253 at a 2,048-token budget to 0.231 at 4,096.

Methodology in Plain English

The authors take a state-published bank of judicial examination questions — each with exactly four options and exactly one official key — and run a "blind" test: they show a model the four options with the question deleted and ask for a single letter. A model that knows nothing would be right a quarter of the time. They repeat this over eight random shuffles of the option order, so a single lucky placement cannot masquerade as knowledge.

To decide which items leak, they use a binomial threshold: five or more hits out of eight is unlikely by chance (P(X ≥ 5) = 0.027), and they additionally require at least six of the eight trials to have produced a usable letter. Importantly, the model doing the selecting is not the model being reported on — Claude Haiku 4.5 builds the gate, and every reported blind number comes from models that took no part in selection.

Because a model that just likes option A will look knowledgeable whenever the key happens to sit at A, the authors add a cleaner test: they force the key into each of the four positions in turn. If a model's choice is independent of the option's content, the four accuracies must sum to one, so their mean is exactly chance regardless of how lopsided the preference is. Anything above that is content.

They also rule out simpler explanations — searching 280,059 editions of Ukrainian legislation to see whether the key is simply copied from a statute, re-running the probe under five different prompt phrasings, relabelling options in Cyrillic and digits, and comparing direct answers against step-by-step prompting and extended thinking.

The corpus itself comes from five documents totalling 1,672 pages published by Ukraine's Higher Qualification Commission of Judges under decision No. 221/зп-24 of 15 July 2024. Extraction was verified by a second, independent parsing path sharing no code with the first, which confirmed the key in 0.9841 of items over the full bank; the 192 items (1.6%) failing at least one check are listed by identifier in the release.

Why This Matters

Impact on research. The paper argues that the standard repair for a leaky benchmark — filter out the items one model can guess — does not work, because a bank cleaned against one model is not clean for a better one. That is a direct challenge to a common practice, and it distinguishes this work from prior results where removing the leaking features left the task intact (as Watson et al. report on 33,158 UK Employment Tribunal claims): deleting a cue from every instance and deleting the instances that carry it are not the same operation. The paper also supplies a reusable estimator that separates positional habit from genuine content extraction, which prior position-bias work (Zheng et al. across 20 models; Pezeshkpour and Hruschka) measures differently and which is easy to confuse with leakage in a single-order experiment.

Real-world applications:

• Professional licensing and certification exams. Any exam where distractors are self-contained propositions — bar exams, medical boards, judicial examinations — may be measuring recognition of well-formed domain statements rather than the reasoning it claims to measure.
• Model selection and procurement. Buyers comparing models on public benchmarks may rank a model highly because it answers A to 92% of items (Llama 3.1 8B at 0.292 blind) rather than because it reads the options.
• Benchmark construction and audit. The forced-placement protocol gives benchmark maintainers a procedure for checking whether leakage exists before publication, and the LEXam result shows reference-style options ("i and iii", none longer than 33 characters) avoid the problem entirely.
• Contamination and dataset-release policy. The resource ships with verified extraction, a stranded subset, and every prediction behind the tables, giving others a reproducible target for re-testing as new models arrive.

Industry relevance. The finding that blind accuracy scales with capability implies a treadmill: filtering against the models available today will not hold against the models available later, and each round of filtering removes the items that discriminate best. For anyone running an evaluation suite, that is a maintenance cost and a validity risk, not a one-time cleanup.

Future Directions

• Test whether the capability–leakage relationship continues above the tested range. The eleven models span full-condition accuracy from 0.436 to 0.958, and the paper states the extrapolation as a conjecture rather than a result: if blind accuracy keeps tracking capability beyond that range, today's filter will not protect against tomorrow's models.
• Find a filtering procedure that transfers. The paper reports finding no version of the procedure that made the cleaned set neutral for a stronger model — an open problem rather than a solved one.
• Determine what property of an item set decides whether the problem arises. The paper points to option format (self-contained propositions versus pointers into the stem) rather than subject matter, which invites a systematic test across benchmark families.
• Establish whether human expert readers exploit the same signal. The paper explicitly notes it does not test human readers, so the claim that domain knowledge lets a reader recognise the one option that reads like real law remains untested.

Target Audience

Benchmark authors and evaluation researchers in NLP; practitioners who build or audit domain-specific multiple-choice evaluations, particularly in law, medicine, and professional licensing; teams comparing model capabilities where benchmark scores feed procurement or deployment decisions; and Ukrainian-language and legal-NLP researchers interested in the released UA-JudgeExam corpus.

Authors’ abstract

Multiple-choice benchmarks are graded on whether a model picks the right option, not on whether it needed the question. Measuring that gap takes care: a model answering A to most items scores above chance wherever the key sits at A, and reads as recognition when it is not. We measure it on UA-JudgeExam: 11,990 four-option items with official keys, published by Ukraine's Higher Qualification Commission of Judges. Shown the options and no question, Claude Haiku 4.5 scores 0.383 against chance, and the leak is concentrated: 11.8% of items are answered blind on all eight option orders, against 0.2 items expected by chance. It is not quotation: search over 280,059 editions of Ukrainian legislation recovers 0.128. Gating those out retains 8,128 items, on which the gating model itself now scores 0.204, and GPT-5.6, which took no part in the selection, still answers 0.515 of them with the question hidden. Scoring twelve held-out models on the whole set and subtracting each one's answer-position habit, only two keep an excess: GPT-5.6 at +0.265, Sonnet 4.6 at +0.081. Without it the ranking misleads: Llama 3.1 8B scores 0.292 blind, above every model but those two, purely by answering A to 92% of items. The gate does select something real: on the items it rejected, eleven of twelve models score 0.518-0.789, every interval clear of what the same model scores on the items it kept. But that signal is one model's, and filtering on it does not transfer upward. Neither is visible on a 400-item sample, where nine models read as "statistically at chance". Rewriting distractors instead overshoots to 0.168, below chance and as exploitable. The same probe on LEXam returns chance: every option there points into the stem, none longer than 33 characters. Item format decides whether the problem can arise; capability decides how much is extracted. We release the corpus, the predictions and the harness.