The Pulse
Audit Finds Benchmark Graders Shift Scores by 22 Points
A study of multimodal AI benchmarks finds that grading rules can change accuracy scores by up to 22 percentage points on identical model outputs. The authors recommend reporting non-compliance rates and results under at least two scoring ru
AI.info Team · Written with AI · fact-checked ·
Identical answers, different grades
A multimodal benchmark’s grading rule can shift a model’s reported accuracy by as much as 22 percentage points without changing any of its answers. Ahmad Jarjis Hasan and Jonathan Lwowski submitted their study to arXiv on Nov. 2, 2026, arguing that benchmark scores can depend on how software interprets a response as much as on what the model says. Their audit covers eight peer-reviewed benchmarks and five open vision-language models that can run on a single 16 GB GPU.
The difference emerges when a model gives free-form text instead of a clean option letter. A benchmark must decide whether to reject that response, guess which answer it resembles, or use another fallback; the study finds those choices are often buried in scoring code. “The grade came from the grader, not the model,” the authors write.
MMAD’s 22-point scoring gap
The largest measured gap comes from MMAD, a binary benchmark. Its closest-option rule assigns an unparseable response to the textually nearest answer, while an abstaining rule marks it wrong. On the same set of 1,670 responses, those rules produce a 22.0-point difference for Llama-3.2-11B and a 21.2-point difference for Pixtral-12B.
The disagreement does not appear when models provide clean option letters: the paper reports a zero-point change for those outputs. But that makes the issue harder to see in leaderboards built around models that follow the requested format. In the authors’ tests, the fallback rule—not a change in the model’s output—accounts for the difference.
Why neither fallback is a safe default
The authors checked graders against human judgments of what a model actually answered. Their blinded process paired human labeling with an LLM judge; the judge agreed with the human label on 127 of 130 MMAD items, with Cohen’s kappa of 0.956. The protocol distinguishes an answer the model committed to from a response that merely mentions an option or never answers the question.
That check exposes errors in both directions. Among 110 non-compliant MMAD responses reviewed, the models committed to an option only 23% of the time, yet the closest-option rule scored 72% as correct. Abstention can also mark a response wrong when a model states the correct answer in words rather than selecting its letter.
The parser can matter even when a model complies
The paper also identifies a separate source of score variation: which matching answer a parser keeps when a response contains more than one. In MMAD’s released harnesses, eleven keep the last matching letter and two keep the first; the authors report that this choice shifts a reasoning model’s score by 9.5 points even with a non-compliance rate of 0.000.
The result complicates a simple fix based only on asking models to follow answer formats. A model may comply with the format and still receive a different grade because its parser reads a different part of its response. The authors therefore distinguish fallback choices from parser behavior, and call for both to be reported clearly.
What the audit does—and does not—establish
The study’s strongest human-checked evidence comes from binary MMAD, and its experiments cover five open models in the 7–12 billion parameter range, run in 4-bit form on a 16 GB GPU. The authors say they did not measure proprietary or much larger models, so the reported effects should not be read as a measurement of every multimodal system. Their results concern the tested scorers and outputs.
For benchmark developers, the proposed reporting change is concrete: publish a non-compliance rate and scores under at least two grading rules rather than presenting one unexamined grade. The authors recommend showing results under both abstention and closest-option scoring, while treating the spread as evidence of rule sensitivity rather than a definitive accuracy range. The paper’s central finding is that one response can produce different benchmark scores depending on the grader that reads it.