# Audit Finds Benchmark Graders Shift Scores by 22 Points

Published 2026-11-04 · AI.info Team · AI.info

> A study of multimodal AI benchmarks finds that grading rules can change accuracy scores by up to 22 percentage points on identical model outputs. The authors recommend reporting non-compliance rates and results under at least two scoring rules.

## Identical answers, different grades

A multimodal benchmark’s grading rule can shift a model’s reported accuracy by as much as 22 percentage points without changing any of its answers. Ahmad Jarjis Hasan and Jonathan Lwowski submitted their study to [arXiv](https://arxiv.org/abs/2611.03377) on Nov. 2, 2026, arguing that benchmark scores can depend on how software interprets a response as much as on what the model says. Their audit covers eight peer-reviewed benchmarks and five open vision-language models that can run on a single 16 GB GPU.

The difference emerges when a model gives free-form text instead of a clean option letter. A benchmark must decide whether to reject that response, guess which answer it resembles, or use another fallback; the study finds those choices are often buried in scoring code. “The grade came from the grader, not the model,” the authors write.

## MMAD’s 22-point scoring gap

The largest measured gap comes from MMAD, a binary benchmark. Its closest-option rule assigns an unparseable response to the textually nearest answer, while an abstaining rule marks it wrong. On the same set of 1,670 responses, those rules produce a 22.0-point difference for Llama-3.2-11B and a 21.2-point difference for Pixtral-12B.

The disagreement does not appear when models provide clean option letters: the paper reports a zero-point change for those outputs. But that makes the issue harder to see in leaderboards built around models that follow the requested format. In the authors’ tests, the fallback rule—not a change in the model’s output—accounts for the difference.

## Why neither fallback is a safe default

The authors checked graders against human judgments of what a model actually answered. Their blinded process paired human labeling with an LLM judge; the judge agreed with the human label on 127 of 130 MMAD items, with Cohen’s kappa of 0.956. The protocol distinguishes an answer the model committed to from a response that merely mentions an option or never answers the question.

That check exposes errors in both directions. Among 110 non-compliant MMAD responses reviewed, the models committed to an option only 23% of the time, yet the closest-option rule scored 72% as correct. Abstention can also mark a response wrong when a model states the correct answer in words rather than selecting its letter.

## The parser can matter even when a model complies

The paper also identifies a separate source of score variation: which matching answer a parser keeps when a response contains more than one. In MMAD’s released harnesses, eleven keep the last matching letter and two keep the first; the authors report that this choice shifts a reasoning model’s score by 9.5 points even with a non-compliance rate of 0.000.

The result complicates a simple fix based only on asking models to follow answer formats. A model may comply with the format and still receive a different grade because its parser reads a different part of its response. The authors therefore distinguish fallback choices from parser behavior, and call for both to be reported clearly.

## What the audit does—and does not—establish

The study’s strongest human-checked evidence comes from binary MMAD, and its experiments cover five open models in the 7–12 billion parameter range, run in 4-bit form on a 16 GB GPU. The authors say they did not measure proprietary or much larger models, so the reported effects should not be read as a measurement of every multimodal system. Their results concern the tested scorers and outputs.

For benchmark developers, the proposed reporting change is concrete: publish a non-compliance rate and scores under at least two grading rules rather than presenting one unexamined grade. The authors recommend showing results under both abstention and closest-option scoring, while treating the spread as evidence of rule sensitivity rather than a definitive accuracy range. The paper’s central finding is that one response can produce different benchmark scores depending on the grader that reads it.

## Sources

1. Original source · Preprint [Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks](https://arxiv.org/abs/2611.03377) arXiv:2611.03377

## Related stories

- [BEAKER Preprint Finds Multimodal Models Struggle to Ground Actions in Chemical Labs](https://ai.info/news/beaker-benchmark-self-driving-chemical-labs)
- [Google launches benchmark for visual knowledge-work agents](https://ai.info/news/google-launches-visual-knowledge-work-agent-benchmark)
- [NVIDIA’s 120B Nemotron 3.5 Super VL is available on SageMaker](https://ai.info/news/nvidia-nemotron-35-super-vl-sagemaker-jumpstart)
- [New risk benchmark finds newer AI models tend toward neutrality](https://ai.info/news/ai-risk-behavior-benchmark-language-models)
- [Meituan Releases LongCat-2.5, a 1.6T-Parameter Agent Model](https://ai.info/news/meituan-releases-longcat-2-5-1-6t-agent-model)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/audit-benchmark-graders-shift-multimodal-scores-22-points
