Token-Set Choice Confounds POPE: A Systematic Audit of Yes/No Extraction in VLM Hallucination Evaluation

AI summary

Overview

Research area: Computer vision / vision-language model (VLM) evaluation methodology, specifically hallucination benchmarking on POPE and the token-level readout used to turn model logits into yes/no answers.

Technical level: Intermediate. Readers need some familiarity with tokenizers, logits, greedy decoding, and F1 scoring, but the core argument is arithmetic rather than mathematical.

Scope: A systematic audit of how the yes/no answer is extracted from LLaVA-1.5-7B on the POPE hallucination benchmark, showing that a two-token logit readout silently inflates or deflates F1, plus a four-readout replication across four further VLMs.

What This Paper Is About

The authors set out to reproduce the LLaVA-1.5-7B baseline on POPE so they could benchmark a hallucination-mitigation idea of their own, and measured 82.21 F1 on the adversarial split. While inspecting the evaluation code, they found that the standard-looking readout compares two vocabulary IDs (3582 for "yes", 1217 for "no") that the model never actually generates after the ASSISTANT: prompt suffix. The paper's goal is to document this "candidate exclusion" failure, quantify how much F1 it moves on its own, and check whether the problem is specific to one model or a protocol-level issue across tokenizer families.

Key Contributions

1. A documented token-extraction artifact. Across 9,000 greedy-decode questions spanning all three POPE splits, LLaVA-1.5-7B never once emits token 3582 or 1217; it emits the space-prefixed forms 3869 (" Yes") and 1939 (" No"), because the LLaMA-2 SentencePiece tokenizer prepends a whitespace byte after the ASSISTANT: suffix. The two-token rule therefore compares two candidates the decoder excludes.

2. A quantified protocol gap with a closed-form account. The eight-token readout scores F1 0.8221, 0.8498, and 0.8713 on the adversarial, popular, and random splits versus 0.7608, 0.7961, and 0.8397 under the two-token rule — a 6.13-point adversarial shift. The authors derive this gap as an accounting identity from two empirically counted transition rates per split.

3. A comparability audit of published POPE results. The paper tabulates baselines, results, and gains for LLaVA-1.5-7B from six surveyed papers, reports that none states its readout, and explicitly declines to compute corrected gains because its own reference ran under 4-bit NF4 quantization.

4. A multi-model, multi-readout validation and a drop-in fix. The audit is repeated on four additional VLMs spanning three tokenizer families under four readouts, and the paper releases a tokenizer-derived readout (Listing 1) plus 9,000 prediction records and a diagnostic script.

Main Findings

• The two-token readout reads tokens that never occur. Token 3582 appeared zero times and token 1217 zero times across all 9,000 questions. Only 3869 (" Yes") and 1939 (" No") carry first-token mass: 3,893 and 5,107 questions respectively, split as 1,356/1,644 (adversarial), 1,303/1,697 (popular), and 1,234/1,766 (random).

• The failure inflates recall through a yes-bias. On the 1,644 adversarial questions the model answers " No", the 3582 logit beats the 1217 logit on 924 of them (56%). Of those, 660 land on true-negative questions and become false positives; 264 land on borderline label-yes questions and become true positives. The yes-rate climbs to 0.760 on a 50/50 split, recall to 0.9587, and F1 sinks to 0.7608, with precision falling to 0.6307.

• The corrected readout restores calibration. Under the eight-token rule the adversarial yes-rate is 0.452, precision 0.8658, recall 0.7827, F1 0.8221. Popular and random behave the same way (0.8498 and 0.8713), with gaps of +0.0537 and +0.0316.

• The gap is an accounting identity, not an independent mechanism test. Fixing T = 1,174, G = 1,318, ρ_N = 660/1,318 = 0.501, and φ = 264/326 = 0.810 reproduces both F1 values exactly. The same rates applied across splits give 0.7690 and 0.7761, well off the observed popular and random values, so rates must be counted per split. The records hold only the sign of ℓ_3582 − ℓ_1217, not the logit values, so no magnitudes, means, or ranks are reported.

• The shift rivals published gains. The 6.13-point adversarial shift exceeds the adversarial gain reported by five of the six surveyed papers. Tang et al. report +10.81 (73.54 → 84.35), +10.56 (76.53 → 87.09), and +8.01 (81.54 → 89.55); Seo et al. +0.70 (79.30 → 80.00); Chen et al. +0.80 (81.00 → 81.80); Huo et al. +1.60 (80.90 → 82.50); Jiang et al. +5.76 (75.06 → 80.82); Jana et al. +0.96 (82.07 → 83.03). Adversarial baselines on one frozen checkpoint spread from 73.54 to 82.07, a range of 8.53 points.

• The confound is protocol-level. Porting the LLaVA-1.5 IDs verbatim scores F1 0.00 on LLaVA-1.6-Mistral-7B and Qwen2-VL-7B-Instruct, whose vocabularies put no yes/no mass on those IDs. On the LLaMA-family models (InstructBLIP-Vicuna-7B, mPLUG-Owl2-LLaMA2-7B) the fixed set still works. A tokenizer-derived single-token readout holds between 0.80 and 0.92 on every model and split.

• String parsing can fail for a different reason. string_parse matches dynamic_single to four decimals on LLaVA-1.6-Mistral, mPLUG-Owl2, and Qwen2-VL, but collapses to the 0.6667 all-yes floor on InstructBLIP-Vicuna-7B because the decoded text did not begin with the answer token.

• Nine of the authors' own corrections failed. Six grounding-based and three contrastive inference-time corrections were run on the adversarial split; every one finished below 0.8221.

• Wrong predictions attend more to the image. In a 3,000-question probe, incorrect answers draw more image attention than correct ones (Cohen's d = −0.46, p = 1.4 × 10⁻²⁴), the opposite of what grounding-based corrections assume. This is presented as a secondary finding.

Methodology in Plain English

The authors ran LLaVA-1.5-7B in 4-bit NF4 quantization via BitsAndBytes on a single NVIDIA RTX 3070 Ti with 8 GB VRAM, using transformers v4.40.1 and torch.manual_seed(42). Each of the 9,000 POPE questions was decoded greedily with max_new_tokens=1 and output_scores=True, and the argmax token ID was logged alongside the decision each readout would have made. The full sweep took 76.3 minutes of wall clock time, 0.508 seconds per question.

They then compared readouts on identical predictions. The two-token protocol asks whether ℓ_3582 > ℓ_1217. The eight-token protocol takes the max logit over the four yes forms {3582, 8241, 4874, 3869} and compares it to the max over the four no forms {1217, 3782, 694, 1939}; the max aggregator is chosen because it matches what a greedy decoder actually does. They derived a closed-form expression for the F1 gap in terms of two per-split transition rates, then checked whether the arithmetic reproduces every confusion-matrix cell.

For the multi-model audit they applied four readouts — legacy_2tok, legacy_8tok (both hard-coding the LLaVA-1.5 IDs), dynamic_single (IDs re-derived from each model's own tokenizer), and string_parse (greedy decode plus first yes/no substring) — to 9,000 questions per model on four further VLMs. LLaVA-1.6-Mistral-7B ran locally on an RTX 3070 Ti Laptop under 4-bit NF4 with transformers v4.44.2; InstructBLIP-Vicuna-7B, mPLUG-Owl2-LLaMA2-7B, and Qwen2-VL-7B-Instruct ran on Kaggle Tesla T4 GPUs. Qwen2-VL used its native chat template; the other three used the LLaVA-1.5 template.

A separate instrumented pass over the adversarial split, run with eager attention so attention maps could be returned, saved the maximum logit within the eight-token yes set and within the no set (averaging 24.63 and 24.53 respectively). It matched the main run's eight-token decision on 2,998 of 3,000 questions, differing on question IDs 77 and 2938, both fp16 ties or near-ties.

Why This Matters

Impact on research. A single-line evaluation choice moves POPE adversarial F1 by 6.13 points, more than the reported gain of five of six surveyed papers — and none of those papers discloses its readout. Because no per-question logs are public and the authors' own reference is 4-bit NF4, the paper does not claim any published method is broken or over-credited. It claims that cross-paper comparisons cannot be interpreted until readout and precision are matched and reported. The authors note that Tang et al. (arXiv:2506.12609) had no public implementation as of June 2026.

Real-world applications:

• Clinical or safety-critical VLM screening, where an inflated recall on "is the object present?" style questions could mask a model that answers yes too often.
• Benchmark and leaderboard maintainers, who can adopt the recommendation to log the generated token ID and its decoded string for every question.
• Model-selection decisions in deployment, where a vendor's reported hallucination score may reflect a readout convention rather than model quality.
• Reproduction studies and artifact review, since releasing prediction-level records lets reviewers recompute any aggregate and check the readout independently.

Industry relevance. The proposed fix is a one-line change in evaluation code with no new generation cost beyond encoding four short strings once at setup. It runs unchanged across tokenizer families, which matters for teams evaluating multiple vendors' VLMs inside one harness.

Future Directions

• Rerun the audit at higher precision. The main 9,000-question sweep uses one checkpoint at 4-bit NF4, so whether the 6.13-point shift holds at higher precision is not reported.
• Matched comparisons against published methods. Each surveyed method would need a rerun with readout and precision fixed and logged before any corrected gain can be estimated.
• Extend the corrections beyond the adversarial split. The nine inference-time corrections were run only on adversarial; the authors suggest that beating the corrected baseline may require training-time rather than decoding-time changes, given the attention probe's finding.
• Measure the excluded logits directly. The records hold only the sign of ℓ_3582 − ℓ_1217, so how strongly those logits track the model's lean toward yes remains unknown; a run that saves the values would test the proposed mechanism independently.

Target Audience

Benchmark maintainers and evaluation engineers for vision-language models; researchers comparing hallucination-mitigation methods on POPE; anyone building a VLM evaluation harness across multiple tokenizer families; and reviewers or reproducibility auditors who need to know which measurement details are load-bearing. Practitioners who only consume published leaderboard numbers will find the practitioner-recommendations section and Listing 1 immediately actionable.

Authors’ abstract

We set out to reproduce a recent hallucination-mitigation result on the POPE benchmark. The LLaVA-1.5-7B baseline we measured, 82.21 F1 on the adversarial split, sat 1.21 to 8.67 points above the adversarial baselines we planned to compare against (81.00, 79.30, and 73.54). We looked at the evaluation code and found a readout choice that moves F1 on its own. A two-token logit readout decides yes versus no by comparing two vocabulary IDs, 3582 ("yes") and 1217 ("no"). Across 9,000 greedy-decode questions spanning all three splits, neither ID is generated even once. The model emits " Yes" (token 3869) and " No" (token 1939), because SentencePiece prepends a space byte after the prompt suffix "ASSISTANT:". Read with an eight-token lookup that covers the space-prefixed surface forms, LLaVA-1.5-7B scores F1 0.8221, 0.8498, and 0.8713 on the adversarial, popular, and random splits. That 6.13-point adversarial shift exceeds the adversarial gain reported by five of the six papers we surveyed. None of them states its readout, and our reference ran under 4-bit NF4 quantization, so we cannot say how much of any reported gain a matched evaluation would keep. That comparison stays unresolved until readout and precision are matched. On the fixed baseline we run nine inference-time corrections of our own (none beat it) and dissect why the adversarial split resists logit-level intervention. In a 3,000-question probe, wrong predictions draw more image attention than correct ones (Cohen's $d=-0.46$, $p=1.4\times10^{-24}$), the opposite of what grounding-based corrections assume. A four-readout audit across four further VLMs from three tokenizer families confirms the confound is protocol-level, not model-specific: porting the LLaVA-1.5 IDs verbatim drops F1 to 0.00 on the two disjoint-vocabulary models, whereas a tokenizer-derived readout holds between 0.80 and 0.92.