DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioning

AI summary

Overview

Research area: Computer vision / multimodal evaluation — specifically automatic evaluation metrics for image captioning using large vision-language models (LVLMs).

Technical level: Advanced. The paper derives a closed-form analytical solution to a loss-minimization problem and assumes familiarity with KL divergence, softmax decoders, and autoregressive token distributions.

Scope (one sentence): The paper proposes a finetuning-free, test-time adaptive score decoder (DISCODE) that reshapes an LVLM's token probability distribution toward a Gaussian prior to produce caption evaluation scores that better match human judgments across six visual domains, and introduces a new six-domain evaluation benchmark (MCEval).

What This Paper Is About

Automatic image caption metrics are supposed to agree with human judgments, but current LVLM-based metrics struggle when images come from domains the models were not tuned for, such as paintings, sketches, and infographics. The authors argue this happens because LVLMs' output token probabilities are biased (for example, the digit 0 is over-predicted) and therefore are not unimodal, whereas human scores tend to follow a Gaussian shape. Their goal is to fix this mismatch at test time, without any finetuning, and to provide a benchmark that measures how robust evaluation metrics are across visual domains.

Key Contributions

1. DISCODE (Distribution-Aware Score Decoder): A finetuning-free test-time adaptive decoder head that takes the latent decoder feature from the last layer of an LVLM and outputs a score distribution by minimizing the Adaptive Test-Time (ATT) loss, which balances fitting the LVLM's token distribution against a Gaussian prior.
2. An analytical (closed-form) solution to the ATT loss minimization, available for the weighted KL divergence and the linear-plus-softmax head architecture, making the method computationally cheap at test time.
3. The MCEval benchmark: a human-annotated caption evaluation dataset covering six visual domains (real, painting, sketch, quickdraw, clipart, infograph) with 18,000 image-text pairs, built from 6,000 images, each with two candidate captions and one reference caption.
4. Extensive evaluation showing state-of-the-art results as a reference-free metric on MCEval and on four existing benchmarks (Flickr8k-Expert, Flickr8k-CF, Composite, Pascal-50S), plus generalization tests across LLMs, LVLMs, divergence measures, and rating scales.

Main Findings

• Best reference-free metric on MCEval: DISCODE reaches 83.6 mean accuracy across the six domains, ahead of the authors' re-implementation of FLEUR (82.5), G-VEval with GPT-4o (81.0), the original FLEUR (76.7), Polos (75.7), CLIP-S (75.3), PAC-S and BERT-S (69.1 each), CIDEr (64.6), METEOR (63.7), BLEU-4 (57.1) and ROUGE (55.6).
• Per-domain MCEval accuracy for DISCODE: 87.8 (real), 85.2 (painting), 83.9 (sketch), 78.5 (quickdraw), 83.5 (clipart), 82.8 (infograph). The largest gap versus FLEUR is on quickdraw, where FLEUR scores 45.6 and DISCODE scores 78.5.
• Competitive on real-image benchmarks: Using LLaVA-Next-72B, DISCODE-LV scores 55.7 / 56.1 (Flickr8k-Expert, Kendall tau-b / tau-c), 40.2 / 20.8 (Flickr8k-CF), 61.1 / 66.0 (Composite) and 84.5 accuracy (Pascal-50S). With InternVL-2.5-78B, DISCODE-IN scores 57.7 / 58.1, 40.1 / 20.8, 60.5 / 64.9 and 83.5 respectively.
• It beats a proprietary-LVLM metric on Flickr8k-CF: DISCODE's 40.2 tau-b exceeds G-VEval's 38.7 tau-b, even though G-VEval uses GPT-4o.
• Every component matters (ablation): Removing the cross-entropy term drops MCEval from 83.6 to 81.8; removing the divergence term drops it to 80.9; replacing the adaptive alpha with a fixed alpha = 0.5 gives 83.0.
• The 0.0–1.0 continuous scale is best: MCEval scores are 83.6 (0.0–1.0), 83.6 (1–5), 83.5 (0–9) and 83.5 (A–E), with the continuous scale slightly ahead on Flickr8k-Expert (56.1 tau-c versus 55.4 for the 1–5 scale and 55.4 for A–E).
• Consistent gains across LLMs and LVLMs: Improvements over raw scores and FLEUR are reported for Llama-3-8B, Vicuna-13B, Hermes-Yi-34B and Qwen-1.5-72B within LLaVA-Next, and for InternVL-2.5-8B/78B, Qwen2-VL-7B/72B-Instruct, CogVLM2-Chat-19B and MiniCPM-V-2.6. Among all results, the best numbers come from InternVL-2.5-78B on Flickr8k-Expert (58.1 tau-c), Qwen-VL-72B-Instruct on Composite (66.7 tau-c) and MCEval (83.8%), and LLaVA-Next-Qwen-1.5-72B on Flickr8k-CF (40.2 tau-b) and Pascal-50S (84.5%).
• Weighted KL divergence is the best divergence choice: MCEval 83.6 for weighted KLD, versus 83.0 for Rényi, 83.0 for standard KL, 81.6 for Jensen–Shannon, 81.5 for beta divergence, and 80.9 with no divergence term.
• Low computational overhead: With the analytical solution, scoring overhead is 0.73% (Llama-3-8B), 0.79% (Vicuna-13B), 0.66% (Hermes-Yi-34B) and 0.11% (Qwen-1.5-72B), compared with 9.46%, 11.5%, 8.13% and 1.90% when the same loss is minimized numerically without the analytical solution.
• Works for reference-based evaluation too: RefDISCODE-LV reaches 87.8 mean accuracy on Pascal-50S and 40.4 tau-b on Flickr8k-CF; RefDISCODE-IN reaches 86.3 on Pascal-50S and 40.5 tau-b on Flickr8k-CF.
• Variance setting is not sensitive: Values of sigma-squared from 0.01 to 1.0 produce nearly identical results, with the default 0.1 giving the best reported Pascal-50S mean (84.5).

Methodology in Plain English

The method starts from an existing LVLM that already knows how to output a caption score, such as "Score: 7", given a carefully designed instruction prompt. Two things are read off the model at the moment it emits that score digit: the distribution over the ten possible digits (0 through 9, or the first decimal place in the 0.0–1.0 scale), and a hidden feature vector from the last decoder layer.

The authors' insight is that this digit distribution is not the same thing as the distribution of human scores. Human ratings, by the central limit theorem, tend to be bell-shaped, while model token distributions are lumpy — for instance, the digit 0 gets too much probability mass. So instead of using the token distribution directly, they train (at test time, per image-caption pair) a small extra head consisting of one linear layer plus a softmax. This head must satisfy two objectives at once: stay close to the LVLM's own digit probabilities, and stay close to a Gaussian centered on the raw score the model produced. A weight parameter, alpha, decides how much to trust the Gaussian; it is set high when the raw score is near the extremes of the scale and low in the middle, because extreme scores are easier to judge. The final reported score is the expected value of the resulting distribution, giving a real number such as 0.73.

The clever part is that this optimization does not need gradient descent. If the LVLM's own digit prediction can be written as a linear layer plus softmax, the optimal weights of the new head have a closed-form expression in terms of the LVLM's weight matrix, its bias, and the log of the prior distribution. The authors prove this in Appendix A. When other divergence measures are substituted (for ablation), no closed form exists and they fall back on Adam for ten iterations with a learning rate of 10^-3.

For the benchmark, they drew images from DomainNet and InfographicVQA, sampled 1,000 images per domain, generated candidate captions with GPT-4o-mini, GPT-4o, Gemini 2.0 Flash and Claude 3.5 Sonnet, had annotators turn one candidate into a reference caption, and had three annotators pick the better of two other candidates based on relevance, descriptiveness, correctness and fluency. Disagreements caused the image to be discarded and replaced. Annotation was done by 81 crowdworkers on MTurk and Upwork.

Why This Matters

This work targets a practical bottleneck in multimodal AI: if you cannot measure caption quality reliably, you cannot compare models, filter training data, or monitor deployed systems. The paper argues that the gap between model probability distributions and human judgment distributions is the root cause of evaluation brittleness under domain shift, and shows that a correction can be applied at inference time without retraining anything. It also supplies a benchmark (MCEval) that deliberately stresses metrics with non-photographic imagery, which previous benchmarks of the compared datasets (Flickr8k-Expert, Flickr8k-CF, Composite, Pascal-50S) all label as real-domain data.

Real-world applications:

• Alt-text and accessibility pipelines: scoring machine-generated image descriptions before they reach screen-reader users, including for diagrams, clip art, and infographics.
• Image generation and editing systems: automatically ranking candidate captions or checking whether a generated image matches its prompt across stylized outputs such as sketches and paintings.
• Training-data curation: filtering or ranking large image-caption datasets where no human reference captions exist, since DISCODE is used as a reference-free metric.
• Benchmarking and model selection: comparing LVLMs on captioning quality in domains beyond photographs, where the paper shows learning-based metrics like PAC-S and Polos degrade.

Industry relevance: The method is drop-in for any team running open-source LVLMs, adds under 1% scoring overhead in the reported timing measurements, and requires no labeled data or finetuning. The limitation is significant for commercial deployment: because DISCODE reads latent decoder features, it cannot be applied to proprietary models such as GPT-4o.

Future Directions

• Extending to closed models. The authors state that using latent decoder features blocks application to proprietary LVLMs like GPT-4o that do not expose feature extraction; finding an alternative signal would broaden applicability.
• Bias mitigation beyond caption evaluation. The paper names extending bias-mitigation techniques to tasks other than image captioning evaluation as a promising direction.
• Stronger open-source LVLMs. Improving the underlying open-source models is framed as a route to further performance gains rather than improving the decoder alone.
• Finer-grained bias analysis. The paper documents symbolic bias (for example, over-estimated probability for the digit 0) but does not report a systematic characterization of which tokens or domains are most affected beyond the four example distributions shown for LLaVA-Next.

Target Audience

Researchers and engineers working on multimodal evaluation metrics, LVLM benchmarking, and caption quality assessment will get the most from this paper. It is also relevant to practitioners who need a finetuning-free scoring method for image-caption data in non-photographic domains, and to dataset builders interested in how MCEval was constructed (1,000 images per domain, three-annotator consensus, 81 crowdworkers). Readers without a background in probability divergences will find the analytical solution section demanding, though the intuition behind the method is described in accessible terms.

Authors’ abstract

Large vision-language models (LVLMs) have shown impressive performance across a broad range of multimodal tasks. However, robust image caption evaluation using LVLMs remains challenging, particularly under domain-shift scenarios. To address this issue, we introduce the Distribution-Aware Score Decoder (DISCODE), a novel finetuning-free method that generates robust evaluation scores better aligned with human judgments across diverse domains. The core idea behind DISCODE lies in its test-time adaptive evaluation approach, which introduces the Adaptive Test-Time (ATT) loss, leveraging a Gaussian prior distribution to improve robustness in evaluation score estimation. This loss is efficiently minimized at test time using an analytical solution that we derive. Furthermore, we introduce the Multi-domain Caption Evaluation (MCEval) benchmark, a new image captioning evaluation benchmark covering six distinct domains, designed to assess the robustness of evaluation metrics. In our experiments, we demonstrate that DISCODE achieves state-of-the-art performance as a reference-free evaluation metric across MCEval and four representative existing benchmarks.