Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices

AI summary

Overview

Research area: Natural Language Processing / multimodal evaluation — specifically, using Large Vision-Language Models (LVLMs) as automated judges for chart comprehension outputs.

Technical level: Advanced. The paper assumes familiarity with LVLM fine-tuning, knowledge distillation, judging rubrics (pairwise/pointwise), and deployment cost metrics.

Scope: The paper investigates whether a 2B-parameter LVLM can serve as a reliable, low-cost evaluator of chart models through multi-criteria prompting and domain-adaptive fine-tuning, producing the ChartJudge-2B model (arXiv:2510.07545v2, cs.CL, 10 Oct 2025, CC BY 4.0).

What This Paper Is About

Evaluating chart-understanding models traditionally requires costly human judgment, and while 7B-parameter LVLMs have shown promise as automated judges on chart tasks, tiny models of 2B parameters or fewer still perform poorly, limiting their use where compute and inference budgets are constrained. The authors build ChartJudge-2B by fine-tuning Qwen2-VL-2B-Instruct on synthetic judgments generated by a stronger LVLM, and also test multi-criteria prompting, which fuses several evaluation dimensions into one instruction to cut latency and cost. The goal is a scalable, low-cost evaluation pipeline for chart reasoning tasks without relying on closed-source APIs.

Key Contributions

1. Demonstrating that ChartJudge-2B transfers judgment knowledge across chart datasets, even when the LVLM that produced the training judgments differs from the one used for reference at evaluation time.
2. Showing that multi-criteria prompting exposes severe robustness gaps in open-source LVLMs, including specialized judges such as LLaVA-Critic-7B and LLaVA-Critic-70B, while a fine-tuned tiny LVLM can still perform well in that setting.
3. Providing practical guidance on model selection, prompt design, error analysis, and transfer strategies for scalable deployment of LVLM judges on chart reasoning tasks.
4. Releasing code and data at https://github.com/tahmedge/chart_lvlm_judge.

Main Findings

• Fine-tuning lifts the 2B backbone in pairwise judging: ChartJudge-2B reaches pairwise judgment accuracy of 61.7 on OpenCQA, 64.6 on VisText L1, and 52.3 on VisText L2/L3, versus 54.0, 27.2, and 3.0 for its zero-shot Qwen2-VL-2B-Instruct backbone. It also beats larger models in several settings, including Qwen2-VL-7B on VisText L1 (64.6 vs 57.6) and Phi-3.5-Vision-3.8B-Instruct on OpenCQA (61.7 vs 50.7) and VisText L2/L3 (52.3 vs 49.5).
• Pointwise gains are limited: Fine-tuning does not reliably improve pointwise error distance except on VisText L1 (1.6 vs 2.1). The authors attribute this to using different LVLM judgment scores as reference during training (Gemini-1.5-Pro) versus evaluation (GPT-4o / LLaVA-Critic-70B), though they found a positive Spearman's rank correlation between ChartJudge-2B and larger models in how they score examples.
• Instruction following and format improve, bias worsens: On Chart-Instruct-Eval, ChartJudge-2B scores 36.5 for instruction following (backbone: 13.5), 95.9 for format adherence (backbone: 78.9), and 79.5 for bias (backbone: 63.5), where higher bias is worse. The authors note LLaVA-Critic-7B, the strongest model in Table 2, also shows high bias (58.0).
• Multi-criteria prompting breaks 7B judges: On OpenCQA, LLaVA-Critic-7B drops from pairwise average 76.78 (single-criterion) to 0.00 (multi-criteria) with pointwise error distance rising from 0.64 to 5.00 and format following falling from 99.57 to 0.00. LLaVA-Next-v1.6-Mistral-7B falls from 72.13 to 0.93 pairwise. LLaVA-based models (LLaVA-Next, LLaVA-Critic 7B and 70B) fail to follow the multi-criteria instruction, which the authors suggest may stem from their training data lacking multi-criteria judgments.
• The tiny fine-tuned model handles multi-criteria: ChartJudge-2B (Multi-Criteria) achieves pairwise average 46.86 in the multi-criteria setting, above Qwen2-VL-2B-Instruct (22.16), Phi-3.5-vision-3.8B-instruct (46.74), XGen-MM-Phi3-3.8B-Instruct (44.07), and Qwen2-VL-7B-Instruct (40.62), though below its own single-criterion score of 66.35.
• Ablations show both evaluation types are needed: Removing query relevance from training drops pairwise performance; training on only pairwise or only pointwise data destroys capability in the untrained setting; and model merging (via the mergekit library with linear merging, weight 1.0, and an added visual.merger.mlp layer initialized to 0) produces the best pairwise result when combining separately fine-tuned models.
• Complex charts are harder: Two human experts annotated 896 OpenCQA samples as containing complex charts and queries. On these, ChartJudge-2B achieves 63.3% judgment accuracy versus 54.5% for zero-shot Qwen2-VL-2B-Instruct.
• Fine-tuning rescues a very weak base model: PaliGemma-3B, which scores 0% as a zero-shot chart judge, improves from 0% to 55.9% pairwise accuracy on VisText after fine-tuning, with format adherence rising from 0% to 77.0%.
• Out-of-domain generalization fails: On 3K samples of the ChEBI-20-MM molecular image captioning dataset, ChartJudge-2B scores 49.53 judgment accuracy, below its backbone Qwen2-VL-2B-Instruct (80.26) and Qwen2-VL-7B-Instruct (90.64), indicating the fine-tuned model loses zero-shot ability outside charts.
• Deployment cost is roughly halved: ChartJudge-2B runs on 8GB VRAM while 7B baselines need at least 24GB. On one A30 GPU it achieves about 110 tok/sec (8.1 ms/token latency) versus about 63 tok/sec (15.9 ms/token) for 7B models. On GCP, 7B models can run on one L4 GPU (24GB) at $0.7–$0.9 per hour, whereas 2B models can run on a T4 at $0.3–$0.5 per hour, making ChartJudge-2B twice as fast and 2x cheaper. On an AMD machine with a 64-core processor, the 2B model takes 2.5 seconds per token on CPU versus 15.9 seconds for the 7B model.

Methodology in Plain English

The authors take a small vision-language model, Qwen2-VL-2B-Instruct, and turn it into a chart judge through distillation rather than human labels. They start from the Chart-to-Text benchmark: the Statista split is used as-is for chart captioning, and synthetic questions are generated on the Pew split by prompting Gemini-1.5-Pro with the chart and its gold reference to turn captioning into open-ended QA. Responses are produced by GPT-4V and Phi-3, and the training labels are judgments written by Gemini-1.5-Pro. To avoid teacher-model bias, the pipeline deliberately uses different models at each stage: training responses come from GPT-4V and Phi-3, training labels from Gemini-1.5-Pro, while evaluation responses come from Claude-3-Haiku and Gemini-1.0-Pro, and reference judgments from GPT-4o and LLaVA-Critic-70B.

The fine-tuning data covers both pointwise ratings (1–5 with a textual justification) and pairwise comparisons (Model A, Model B, or Tie with a rationale), totaling 9,725 samples in the single-criterion setting and 2,827 samples in the multi-criteria setting (built on the Pew split using informativeness and factual correctness). Training ran for three epochs with batch size 2 and a learning rate between 1e-4 and 2e-5 on two NVIDIA H100 GPUs. For prompting-based evaluation, they reduce cost by packing multiple criteria (informativeness and factual correctness) into a single instruction that returns an array of JSON judgments. Evaluation covers OpenCQA, VisText (L1 and L2/L3 captions), and Chart-Instruct-Eval, using judgment accuracy, error distance, bias, instruction following, and format adherence as metrics; the judgment annotations for OpenCQA and VisText comprise 100K judgment data, while Chart-Instruct-Eval contains 400 samples. Inference used temperature 1.0 and a maximum output length of 300 tokens.

Why This Matters

The work shows that a 2B-parameter judge can match or beat some 7B judges on chart tasks at roughly half the cost, which lowers the barrier to automated evaluation in settings where proprietary charts cannot be sent to closed-source APIs. It also documents a clear failure mode: multi-criteria prompting, which looks attractive for cost reduction, collapses the performance of several 7B models, including the specialized LLaVA-Critic family, while a fine-tuned tiny model remains functional.

Real-world applications:

• Industrial CI/CD pipelines that need a fast, cheap judge to compare a newly developed chart model against the deployed one in pairwise mode.
• Organizations handling sensitive proprietary charts that cannot share data with closed-source LLM providers and therefore need on-premise evaluation.
• Resource-constrained research groups and startups that lack GPUs with 24GB VRAM and instead run a judge on 8GB or on T4-class hardware.
• Evaluation of chart captioning and open-ended chart QA systems where human judgment would otherwise be required.

Industry relevance: The paper explicitly targets industrial deployment, reporting VRAM requirements, tokens per second, and GCP hourly costs, and recommends deploying ChartJudge-2B as a pairwise judge rather than a pointwise one because pointwise error distance remains comparatively high.

Future Directions

• Developing more flexible, modular prompting strategies that can be programmatically adjusted to different application requirements, enabling dynamic evaluation across diverse chart-related tasks and other domains.
• Mitigating the increased bias in the fine-tuned model, with specific plans to conduct experiments on length bias.
• Improving the quality of synthetic judgment data to raise overall judge performance.
• Extending multi-criteria prompting beyond factual correctness and informativeness to criteria such as relevance and conciseness, which the paper did not test.

Target Audience

Researchers and practitioners working on multimodal evaluation, chart understanding, and LLM-as-a-judge systems; industry engineers deploying evaluation pipelines under compute, latency, or data-privacy constraints; and anyone interested in knowledge distillation from large LVLMs into 2B-scale models. Readers should be comfortable with LVLM benchmarking terminology, since the paper's contribution is primarily empirical and deployment-oriented rather than introducing a new architecture.

Authors’ abstract

Large Vision-Language Models (LVLMs) with only 7B parameters have shown promise as automated judges in chart comprehension tasks. However, tiny models (<=2B parameters) still perform poorly as judges, limiting their real-world use in resource-constrained settings. To address this, we propose two approaches to ensure cost-efficient evaluation: (i) multi-criteria prompting, which combines separate evaluation criteria into a single query, and (ii) domain-adaptive transfer learning, in which we fine-tune a 2B-parameter LVLM on synthetic judgments in a chart dataset to create the ChartJudge. Experiments show that multi-criteria prompting exposes robustness gaps, which led to a huge drop in performance for 7B models, including specialized LVLM judges like LLaVA-Critic. In addition, we find that our tiny LVLM (ChartJudge) can effectively transfer knowledge from one dataset to another to make it a more specialized model. Our fine-grained analysis across chart types and query complexities offers actionable insights into trade-offs between model size, prompt design, and transferability, enabling scalable, low-cost evaluation for chart reasoning tasks.