ChartAnno: Evaluating MLLMs for Chart Annotation Generation

AI summary

Overview

Research area: Multimodal large language models (MLLMs) for visualization authoring, specifically the generation of chart annotations as executable code. This sits at the intersection of chart generation, chart editing, and benchmark design for AI evaluation.

Technical level: Intermediate. The task setup (give a model chart code plus instructions, get annotated chart code back) is easy to grasp, but the evaluation framework involves detailed rule-based graphical-element comparison and LLM-as-judge rubric design that is more advanced.

Scope: The paper introduces ChartAnno, a benchmark of 1,200 real-world charts with paired annotated and unannotated executable code and 3,600 annotation instructions at three levels of specificity, together with a four-dimension evaluation framework, used to test 10 MLLMs.

What This Paper Is About

Chart annotations explain data, highlight findings, and direct a reader's attention, but writing them is still largely manual work. MLLMs might be able to automate this, yet the paper argues that systematic evaluation of MLLM chart annotation generation is underexplored, since existing chart benchmarks focus on understanding, generation, or editing rather than on annotating an existing chart from a communicative goal. The goal is to build a benchmark and evaluation framework that measures whether models can turn anything from an abstract intent to a concrete implementation spec into executable, faithful, well-designed annotations without damaging the underlying chart.

Key Contributions

1. A new benchmark task formulation. The authors formulate chart annotation generation as an MLLM benchmark task that spans from high-level communicative intents down to concrete implementation specifications, distinguishing it from chart editing, where the desired modification is normally stated explicitly.

2. A large, real-world dataset with paired resources. ChartAnno contains 1,200 real-world charts with paired unannotated and annotated executable code, plus 3,600 instructions at three levels of specificity, providing reusable resources for tasks such as chart generation. The dataset includes 653 public-facing and 547 scientific charts, covers 17 chart types from seven data sources, and contains 25,772 annotation elements.

3. A multidimensional evaluation framework. The framework combines rule-based metrics (Execution Rate, Structural Compliance) with LLM-judged metrics (Semantic Consistency, Design Effectiveness) to assess execution, structural compliance, semantic consistency, and design effectiveness.

4. A large-scale evaluation with validation and generalization checks. The authors evaluate 10 MLLMs across three instruction levels, two primary chart input settings, and an auxiliary image-only ablation, validate the evaluation framework against human agreement, and extend the benchmark to D3 and SVG on a 120-chart subset.

Main Findings

• Proprietary models lead overall, open-source models narrow the gap. Across the evaluation, proprietary models hold the top positions, while open-source models come closer than the overall ordering alone would suggest.

• More specific instructions produce better annotations. Performance generally improves as instructions move from Intent to Operation to Implementation level.

• Inferring abstract communicative intent stays hard. Intent-level generation remains challenging across all models, even though it is the least specified instruction level.

• Chart images add little when code is available. Providing the chart image alongside the code yields only marginal benefit in the two primary settings.

• Removing code hurts substantially. In the image-only ablation, performance drops sharply, especially for open-source models and for the more detailed instruction levels.

• Performance falls as complexity rises. Results decrease systematically as chart and annotation complexity increase.

• The judge is reliable. Human ratings of the rubric were highly consistent (ICC of 0.869 for Semantic Consistency, 0.903 for Design Effectiveness, and 0.918 overall), and GPT-5.4 showed the strongest alignment with aggregated human ratings among the three candidate judges (Spearman's rho of 0.812, 0.873, and 0.867), compared with Claude Sonnet 4.6 (0.808, 0.805, 0.832) and Gemini 3.1 Pro Preview (0.714, 0.732, 0.748). Cross-judge rank consistency between GPT-5.4 and the other candidates was also strongly correlated.

• Trends generalize beyond Python. Experiments with D3 and SVG on a 120-chart subset largely reproduce the Python trends, and the representations show performance trade-offs across instruction levels, suggesting code length alone does not explain performance.

• A concurrent benchmark differs in setting. The related concurrent preprint AnnoBench contains 342 charts across six visualization representations and two instruction levels, uses reference-free LLM judging, and derives annotation tasks for 58 professional charts from original real-world charts while the remaining 284 Vega/Vega-Lite charts are built through an LLM-assisted pipeline. The authors report that only 28 samples from AnnoBench support a matched reference-based comparison at the Operation and Implementation levels, which they consider insufficient for a benchmark-level comparison.

Methodology in Plain English

The authors first assembled a large pool of candidate figures — 113,014 in total — from existing chart benchmarks (ChartQAPro, CharXiv, ChartMimic, MatPlotBench), from visualization studies on annotations and textual content, and from CC BY 4.0 arXiv papers released between February 2025 and February 2026 that were accepted at leading peer-reviewed venues. They used Semantic Scholar for publication metadata and MinerU to parse PDFs and extract figures, following the pipeline used in ChartFI. A two-stage MLLM screening removed non-chart figures and scored the rest on chart completeness, information richness, and visual complexity; charts scoring below 90 were removed, leaving 24,288 candidates for manual review. Three authors then reviewed these for readability, information sufficiency, diversity, and redundancy, producing 1,200 annotated charts.

Each chart was reconstructed in Python with initial code generated by GPT-5.2 and Gemini 3 Pro, scored with a 100-point rubric adapted from ChartMimic that checks chart type, layout, text content, data, style, and clarity. Initial reconstructions averaged 92.34 on charts from sources other than ChartMimic. Five authors manually refined the reconstructions and corrected three kinds of source-chart problems: ambiguous annotation intent, factual inconsistencies, and visual presentation issues.

To create the unannotated counterpart of each chart, annotation elements were categorized by annotation type and information source following prior annotation design-space research. LLMs then removed annotation-specific code, and five authors manually reviewed the results so the unannotated version preserved the underlying chart content and visual structure.

Instructions were built from a structured schema recording each annotation's communication goal (identify, compare, summarize, or present), target, source, content, annotation type, and markers. Gemini 3 Pro produced initial structured representations and instruction drafts, GPT-5.2 assisted in refinement, and three authors reviewed them. The result is three instruction levels: Intent (what to communicate), Operation (which annotation actions, targets, and placement relations), and Implementation (concrete rendering parameters).

Evaluation runs on the graphical object hierarchy rather than on source code, so equivalent visualizations can be compared regardless of how they were written. Rule-based scoring covers Execution Rate (1 if the code renders without runtime errors, 0 otherwise) and Structural Compliance, which for Operation and Implementation levels is the average of Chart Fidelity, Annotation Matching, and Color Matching, and for Intent level is Chart Fidelity alone. Chart Fidelity is binary, checking preservation of figure aspect ratio, axes layout and aspect settings, and data-carrying marks and their encoded values. Annotation Matching uses a Jaccard-style coefficient over seven annotation categories (enclosure, connector, text, glyph, color, indicator, and geometric), with relaxed matching for non-text annotations and normalized multiset matching for text. Color Matching converts colors to CIELAB, computes CIEDE2000 color difference, converts that to a similarity, and uses the Hungarian algorithm to find the best one-to-one assignment, reporting precision, recall, and their harmonic mean.

For qualities that rules cannot capture, an LLM judge scores five submetrics on an integer scale of 1 to 5, with 0 for failed or missing cases. Semantic Consistency averages Semantic Faithfulness and Semantic Clarity; Design Effectiveness averages Visual Clarity, Annotation Organization Quality, and Attention Guidance. The judge receives the rendered chart and also the quantitative text-overlap and off-canvas statistics as evidence for Visual Clarity. GPT-5.4 was selected as the judge after a pilot study on 90 randomly sampled outputs from Gemini 3 Flash Preview rated by three coauthors.

Why This Matters

Impact on research. ChartAnno gives the visualization and NLP communities a shared, reproducible way to measure annotation generation, a task that sits between chart generation and chart editing but is not covered by existing benchmarks. Its differential-analysis strategy — comparing generated and ground-truth charts against a shared unannotated reference — offers a template for evaluating modifications to existing artifacts rather than generating them from scratch, and its three instruction levels give a way to separate "can the model follow a spec" from "can the model infer what to say."

Real-world applications:

• Data journalism and news graphics, where reporters need to add callouts, highlights, and explanatory text to existing charts quickly.
• Scientific paper authoring, where figures need annotations that point readers to the relevant trend or comparison (the paper explicitly curates 547 scientific charts alongside 653 public-facing ones).
• Business dashboards and analytics tools, where charts are regenerated frequently and annotations must be reapplied or adapted automatically.
• Chart authoring libraries and declarative annotation grammars, which could use these instructions and structured annotation representations as training or evaluation data.

Industry relevance. The finding that chart images add only marginal benefit when code is available has direct cost implications: pipelines that already have chart code may not need to pay for image input. The finding that open-source models are closing the gap, combined with the fact that all 10 evaluated models include four Qwen3.5 variants from 9B to 397B parameters, is relevant to teams deciding where a hosted proprietary model is actually necessary. The image-only ablation results matter for tools that operate on screenshots of charts, which is a common constraint in real products.

Future Directions

• Improving intent-level inference. Since Intent-level generation remains difficult for every model tested, closing that gap — inferring what a chart should communicate without being told the annotation operations — is the clearest open problem.
• Explaining and exploiting the D3 and SVG trade-offs. D3 and SVG show performance trade-offs across instruction levels that code length alone does not explain, so what property of a representation makes annotation generation easier remains unresolved.
• Extending beyond static Python charts. The benchmark is instantiated in a controlled static-chart environment in Python, with D3 and SVG only on a 120-chart subset; broader representation coverage and interactive chart settings are untouched.
• Moving from benchmark to authoring tool. The paper frames its dataset as reusable for tasks such as chart generation, but does not report building or testing an end-user annotation assistant, leaving the human-in-the-loop workflow as an open direction.

Target Audience

Researchers working on MLLM evaluation, chart and visualization generation, and human-data interaction will get the most from this paper, since the benchmark design, the rule-based metrics, and the judge-validation protocol are the substantive contributions. Practitioners building chart authoring tools, BI dashboards, or agentic visualization pipelines will benefit from the empirical comparison of 10 models across input settings, and from the finding that code alone is a strong input. Designers of evaluation frameworks more broadly will find the differential-analysis approach and the combined rule-based plus LLM-judged scoring useful, as will anyone who needs a sanity check on using a model as a judge alongside human ratings.

Authors’ abstract

Multimodal large language models (MLLMs) have made significant progress in chart understanding, generation, and editing, but their ability to annotate existing charts remains underexplored. Annotating charts is a common yet challenging communicative task, requiring models to infer intended messages, interpret chart semantics, and place appropriate textual or graphical elements. To address this gap, we introduce ChartAnno, a benchmark for evaluating MLLMs on chart annotation generation. It contains 1,200 real-world charts with paired code and annotation instructions across three levels of instruction specificity. We evaluate 10 representative MLLMs under two primary input settings: (1) chart code alone and (2) both chart code and chart image, and further include a chart image-only ablation study. Results show that proprietary models remain stronger overall, although large-scale open-source models narrow the gap. More specific instructions improve annotation quality, while inferring abstract intent remains most difficult for current MLLMs. Providing chart images brings limited overall gains, with improvements mainly appearing in design-related metrics. These findings highlight chart annotation generation as a challenging task requiring semantic grounding and effective annotation design. Code and data will be released in a future version.