ChartE$^{3}$: A Comprehensive Benchmark for End-to-End Chart Editing

AI summary

Overview

Research area: Computer vision and multimodal large language model (MLLM) evaluation, specifically benchmarking chart image editing in a direct image-to-image setting.

Technical level: Advanced. The paper assumes familiarity with diffusion/image-editing models, CLIP/DINO embeddings, and visual similarity metrics such as SSIM, PSNR, and LPIPS.

Scope: The paper introduces ChartE³, a benchmark of over 800 chart images and over 1,200 editing instructions designed to evaluate whether multimodal models can directly edit chart images end-to-end without intermediate code or natural-language program representations.

What This Paper Is About

Chart editing systems usually work by translating a user request into chart-rendering code, executing that code, and rendering a new image. This pipeline only tests a model's ability to write correct code, not its ability to directly manipulate a chart image while preserving data semantics and visual structure. ChartE³ reframes the task as a true image-to-image transformation and builds a curated benchmark to measure how well current models handle both fine-grained appearance edits and holistic data-centric edits.

Key Contributions

1. A new task formulation for end-to-end chart editing. The paper defines editing as a direct visual transformation, I′ = f_e2e(I, u), where a model maps an original chart image and an instruction to an edited image without intermediate code or program representations.

2. The ChartE³ benchmark. A dataset of 808 images and 1,211 instructions, built from roughly 10K candidate images across more than 40 chart types, spanning 12 fine-grained editing tasks grouped into local and global categories. Each sample is a triplet of chart image, its underlying code, and a multimodal editing instruction.

3. A five-stage, human-verified data pipeline. Collection, diversity filtering, chart–code pair generation, instruction generation, and manual filtering, with three human experts inspecting each generated sample.

4. Extensive benchmarking and analysis of current models. Seven image editing models are evaluated using five objective metrics plus GPT-based subjective scoring, with an empirical validation of GPT–human agreement on 140 randomly selected samples.

Main Findings

• Closed-source models lead, Nano Banana leads overall. Nano Banana records the best overall results: SSIM 0.83, PSNR 17.53, CLIP 0.96, DINO 0.96, LPIPS 0.23, Correctness 4.18, Consistency 4.65. GPT-Image-1.5 follows with SSIM 0.79, PSNR 15.05, CLIP 0.96, DINO 0.95, LPIPS 0.36, Correctness 3.79, Consistency 4.28.

• Open-source models trail, with Qwen-Image-Edit and BAGEL the strongest among them. Qwen-Image-Edit reaches SSIM 0.83, PSNR 16.83, CLIP 0.93, DINO 0.93, LPIPS 0.28, Correctness 2.58, Consistency 3.15. BAGEL reaches SSIM 0.82, PSNR 16.28, CLIP 0.89, DINO 0.93, LPIPS 0.28, Correctness 1.84, Consistency 3.23. InstructPix2Pix, OmniGen2, and Step1X-Edit perform noticeably worse; OmniGen2 records the lowest SSIM at 0.55 and PSNR at 8.70, and the highest LPIPS at 0.52.

• Local edits are consistently easier than global, data-level edits. Nano Banana scores 4.82/4.82 on Color and 4.74/4.95 on Axis, and GPT-Image-1.5 reaches 4.79/4.89 on Title. On global tasks the same models fall: Nano Banana drops to 3.42/4.42 on Order and 3.59/4.07 on Filtering, while GPT-Image-1.5 declines to 2.44/3.35 and 3.05/3.42 respectively. Most open-source models score below 2.0 correctness on ordering and filtering.

• Reference element editing is comparatively manageable. Nano Banana achieves 4.69/4.90 on adding or modifying reference elements, which the authors read as evidence that additive global edits are easier than other global transformations.

• Performance varies by chart type. Nano Banana scores 4.38/4.71 on bar, 4.49/4.68 on heatmap, and 4.43/4.79 on scatter charts, and 4.33/4.64 on the "Other" category. GPT-Image-1.5 is strongest on node charts (4.24/4.57) and line charts (4.00/4.34). Simpler, standardized chart types score higher than specialized types; the average correctness of open-source models on radar charts remains below 2.0.

• Objective and subjective metrics agree. Models that score higher on SSIM, CLIP, DINO, and LPIPS also tend to score higher on GPT-based Correctness and Consistency.

• GPT-based judging tracks human judgment. On 140 randomly selected samples, GPT scoring achieves pairwise accuracy of 0.79 and NDCG@7 of 0.92 for Correctness, and 0.87 pairwise accuracy and 0.97 NDCG@7 for Consistency.

• Failures often stem from text generation and instruction grounding. Error analysis indicates many failures occur even when visual understanding is correct, pointing to limitations in generating and grounding text within charts.

Methodology in Plain English

The researchers built the benchmark in five stages. First, they gathered about 10K candidate chart images from four existing sources: ChartBench (2.1K real images, 9/42 chart types), ChartX (6.0k synthetic, 18 types), ChartM³ (1.0k synthetic, 10 types), and Chart2code (0.9k real, 22 types). Second, they grouped images into 10 chart types and used CLIP image embeddings with k-means clustering, keeping the image closest to each cluster centroid so that each chart type ended up with roughly 100 representative images. Third, for images lacking rendering code, they used Gemini-2.5-Pro to synthesize chart code, deliberately choosing a different model from the GPT used in evaluation to avoid data leakage; code was rendered and checked for executability and CLIP similarity above 0.7, with up to three refinement iterations, and over 99% of examples were retained. Fourth, they sampled an editing operation from predefined primitives, prompted a model to produce edited code and render it, then asked the model to verbalize the change as a natural-language instruction, yielding triplets of (original image, instruction, reference edited image). Fifth, three human experts reviewed every sample, discarding incorrect, artifact-laden, occluded, or ambiguous cases and manually fixing minor glitches.

Evaluation combines five objective metrics — SSIM, PSNR, CLIP Similarity, DINO Similarity, and LPIPS — with two GPT-5.1-judged subjective scores, Correctness and Consistency, each on a 1-to-5 scale. Objective evaluation resizes images to a unified resolution (the smaller of the compared pair, or a prescribed input size such as 336px for CLIP-based metrics), while subjective evaluation uses original resolutions. All models were run on a server with 8× NVIDIA A100 GPUs using official inference code.

Why This Matters

This work shifts chart editing evaluation away from code correctness and toward the actual visual outcome users see. By removing intermediate program representations from the loop, it exposes a capability gap that code-mediated benchmarks can mask: models that produce syntactically valid chart code may still fail to render the intended edit faithfully.

Real-world applications:

• Business and financial reporting tools that let analysts restyle or filter existing dashboards by natural-language request.
• Scientific figure preparation, where authors need to adjust axes, labels, colors, or add reference lines to charts already embedded in manuscripts.
• Data analysis assistants embedded in notebooks or BI platforms, where a user points at a chart and asks for a change rather than editing plotting code.
• Accessibility and localization workflows, where chart titles, legends, fonts, and axis labels must be changed consistently without regenerating the underlying data.

Industry relevance: The finding that closed-source models currently outperform open-source alternatives, and that general-purpose image editors such as InstructPix2Pix and OmniGen2 handle structure-sensitive chart content poorly, gives product teams a clear signal about where current tooling is dependable (appearance edits) and where it is not (data filtering, reordering, and other data-dependent transformations). The validated GPT-based scoring protocol also offers a practical, scalable way to run regression tests without hiring annotators for every evaluation cycle.

Future Directions

• Closing the local–global gap. Global editing, particularly Order and Filtering, remains the weakest area even for the best models; the paper calls for models that reason jointly over visual appearance, data semantics, and editing intent.
• Improving instruction grounding and text generation. The error analysis points to these as failure sources independent of visual understanding, making them a natural target for targeted model improvements.
• Broadening chart-type robustness. Performance degrades on specialized formats such as radar, box, and heatmap charts, so extending robustness across chart types remains an open problem.
• Scaling evaluation and exploring alternative paradigms. The paper references additional analysis on different image editing paradigms, error analysis, and case studies in its appendices, and the validated GPT judge opens the door to larger-scale or automated evaluation of future end-to-end editors.

Target Audience

Researchers and engineers working on multimodal large language models, image editing, and chart or document understanding. It is also relevant to practitioners building data visualization assistants, BI tooling, or automated reporting systems, and to benchmark designers interested in how to construct human-verified, instruction-driven visual evaluation datasets. Readers need a working familiarity with image generation and perceptual similarity metrics to follow the experimental section in detail.

Authors’ abstract

Charts are a fundamental visualization format for structured data analysis. Enabling end-to-end chart editing according to user intent is of great practical value, yet remains challenging due to the need for both fine-grained control and global structural consistency. Most existing approaches adopt pipeline-based designs, where natural language or code serves as an intermediate representation, limiting their ability to faithfully execute complex edits. We introduce ChartE$^{3}$, an End-to-End Chart Editing benchmark that directly evaluates models without relying on intermediate natural language programs or code-level supervision. ChartE$^{3}$ focuses on two complementary editing dimensions: local editing, which involves fine-grained appearance changes such as font or color adjustments, and global editing, which requires holistic, data-centric transformations including data filtering and trend line addition. ChartE$^{3}$ contains over 1,200 high-quality samples constructed via a well-designed data pipeline with human curation. Each sample is provided as a triplet of a chart image, its underlying code, and a multimodal editing instruction, enabling evaluation from both objective and subjective perspectives. Extensive benchmarking of state-of-the-art multimodal large language models reveals substantial performance gaps, particularly on global editing tasks, highlighting critical limitations in current end-to-end chart editing capabilities.