VisCodeBench: a new benchmark dataset for Text-to-Vis reflecting real-world practice

AI summary

Overview

Research area: Natural Language Processing — specifically text-to-visualisation (text-to-vis), where a system turns a natural-language request plus underlying data into executable chart-generating code.

Technical level: Intermediate. Readers should be comfortable with LLM benchmarking, Python plotting libraries (Matplotlib, Seaborn, Pandas), and evaluation metrics, but the paper is written accessibly and reports its results in tabular form.

Scope: The paper introduces VisCodeBench, a 2,672-sample benchmark built from real-world visualisation artefacts across educational, scientific, and public-interest settings, and uses it to evaluate three visualisation systems paired with five LLMs on seven metrics.

What This Paper Is About

Existing text-to-vis benchmarks are mostly synthesised or automatically constructed: their data is often synthetic, their charts carry little real analytical meaning, and their generated code follows fixed templates designed by the dataset creators. As a result, they do not measure how systems behave on the messy, customised visualisations people actually produce.

The goal of this paper is to build a benchmark whose samples come from real users — course assignment notebooks, published academic papers, and a public data-journalism platform — and then to use it to measure how current LLM-driven visualisation systems perform on execution, content fidelity, and visual quality.

Key Contributions

1. A new 2,672-sample benchmark, VisCodeBench, composed of three subsets: GitHub-T2V (843 samples from MIT-licensed educational notebooks found in The Stack), arXiv-T2V (805 samples from a survey of academic authors), and OWID-T2V (1,024 samples from Our World in Data articles).

2. A human-in-the-loop construction pipeline. For arXiv-T2V, the authors invited approximately 30,000 authors of CC BY-SA 4.0 arXiv 2024 papers and received 1,151 valid submissions from 848 participants (a 2.69% response rate); GPT-5 drafted initial requests of roughly 120 words (50 words for OWID-T2V), which expert annotators with over four years of coding experience then rewrote.

3. Two new evaluation metrics added to five existing ones. Displayed Data Similarity adapts the ICDAR 2019 CHART-Infographics Task 6b metric using arc-length-parameterised interpolation and extension to multi-subplot figures; Accessibility Score covers text colour contrast and text spacing.

4. A systematic benchmark of three systems (Codex, Claude Code, LIDA) across five LLMs (GPT-OSS-20B, Gemma-4-31B, Qwen-3.6-35B, GPT-5, Opus-4.7), including an easy/hard split of arXiv-T2V (456 easy, 349 hard samples) and a comparison of human-written versus machine-generated requests.

Main Findings

• Basic execution is nearly solved for strong systems. Task completion is 99–100% and chart-type accuracy 0.90–0.97 for every strong system, across all three subsets.

• Small local models still fail at execution. Co-GPT-OSS-20B completes only 76% (GitHub-T2V), 67% (arXiv-T2V), and 60% (OWID-T2V) of tasks.

• Content fidelity is the unsolved problem. The best displayed data similarity anywhere is 0.81 (Cl-Opus-4.7 on arXiv-T2V), and the top score on OWID-T2V is only 0.69. No system comes close to reproducing the reference data.

• Frontier models separate on correctness and speed. Cl-Opus-4.7 improves chart correctness by around 5 points over Co-GPT-5 on each of the three subsets while running roughly 20 seconds faster (a 20–43% speed-up), making Opus-4.7 the strongest model overall.

• System design matters as much as the model. Codex and LIDA both run on GPT-5, yet on OWID-T2V LIDA completes 79% of tasks versus Codex's 100%, with displayed data similarity 0.52 versus 0.68 and correctness 54.37 versus 73.67.

• Open models rival frontier systems on easy charts but not hard ones. Cl-Gemma4-31B is the strongest system on GitHub-T2V (correctness 75.41, displayed data similarity 0.70) and stays close on arXiv-T2V, but underperforms on OWID-T2V; Cl-Qwen3.6-35B follows the same trajectory.

• Latency is a barrier to interactive use. Even the strongest systems take 39–78 seconds; Cl-Qwen3.6-35B requires up to 231s on arXiv-T2V, and LIDA-GPT-5 exceeds 110s on every subset.

• Readability and contrast are high, spacing is not. Readability exceeds 4.5 out of 5 across all systems and text contrast is roughly 0.86–0.92 on a [0,1] scale, but text spacing is negative for every system, ranging from −3.58 to −0.33 on a (−∞, ∞) scale where positive values denote adequate spacing.

• The hard split degrades everything. All systems show longer prediction times and lower chart-type accuracy, displayed data similarity, correctness, readability, and text spacing on arXiv-T2V-hard (349 samples) than on arXiv-T2V-easy (456 samples).

• Machine-generated requests inflate results. Under machine-generated request variants, Co-GPT-5 improves significantly on GitHub-T2V and arXiv-T2V, suggesting human-authored requests give a more realistic and challenging evaluation.

• The dataset is harder and more diverse than prior benchmarks. GitHub-T2V uses 4.08 distinct libraries per sample and arXiv-T2V 3.15, versus 1.0 for nvBench and 1.71 for Text2Vis; the subsets cover 18 and 21 plot types respectively, with free-form data schemas rather than fixed ones.

Methodology in Plain English

The authors gather charts the way people actually make them, from three sources, and then turn each one into a test case consisting of data, a natural-language request, and a reference chart.

GitHub-T2V comes from teaching materials. The team scanned The Stack for MIT-licensed Python notebooks whose filenames contain "solution", randomly sampled 5,000 of them, pulled in the associated environment and data files, ran them with a custom script keeping only those that finished within a 60-second limit, used GPT-4o to detect visualisation requests in markdown cells, and then manually verified the results.

arXiv-T2V comes from a survey. Authors of CC BY-SA 4.0 arXiv 2024 papers were invited by email to submit a chart from their paper along with the code, the underlying data, and a newly written request. Because many submitted requests were abstract or looked LLM-generated, the team replaced them: GPT-5 drafted an approximately 120-word request, and native English-speaking experts with over four years of Python visualisation experience revised it while seeing only the data and the chart image — never the original code. A separate expert with over ten years of experience rated 60 samples for naturalness on a five-point scale; 52 of 60 scored 4 or 5, with an average of 4.34.

OWID-T2V comes from Our World in Data. Because those charts are produced by custom JavaScript, no code exists to release; instead each sample pairs a data file, a request, the chart image, and reconstructed reference displayed data points. Those reference points were produced by having Codex with GPT-5 replicate each chart from the published CSV and metadata, iterating until visually similar, discarding samples that did not converge after three attempts, and then extracting the points with custom scripts.

Evaluation uses seven metrics: task completion rate, prediction time, chart type accuracy, displayed data similarity, chart correctness (an LLM judge scoring pairs of images out of 100), readability (an LLM judge scoring out of 5), and the two accessibility measures. Displayed data points are extracted through a monkey-patching technique, and the similarity metric is taken from ICDAR 2019 CHART-Infographics Task 6b with two modifications: arc-length-parameterised interpolation replaces linear interpolation, and the data-series score is aggregated up to figures with multiple subplots.

The full dataset is kept private to prevent it being used for LLM training; access is through a leaderboard on the project's GitHub page.

Why This Matters

Impact on research. The paper shows that benchmark realism changes conclusions. Because VisCodeBench pulls from free-form, multi-library, multi-subplot code with variable data schemas, it exposes failures — particularly in reproducing the actual displayed data — that template-based benchmarks such as nvBench, ChartX, and Text2Vis largely hide. It also demonstrates that the surrounding pipeline (prompting, execution, refinement) can shift results as much as swapping the underlying model, a dimension that model-only benchmarks cannot capture.

Real-world applications:

• Data journalism and public communication, where a reporter needs a chart from a CSV on deadline and cannot verify by eye whether the plotted values are correct.
• Scientific figure production, where researchers re-plot experimental results and need the chart to match the underlying measurements exactly.
• Data science education, where students describe a visualisation in words and a system produces the code, drawn directly from the assignment-notebook subset.
• Business intelligence and dashboarding, where analysts want charts generated from natural-language requests over privacy-sensitive local data — the reason the paper includes open models run locally.

Industry relevance. The latency findings (39–78 seconds even for the best systems, up to 231s for some configurations) and the cost findings ($175 per run for Co-GPT-5 and $780 for Cl-Opus-4.7 across three splits) are directly relevant to anyone deploying these systems. The Codex-versus-LIDA result is a practical argument that engineering the surrounding agent pipeline can be as valuable as upgrading the model.

Future Directions

• Closing the content-fidelity gap. Displayed data similarity tops out at 0.81 and drops to 0.69 on OWID-T2V; understanding why systems reproduce chart shape but not chart values is the central open problem the benchmark exposes.

• Reducing latency for interactive use. No system reaches interactive speed. Whether iterative, multi-agent workflows — which the authors explicitly note their single-turn setup does not capture — can improve quality without worsening the 39–78 second delay is untested here.

• Broadening model coverage. The benchmark covers only seven system-model pairings and two proprietary models, a choice the authors attribute to cost; whether the rankings hold for other frontier and open models is unknown.

• Strengthening request naturalness. Because arXiv-T2V and OWID-T2V requests were GPT-5-drafted and expert-revised, the authors release an arXiv-T2V-Author variant with original author-written requests, and future work could test how much the drafting process shapes measured difficulty.

Target Audience

Researchers building or evaluating text-to-vis and code-generation systems will find the benchmark design, metrics, and comparison tables most directly useful. Practitioners deploying LLM coding agents for data analysis — and anyone choosing between a general coding agent and a specialised visualisation pipeline — will benefit from the latency, cost, and system-design findings. Benchmark designers more broadly will find the human-in-the-loop construction methodology and the critique of synthesised datasets relevant, as will educators interested in how well models handle assignment-style visualisation tasks.

Authors’ abstract

Generating visualisations from natural language requests (text-to-vis) enables users to explore data more efficiently. While LLMs can complete simple visualisation requests, their performance on challenging ones is not well covered by existing benchmarks. We introduce VisCodeBench, an end-to-end text-to-vis benchmark with 2,672 samples from three different domains: educational, scientific, and general audience scenarios with varying levels of complexity. We benchmark three visualisation systems---Codex, Claude Code, and LIDA---paired with a range of open and commercial LLMs across seven metrics covering execution, chart content, and visual quality. While basic execution is near-solved for strong systems, no system comes close to reproducing the reference data, with the best displayed data similarity reaching only $0.81$ and falling to $0.69$ on our hardest subset for this metric. We further find that frontier models separate on chart correctness and efficiency, with ClaudeCode-Opus-4.7 improving chart correctness by around 5 points over Codex-GPT-5 while running roughly 20 seconds faster. Separately, we find that system design matters as much as the underlying model: despite both running on GPT-5, Codex clearly outperforms LIDA, showing that the surrounding prompting, execution, and refinement pipeline can shape outcomes as much as the model itself. Finally, latency remains a practical obstacle, as even the strongest systems take 39-78 seconds to produce results. VisCodeBench is available at \href{https://github.com/SAIPLL/VisCodeBench}{https://github.com/SAIPLL/VisCodeBench} and accessed via a leaderboard; the full dataset is kept private to prevent LLM training use.