Edit2TikZ: A Comprehensive and Challenging Benchmark for Scientific Figure Editing with TikZ

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically the evaluation of instruction-guided scientific figure editing into compilable TikZ/LaTeX code.

Technical level: Advanced. The paper assumes familiarity with multimodal model evaluation, graphics-program synthesis, LaTeX/TikZ compilation pipelines, and program-level metrics such as cBLEU and TED.

Scope: The paper introduces Edit2TikZ, a 1,548-sample benchmark for editing scientific figures through TikZ code, together with a human-aligned evaluation framework (RestorationScore and EditCorrectnessScore), an evaluation of 14 MLLMs, and a mixed training set (TikZEditMix) used for reconstruction-then-editing curriculum training of compact models.

What This Paper Is About

When the original source code of a scientific figure is unavailable, a model that receives only a rendered figure plus an edit request must recover the figure's latent structure, locate the objects the instruction refers to, emit compilable TikZ code, and leave everything else unchanged. Existing TikZ benchmarks mostly measure figure reconstruction or generation, and existing chart-editing benchmarks mostly target plotting-library graphics such as bar and line charts. Edit2TikZ fills this gap by providing a dedicated benchmark for end-to-end, instruction-driven editing of diverse scientific figures into compilable TikZ, along with metrics that separately judge whether the requested edit was performed and whether unrelated content was preserved.

Key Contributions

1. A comprehensive scientific-figure editing benchmark (Edit2TikZ): 1,548 samples combining 534 real-world edits, 580 text-only synthetic edits, and 434 visual-prompt synthetic edits. Each sample pairs a source figure, an edit instruction, an optional visual localization prompt, a gold edited TikZ/TeX program, and a human-verified sequence of edit units drawn from eight atomic operations.

2. A human-aligned evaluation framework: RestorationScore (RS), a 0–100 measure of non-target preservation split into 30 points for text/style, 30 for objects/relations, and 40 for layout/scale, and EditCorrectnessScore (ECS), which scores each labeled edit unit on six levels ({0, 40, 80, 90, 95, 100}). Both metrics are validated against two human annotators and three candidate model judges on 100 randomly sampled cases.

3. A broad empirical study of 14 MLLMs: 8 proprietary models (Qwen3.7-plus, Doubao-Seed-2.1-Pro, GPT-5.6 Luna/Terra/Sol, Claude Sonnet 4.6, Claude Opus 4.8, Gemini-3.1-Pro) and 6 open-source models (Gemma-4-E2B-it, Gemma-4-E4B-it, Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B, Qwen3.6-27B), reporting compilation success, cBLEU, TED, DSim, RS, ECS, and generated code length.

4. TikZEditMix and a two-stage curriculum: A 32,448-sample mixed training set (20,244 image-to-TikZ reconstruction pairs filtered from DaTikZ-v3 plus 12,204 constructed editing samples) used to train Qwen3.5-2B and Qwen3.5-4B through reconstruction first, then editing. On Qwen3.5-4B this raises compilation success from 45.35% to 83.40% and yields an average improvement of 18.7 points across the proposed evaluation metrics.

Main Findings

• Current models remain unreliable. The 8 proprietary models average 75.15% compilation success, 58.76 RS, and 59.80 ECS, while the 6 open-source models reach only 39.95%, 23.49, and 21.99. The strongest proprietary model, Gemini-3.1-Pro, reaches 88.37% compilation success, 74.51 RS, and 75.02 ECS, still failing on about 12% of test samples.

• Code similarity does not track editing quality. Among proprietary models, cBLEU and TED vary within narrow ranges (13.65–18.64 and 48.12–50.45) while ECS spans 49.04–75.02. Gemma-4-E2B-it and Gemma-4-E4B-it have similar cBLEU (10.45 vs. 11.70) but ECS differing by 27.21 points (2.97 vs. 30.18).

• DSim overestimates fidelity and weakly matches human judgment. DSim yields Pearson correlations of only 0.496 with human RS and 0.230 with human ECS. In one failed Qwen3.5-9B case, the output retains edges that should be removed, omits requested edges, and introduces unintended changes, yet scores a DSim of 0.914 with an ECS of 0.

• Program completeness is a primary failure mode for compact models. Among compilation failures from Qwen3.5-2B, 4B, and 9B, 77.1%, 29.4%, and 36.5% have incomplete document structures. Failed outputs average 479, 176, and 194 lines versus 45, 62, and 54 lines for successful outputs.

• Better reconstruction does not automatically yield better editing. Scaling Qwen3.5 from 4B to 9B raises all-example RS from 26.10 to 32.35, but ECS only from 22.54 to 24.99. On the 461 samples both models rendered successfully, mean RS rises from 60.74 to 65.09 while mean ECS slightly falls from 50.23 to 49.22.

• Difficulty scales with program length and instruction complexity. Compared with 1–50-line source programs and instructions with one or two edit units, programs longer than 100 lines and instructions with five or six edit units raise the failure rate by 19.13 and 12.68 percentage points respectively, averaged over all 14 models.

• Some edit operations are much harder than others. Discrete-form and spatial/layering updates are the hardest: open-source ECS is 48.31 and 49.07 versus 74.90 and 74.27 for proprietary models, gaps of 26.59 and 25.20 points. Text/symbol and removal operations score 61.72 and 67.86 for open-source models and 84.46 and 90.53 for proprietary models, narrowing the gaps to 22.74 and 22.67 points.

• Reconstruct-then-edit gives no consistent gain; gold source code does. Relative to direct generation, reconstruct-then-edit changes compilation success by +1.48, +8.08, and −4.46 percentage points for Qwen3.5-4B, 9B, and Qwen3.6-27B; RS by −0.04, +4.35, and −4.32; and ECS by −0.82, +5.14, and −1.69. Replacing the reconstructed source with the gold pre-edit program raises compilation success by 17.83, 30.36, and 25.45 points, RS by 34.41, 49.81, and 42.62 points, and ECS by 15.04, 23.95, and 27.23 points — identifying source reconstruction as the main bottleneck.

• Iterative compiler feedback helps only sufficiently capable models. With up to six revisions driven by LaTeX compiler and renderer feedback, Qwen3.5-9B improves sharply within the first two revisions, whereas Qwen3.5-4B shows little improvement across all metrics.

• Two-stage curriculum training works, and stage two helps larger compact models more. On Qwen3.5-4B, compilation success rises from 45.35% to 83.40%, and the two-stage curriculum beats an equal-data single-stage ablation (83.40 vs. 75.84 compilation success, 48.72 vs. 45.67 RS, 37.32 vs. 36.33 ECS). From Stage 1 to Stage 2, 2B gains 2.39 points in compilation and 5.01 in ECS, while 4B gains 31.98 and 14.50.

• Training also improves general reconstruction. On the 442 publicly released DaTikZv2 test examples, the two-stage-trained Qwen3.5-4B reaches cBLEU 4.215, TED 56.193, and DSim 73.189, improving on its base checkpoint (2.945, 57.025, 68.168), surpassing base Qwen3.5-9B on all three metrics, and achieving the best cBLEU and TED among the compared models despite a 32k-sample training set versus roughly 360K examples for the four DeTikZify models.

• Human alignment of the evaluation framework is strong. With two human annotators and three model judges on 100 sampled cases, all reported correlation coefficients exceed 0.70. GPT-5.6-Terra achieves the lowest MAE for both RS (10.19) and ECS (12.82) and was selected as the final judge.

Methodology in Plain English

The researchers first built a source pool by collecting self-contained TikZ code blocks from recent arXiv TeX sources, compiling and rendering them, foreground-cropping the results, and removing near-duplicates with pHash using a shared source URI.

They then produced three kinds of edit samples. Real-world edits exploit the fact that figures within the same paper often share structure while differing in labels, objects, relations, styles, or layout; candidate pairs are found by code similarity and foreground-aware visual similarity, ranked and filtered to avoid reusing the same figure, then an instruction-generation model sees both renderings and both programs and writes the edit instruction, while an annotation model decomposes it into edit units labeled with one of the eight atomic operation types. Controlled synthetic edits come from unused source figures: a planner writes the instruction and edit units, an editing model applies them to produce a candidate program, which is compiled and checked by deterministic tests and a multimodal validator for compilability, instruction satisfaction, and non-target preservation, with at most two repair attempts for failures. Visual localization prompts add translucent red boxes over regions that language alone cannot identify unambiguously.

Human reviewers inspect every sample and reject those with imperceptible edits, damage to non-target content, ambiguous instructions, or inaccurate localization, while converting unnecessary visual prompts into text-only instructions.

For evaluation, the team reports compilation success rate, cBLEU, and TED for comparability with prior TikZ work, plus DSim (one minus DreamSim perceptual distance), and adds the two human-aligned scores RS and ECS. RS gives 30 points to text/style, 30 to objects/relations, and 40 to layout/scale, with a primary-error rule preventing the same failure from being penalized repeatedly; ECS averages per-edit-unit scores on the six-level scale, and both are set to zero for rendering failures. All outputs are compiled in a shared TeX Live environment, rendered, foreground-cropped, and resized to a maximum long-edge of 1,024 pixels.

Finally, they assembled TikZEditMix and trained Qwen3.5-2B and Qwen3.5-4B in two stages: one epoch on the reconstruction subset at a learning rate of 10⁻⁵, then one epoch on the editing subset at 5×10⁻⁶, on 8×A800 GPUs, with an equal-data single-stage ablation to isolate the effect of the curriculum from the effect of more data.

Why This Matters

The paper shows that a capability users might assume is nearly solved — asking a model to change one part of a scientific figure while leaving the rest intact — is in fact unreliable even for strong proprietary systems, and that the usual reconstruction-oriented metrics (cBLEU, TED, DSim) do not capture whether the requested edit actually happened. By separating edit correctness from non-target preservation and validating both against human annotators, the work provides a measuring stick that is aligned with what people actually care about when editing a figure.

Real-world applications:

• Researchers and students revising diagrams, circuit schematics, and model-architecture figures in papers when the original source file is missing or unusable.
• Publishers and editorial workflows that need figures regenerated in editable code form for production, correction, or reuse.
• Accessibility and archival work that requires converting raster figures into structured, editable vector descriptions.
• Assistants that help authors iteratively adjust labels, colors, or layout through natural-language requests rather than manual TikZ editing.

Industry relevance: The findings matter for teams building multimodal coding assistants and document tools, for LaTeX and scientific-publishing toolchains, and for anyone deciding whether compact on-device models can handle graphics-code editing. The results indicate that models below 9B parameters struggle with instruction following and complete figure generation, but that targeted two-stage curriculum training can substantially raise the floor: on Qwen3.5-4B, compilation success rises from 45.35% to 83.40%, an improvement of practical significance for deployment. The result that compiler and renderer feedback helps only sufficiently capable models (clearly helping Qwen3.5-9B but not 4B) is directly relevant to agent design.

Future Directions

• Improving source reconstruction, the identified bottleneck. Since replacing a reconstructed source program with the gold program increases compilation success by 17.83, 30.36, and 25.45 points and ECS by 15.04, 23.95, and 27.23 points across 4B,

Authors’ abstract

Although multimodal large language models (MLLMs) have shown substantial potential in visual understanding and graphic code generation, editing scientific figures through code presents a greater challenge: a model must jointly recover visual structure, ground the requested change, generate compilable code, and preserve all unrelated content. While existing TikZ benchmarks mainly focus on figure reconstruction and generation, few systematically evaluate instruction-guided scientific figure editing with compilable code. We introduce Edit2TikZ, a comprehensive benchmark for scientific figure editing tasks, featuring 1,548 diverse and high-quality samples. Edit2TikZ combines real-world and controlled synthetic edit cases, supports both textual and visual localization request, and contains multi-step editing, each with step-level annotations. We further construct a human-aligned evaluation framework to measure whether a requested edit is completed while irrelevant content is preserved. Utilizing Edit2TikZ, we evaluate 14 mainstream MLLMs and find that current systems remain unreliable: on average, proprietary models achieve a compilation success rate of merely 75% and remain limited in both figure restoration and edit correctness, while compact models below 9B struggle further with instruction following and complete figure generation. Therefore, we build a mixed training set TikZEditMix and adopt reconstruction-then-editing curriculum learning for compact models. On Qwen3.5-4B, this training improves the compilation success rate from 45.35% to 83.40% and yields an average improvement of 18.7 points across our proposed evaluation metrics. The code and data will be released at https://github.com/Solunny/Edit2TikZ.