VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning

AI summary

Overview

• Research area: Multimodal code generation — training vision-language models (VLMs) to produce executable code (Python, HTML, SVG, LaTeX, chemical notation) from images, using reinforcement learning.
• Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, GRPO-style reinforcement learning, vision transformers, and rendering-based evaluation.
• Scope in one sentence: The paper introduces VinciCoder, a unified 7B/8B multimodal code generation model trained on 1.6M supervised samples plus 42k reinforcement learning samples, with a "coarse-to-fine" visual reward that scores generated code by comparing rendered images to targets via DINOv2 embeddings.

Note: the supplied paper content is truncated partway through Appendix B.5, so results described there are only partially available.

What This Paper Is About

Existing multimodal code generation models tend to be specialists — one model for charts, another for webpages, another for SVG — and they are trained almost entirely with supervised fine-tuning, which does not guarantee that generated code actually runs or that its rendered output matches the input image. The paper's goal is a single unified model that generates code across five visual domains and that is further optimized by a reinforcement learning reward derived from visual similarity rather than brittle text-matching rules.

Key Contributions

1. A unified multimodal code generation framework (VinciCoder). Built on the Qwen-VL series via a two-stage SFT-then-RL pipeline, covering chart-to-code, web-to-HTML, image-to-SVG, image-to-LaTeX, and scientific-plot/molecule code.
2. A visual-based refinement task with 300k training samples. Unlike conventional editing, the model receives flawed code containing logical errors or partial renderings and must repair it so the rendered output aligns with the target image. The authors state this is a non-trivial task they pioneer.
3. Coarse-to-fine Visual Reinforcement Learning (ViRL). A reward mechanism that compares DINOv2 embeddings of global thumbnails and 448×448 image patches, providing an implementation-agnostic signal that is indifferent to semantically equivalent but syntactically different code. The authors claim VinciCoder is the first unified model to leverage RL for domain-agnostic visual fidelity in multimodal code generation.
4. Extensive benchmarking and ablations. Evaluation across ChartMimic, UniSVG-ISVGEN, Design2Code, Image2Struct, and ChemDraw, plus ablations on training stages, SFT data scale, refinement data, and reward formulation.

Main Findings

• State-of-the-art among open-source models: VinciCoder-8B reaches 91.6 execution rate / 78.9 low-level / 80.6 high-level on ChartMimic-direct-v2; 77.1 / 94.1 with an overall score of 87.3 on UniSVG-ISVGEN; 88.4 / 89.3 on Design2Code; 77.3 rendering success and 60.9 EMS on Image2Struct-plot; 88.3 execution rate and 62.6 Tanimoto similarity on ChemDraw. VinciCoder-7B scores 91.2 / 78.3 / 79.8, 77.0 / 92.0 / 86.0, 88.2 / 89.1, 84.7 / 60.9, and 87.5 / 56.0 respectively.
• Large margins over comparable base models: Qwen2.5-VL-7B scores 68.7 execution rate on ChartMimic versus VinciCoder-7B's 91.2. Qwen3-VL-8B scores 78.3 versus VinciCoder-8B's 91.6. On UniSVG-ISVGEN, Qwen2.5-VL-7B scores 47.5 low-level and 73.8 high-level, against 77.0 and 92.0 for VinciCoder-7B.
• Gaps remain against the strongest closed-source models on some metrics: Gemini-2.5-Pro reaches 97.3 / 88.7 / 83.8 on ChartMimic, Claude-4.5-Sonnet 97.8 / 89.6 / 82.9, and GPT-5 94.8 / 81.9 / 78.3. On ChemDraw execution rate, Claude-4.5-Sonnet scores 95.3 and GPT-5 93.8 versus 88.3 for VinciCoder-8B, though VinciCoder-8B's Tanimoto similarity of 62.6 exceeds GPT-5's 52.1 and Claude-4.5-Sonnet's 41.7.
• Both training stages matter, in sequence: In the ablation (VinciCoder-7B), the base model averages 50.3 on ChartMimic, 63.3 on UniSVG, 85.5 on Design2Code, 34.1 on Image2Struct, and 16.4 on ChemDraw. Adding ViRL without SFT raises these to 72.5, 70.3, 87.7, 57.6, but drops ChemDraw to 2.4. Adding SFT instead gives 81.1, 85.9, 86.7, 65.8, 70.4, and the complete two-stage model reaches 83.1, 86.0, 88.7, 72.8, 71.8.
• ViRL helps most when the SFT model is less saturated: On the full SFT checkpoint, ViRL adds 2.0, 0.1, 2.0, 7.0, and 1.4 points across the five benchmarks. On a 20% SFT checkpoint, ViRL adds 5.9, 10.1, 2.2, 17.2, and 2.4 points — relative gains that are markedly larger.
• Coarse-to-fine beats alternative rewards: Against an edit-distance textual reward (77.1 / 76.7 on ChartMimic low/high-level), an SSIM low-level visual reward (75.7 / 75.3), and an InternViT-300M semantic visual reward (77.9 / 78.1), the full coarse-to-fine ViRL achieves 78.3 / 79.8. Removing the coarse component gives 77.8 / 78.9; removing the fine component gives 77.0 / 78.0; removing the alignment reward gives 78.2 / 79.4.
• Multi-turn refinement works but is domain-dependent: Asked to repair non-executable first-turn code, the model fixed 12 of 66 ChartMimic cases (18.2%) and 8 of 85 UniSVG cases (9.4%). For already-executable code, high-level scores improved from 88.6 to 89.8 (+1.2) on ChartMimic and 91.0 to 92.3 (+1.3) on UniSVG.
• Refinement data improves even direct generation: Comparing SFT models with and without the 300k refinement tasks, the with-refinement model improves high-level ChartMimic from 76.4 to 78.6, UniSVG low-level from 75.6 to 78.2 and high-level from 89.4 to 91.0, and Design2Code low-level from 86.1 to 86.5 and high-level from 86.6 to 87.0. Low-level ChartMimic dipped slightly from 76.2 to 75.8.
• Comparison with task-specific models is mixed: On ChartMimic, MSRL scores 96.5 execution rate / 78.6 low-level / 83.8 high-level and ChartMaster 93.8 / 78.2 / 85.1, both above VinciCoder-8B's 91.6 / 78.9 / 80.6 on execution rate or high-level score. On UniSVG-ISVGEN, VinciCoder's 77.4 SSIM, 23.3 LPIPS, 92.0 CLIP score, and 86.0 overall score beat tuned LLaVA 1.5, Llama 3.2, and Qwen2.5-VL baselines.
• Training configuration: SFT for one epoch on 24 H800 GPUs with global batch size 96; GRPO in the RL phase with global batch size 256, 8 rollouts per query, ω_v = 0.9 and ω_l = 0.1, 16 GPUs for the policy model and 4 for reward scoring with DINOv2-L.

Methodology in Plain English

The authors first build a large supervised dataset by gathering open-source image-code collections and re-executing the code to render images, filtering out samples that fail or produce mismatched output. For charts they fix syntax errors in Chart2Code-160k with automated linters and pick 100k diverse samples from MSRL using perceptual hashing and mini-batch K-means clustering. For webpages they filter Web2M to English entries without hyperlinks or embedded images, yielding 60k entries. For SVG they use 360k image-SVG pairs from the UniSVG ISVGEN subset. For LaTeX they standardize all code inside a standalone TikZ environment and drop anything that errors or renders multiple pages. Scientific plots add document, molecule, diagram, table, graphic, and circuit categories, including 40k molecular pairs rendered from USPTO SMILES strings.

Crucially, they then create a second kind of training sample: they take a model trained on the initial data, let it produce first-pass code for 100k additional examples per domain, and pair that imperfect code with the target image and ground-truth code. Training on these pairs teaches the model to debug and refine its own flawed output — 100k refinement samples for charts, 92k for webpages, and 103k for SVG.

After supervised fine-tuning, they run reinforcement learning with GRPO. For each prompt the model generates 8 candidate code snippets. Each candidate is executed; if it renders, its rendered image is compared to the target using DINOv2 embeddings at two scales — a downsampled global thumbnail for layout and non-overlapping 448×448 patches for fine detail. The reward is the arithmetic mean of the global score and all patch scores, and it is set to 0 if rendering fails. A separate binary alignment reward checks whether the generated code is in the language the instruction requested, with alias handling such as mapping "tikz" to "latex". The two rewards are combined with weights 0.9 and 0.1.

Why This Matters

• Research impact: The paper argues that text-based verifiable rewards are ill-suited to code generation because many syntactically different programs are functionally identical, and demonstrates that perceptual rewards sidestep this problem. It also shows that a single model can cover chart, web, SVG, LaTeX, and chemistry code rather than requiring one specialist per domain.
• Real-world applications:
  • Recreating charts and scientific plots as editable code from screenshots.
  • Converting webpage designs or mockups into HTML.
  • Turning raster icons and illustrations into scalable SVG graphics.
  • Recovering LaTeX/TikZ source for figures extracted from papers, and reconstructing molecular structures from chemical images.
• Self-correction as a product feature: The refinement task means the model can take broken code plus a reference image and repair it — useful in iterative design tools and automated debugging pipelines.
• Industry relevance: The work comes from Meituan and the Chinese Academy of Sciences, and the paper releases data, code, and model weights at a public repository, positioning it as an open alternative to closed models like Gemini-2.5-Pro and GPT-5 for multimodal code generation. The authors also flag practical costs: rendering and generating many RL samples demands substantial computing power, which they say limits scaling to very long code files or high-resolution images.

Future Directions

• Extending the reward beyond static images to interactive or animated elements in web code such as HTML and JavaScript, which the authors explicitly identify as unaddressed.
• Preventing visual reward hacking in knowledge-poor domains. The authors note that on ChemDraw the visual reward can drive the model to produce shapes that look like the target molecule but are logically incorrect, because a perceptual signal cannot supply missing domain knowledge.
• Reducing the computational cost of rendering and multi-sample RL so the approach scales to longer code and higher-resolution inputs.
• Determining how much SFT is actually needed. The 20% SFT ablation shows the largest ViRL gains, leaving open the question of the optimal balance between supervised initialization and reinforcement learning.

Target Audience

Researchers and engineers working on vision-language models, code generation, or reinforcement learning with verifiable rewards will find the reward-design ablations and the SFT-versus-RL comparisons most useful. Practitioners building chart-to-code, design-to-HTML, or image-to-SVG tools will benefit from the benchmark numbers and the multi-turn refinement results. Readers seeking a labeled dataset or an open-weight baseline for unified multimodal code generation can use the released corpus of 1.6M SFT samples and 42k RL samples.

Authors’ abstract

While recent specialized multimodal code generation models excel in tasks like chart-to-code generation, their reliance on single-task training limits generalization and hinders the development of \textbf{VI}sio\textbf{N} \textbf{C}ode \textbf{I}ntelligence. In this work, we introduce \textbf{VinciCoder}, a unified framework designed for generalized multimodal code generation. We first curate a large-scale SFT corpus comprising 1.3M direct generation pairs and 300k visual-based refinement tasks. This composition fosters self-refinement capabilities, enabling the model to directly rectify code to align with input images. Subsequently, we propose coarse-to-fine Visual Reinforcement Learning (ViRL) to overcome the brittleness of textual metrics in handling semantically equivalent but syntactically diverse code. By quantifying visual similarity across multi-scale patches, ViRL provides an implementation-agnostic reward mechanism that ensures high-fidelity alignment between rendered outputs and input visuals. Extensive experimental results across diverse benchmarks demonstrate that VinciCoder achieves superior performance, while comprehensive ablation studies validate the effectiveness of our proposed ViRL strategy. The data, code and model are available at https://github.com/DocTron-hub/VinciCoder.