VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation

AI summary

Overview

• Research area: Multimodal computer vision and code generation — specifically, evaluating whether vision-language models (VLMs) can represent natural images as executable SVG code.
• Technical level: Intermediate. The paper is readable without deep implementation knowledge, but assumes familiarity with VLMs, multimodal benchmarks, and rendering/execution concepts.
• Scope: The paper introduces VCode, a 464-item benchmark that recasts multimodal image understanding as SVG code generation, plus CodeVQA, a render-then-answer evaluation protocol, and VCoder, an agentic framework that improves SVG fidelity through iterative revision and external vision tools.

What This Paper Is About

Existing coding benchmarks are mostly linguistic-centric (program synthesis, debugging, competitive programming), and existing image representations rely on dense RGB pixels that capture appearance but offer little symbolic abstraction. This paper asks whether visual representation can move beyond raw pixels toward compact, interpretable, executable SVG code that preserves the symbolic meaning needed for downstream reasoning. The goal is to measure how faithfully frontier VLMs can convert a natural image into SVG, and to improve that fidelity with an agentic wrapper.

Key Contributions

1. VCode: a multimodal coding benchmark. Given an image, a model must generate SVG code that reconstructs the image's symbolic structure. It repurposes three sources spanning General commonsense (MM-Vet), College-level disciplines (MMMU), and Visual-centric perception (CV-Bench), totaling 464 image-question pairs.
2. CodeVQA: a symbolic-fidelity evaluation protocol. A policy model answers the original image's question using only the rendered SVG; correct answers indicate the SVG faithfully preserved the semantics needed for reasoning.
3. VCoder: an agentic framework for stronger coders. It augments VLMs along two axes — Thinking with Revision (iteratively comparing intermediate renderings against the original and refining the SVG) and Acting with Visual Tools (detectors and parsers supplying structured cues about objects, shapes, and text).
4. Evaluation and insights. Extensive experiments, ablations on tools/revision/input modality, and human studies comparing humans and VLMs on original images versus rendered SVGs.

Main Findings

• Frontier VLMs struggle with visual-centric coding. GPT-5 achieves the best SigLIP score (72.3) and the highest CodeVQA overall (46.8), yet this remains well below the original-image upper bound of 61.7, indicating substantial headroom.
• Stronger reasoning yields better visual coding. Closed-source models consistently outperform open-source counterparts, and models with stronger reasoning produce more faithful symbolic renderings. Semantic similarity (SigLIP) and CodeVQA are positively correlated.
• SVG specialists underperform. OmniSVG and StarVector rank lowest — StarVector has an 8.3% success rate — which the authors attribute to low success on long-context outputs and a gap between neatly authored SVG corpora and SVGs derived from natural images.
• Knowledge is the hardest dimension. In MM-Vet, the Knowledge dimension is consistently the lowest, reflecting the compounded difficulty of recalling facts and then encoding them faithfully in SVG.
• Professional disciplines are hard to differentiate. On MMMU, models cluster within a narrow, modest band and most fail the more demanding disciplinary settings.
• Vision-centric perception is tough. CV-Bench scores hover near the 50% random baseline, especially on 3D relations (depth or spatial ordering).
• VCoder delivers a +12.3-point overall gain. Built on Claude-4-Opus, VCoder lifts Overall from 41.7 to 54.0, improving all three domains: MM-Vet 54.2 (+16.7), MMMU 48.6 (+6.2), CV-Bench 61.3 (+11.4).
• Code token length correlates with expressiveness. Short-output models underperform (Qwen2.5-VL-72B emits 0.3K tokens), while stronger models such as GPT-5 and Gemini-2.5-Pro produce substantially longer sequences (often >2K tokens) and score higher. Length alone is not sufficient, but performance scales with usable context.
• Vision tools produce steady gains. Adding location and category cues raises the MM-Vet average to 39.7; adding shape reaches 43.3 (notably helping spatial reasoning); adding text reaches 41.5; the full vision-tool ensemble reaches 54.1, a 16.6-point improvement over Claude-4-Opus.
• Revision helps, but depends on reasoning strength. GLM-4.5V shows the most substantial gains from the first revision, which the authors attribute to its built-in thinking mode; GPT-4o initially struggles on the first revision but improves in later rounds.
• Language as an intermediate representation helps. Img2SVG gives the weakest results (SigLIP 65.6, Overall 42.5); Img2SVG-Thinking is low too (69.8, 43.5); Img2Text2SVG achieves the best performance (68.5, 46.4).
• Humans and VLMs show aligned degradation on SVGs. On original images, models substantially surpass human performance (50.4 for humans vs. 75.5 for GLM-4.5V on MM-Vet); on SVG representations, all models drop, with the human score falling to 40.6, and both humans and VLMs exhibit a form of alignment when interpreting symbolic representations.
• DINO-V2 features score lower than SigLIP2. The authors attribute this to DINO's focus on low-level visual representation versus SigLIP2's semantic space (for example, VCoder scores 72.3 with SigLip2 versus 33.0 with DINO-V2).

Methodology in Plain English

The researchers took three existing multimodal benchmarks and converted them into a code-generation task: a model sees an image and must output SVG code, with no extra query guidance. They curated the data into three scenario types — commonsense perception (all 218 MM-Vet image-question pairs), professional knowledge (MMMU development set filtered to single-image VQA instances, yielding 146 pairs), and visual-centric perception (a balanced 100-pair CV-Bench subset built by shuffling and interval selection, distributed as spatial relationship 20, object count 20, depth order 30, and relative distance 30). Together this yields 464 image-question pairs.

Evaluation uses three signals. SigLIP score measures embedding cosine consistency between the original image and the rendered image using a pretrained visual encoder (siglip2-so400m-patch14-384). CodeVQA renders the generated SVG and asks a policy model to answer the original question from the rendering alone, with correctness judged by rule-based matching in multiple-choice settings and LLM-as-Judge for open-ended questions. Code token length (measured with tiktoken's cl100k_base encoding) captures representational efficiency.

To improve performance, VCoder adds two test-time strategies. Thinking with Revision runs a loop: the model comments on differences between the original image and the intermediate rendering, then regenerates the SVG conditioned on those differences plus the current code, repeating for t iterations. Acting with Visual Tools supplies external perception cues: Florence-2 for object categories and bounding boxes as absolute coordinates with image width and height, SAM-2 for segmentation masks downsampled into sparse coordinate points to represent irregular contours, and OpenOCR to detect and transcribe text regions into native SVG <text> tags.

Baselines span proprietary models (Claude-4.5-Sonnet, Claude-4-Opus, Claude-4-Sonnet, GPT-5, GPT-4.1, GPT-o3, GPT-4o, GPT-4o-mini, Gemini-2.5-Pro, Gemini-2.5-Flash, Seed-1.6-thinking) and open-source models (LLaMA-4-Scout, Qwen3-VL, Qwen2.5-VL-72B and 7B, InternVL3.5-241B-A28B, Intern-S1, InternVL3-78B, MiniCPM-V-4.5, GLM-4.5V, GLM-4.1V-Thinking, OmniSVG, StarVector). All models are queried under a unified prompting interface; the primary automatic evaluator is GPT-4o-mini. Implementation runs in PyTorch on an NVIDIA RTX 4090 GPU with 24GB of memory, with maximum output length 16,384 tokens (8,192 for Qwen2.5-VL models).

Why This Matters

The paper argues that code is a precise, structured, executable medium for reasoning and action in the agent era, but progress has concentrated on language-centric coding while visual-centric coding remains underexplored. By showing that strong linguistic reasoning does not automatically translate into faithful visual code, and by showing that humans and VLMs degrade in similar ways when reasoning over rendered SVGs, the work positions symbolic visual representation as a potential pathway toward more human-aligned multimodal intelligence.

Real-world applications implied by the paper's framing (these are directions the paper motivates rather than systems it builds and tests):

• Agentic interaction with digital environments, where code is used to interface with tools or accomplish complex tasks, particularly in embodied agents.
• Generation of synthetic visual assets such as charts, diagrams, websites, icons, and logos, where SVG is already a standard format.
• Compact, interpretable, executable image representations that could stand in for dense RGB pixels in pipelines requiring symbolic structure such as spatial relationships, object counts, and outlines.
• Vision-centric perception tasks involving counting, distance estimation, and relative spatial relationships in 2D or 3D, which the paper highlights as especially challenging for current models.

Industry relevance: the results quantify where frontier commercial and open-source VLMs fail on visual code generation, provide a concrete evaluation protocol (render, then ask) that vendors can adopt, and demonstrate that a test-time wrapper with off-the-shelf detectors and segmenters recovers meaningful gains without retraining the underlying model.

Future Directions

• Developing end-to-end vision-language coders with scalable training data, as the conclusion explicitly proposes, to produce more faithful symbolic representations without test-time scaffolding.
• Closing the long-context bottleneck. The paper identifies long-context reasoning and generation as a central bottleneck, since fully representing an image requires thousands of tokens and performance scales with usable context.
• Improving 3D and fine-grained relational reasoning. CV-Bench scores hover near random even with VCoder, leaving substantial headroom on depth and spatial relations, and the paper notes models fail on fine-grained relations such as near versus far.
• Enabling coders to autonomously call tools. The paper observes a 16.6-point improvement from the full vision-tool ensemble and states this implies strong potential for the Coder itself to autonomously invoke tools and leverage contextual information.

Target Audience

Researchers and practitioners in multimodal AI, vision-language model development, and agentic systems who want to evaluate or improve how models convert natural images into executable symbolic code. It is also relevant to benchmark designers interested in evaluation protocols that test representational fidelity rather than answer accuracy alone, and to engineers building SVG generation, UI/chart/diagram synthesis, or tool-augmented visual agents. Readers need a working understanding of VLMs and multimodal benchmarks; the paper is accessible to graduate students and applied engineers, and the benchmark and code are released at https://github.com/CSU-JPG/VCode.

Authors’ abstract

Code has emerged as a precise and executable medium for reasoning and action in the agent era. Yet, progress has largely focused on language-centric tasks such as program synthesis and debugging, leaving visual-centric coding underexplored. Inspired by how humans reason over sketches, we advocate SVG code as a compact, interpretable, and executable visual representation. We introduce VCode, a benchmark that reframes multimodal understanding as code generation: given an image, a model must produce SVG that preserves symbolic meaning for downstream reasoning. VCode covers three domains - general commonsense (MM-Vet), professional disciplines (MMMU), and visual-centric perception (CV-Bench). To assess symbolic fidelity, we propose CodeVQA, a novel evaluation protocol in which a policy model answers questions over rendered SVGs; correct answers indicate faithful symbolic preservation. Empirically, frontier VLMs struggle to generate faithful SVGs, revealing a persistent gap between language-centric and visual-centric coding. To close this gap, we introduce VCoder, an agentic framework that augments VLMs along two axes: (i) Thinking with Revision, which iteratively analyzes discrepancies and refines SVG code; and (ii) Acting with Visual Tools, where detectors and parsers supply structured cues such as objects, shapes, and text beyond the model's intrinsic capacity. Across benchmarks, frontier VLMs with strong reasoning capabilities score well overall yet remain limited in professional knowledge and 3D reasoning. VCoder delivers a 12.3-point overall gain over the top-performing Claude-4-Opus. Human studies show that both humans and VLMs perform worse on rendered SVGs, their consistency reveals the promise of symbolic visual representation. The benchmark and code are available at https://github.com/CSU-JPG/VCode.