How Far Can Coding Agents Go in Visual Generation?

AI summary

Overview

Research area: Computer vision and agentic AI — specifically, program-based (code-driven) image generation by general-purpose coding agents, compared with native text-to-image models.

Technical level: Intermediate.

Scope: The paper benchmarks how well frontier coding agents can draw images zero-shot through executable drawing interfaces, tests whether code can act as a medium for visual reasoning, and probes for behavioral evidence of a shared internal visual representation across interfaces and viewpoints.

What This Paper Is About

Modern image generation is dominated by specialized generative models (latent diffusion, rectified-flow transformers, autoregressive models) that leave the construction process implicit inside learned parameters. This paper asks an alternative question: how far can general-purpose coding agents go at building images by writing and executing programs — pixels, shapes, Canvas 2D JavaScript, SVG, a mouse-driven paint GUI, or Blender scenes — with no task-specific training. Beyond raw drawing quality, the authors ask whether visual code can also serve as a medium the agent uses to reason about scenes, and whether behavior across different drawing interfaces hints at a shared internal representation of scene content.

Key Contributions

1. A characterization of coding agents as zero-shot drawers. Using a new benchmark and six-plus drawing harnesses, the authors map where agents succeed (structural fidelity, precise constraints, legible design) and where they fail (perceptual realism, anatomy, materials, light).

2. AgentDrawBench, a 200-prompt benchmark spanning seven prompt families, assembled from four published benchmarks (110 from Arena-T2I-Hard, 40 from HPDv3, 35 from BizGenEval, 15 from GraphicWeaver), with structural, perceptual, and holistic evaluation protocols. The authors state they will publicly release it.

3. Code as an "external visual language of thought." The paper shows agents can construct images from code alone without seeing renders (83.0% structural fidelity), and that reconstructed SVG source supports visual question answering — 69.0% accuracy from code versus 65.0% from its render and 74.0% from the original image, with code beating the original image in the academics and digital-world domains.

4. An "internal visual language-of-thought" hypothesis with three behavioral probes: cross-harness consistency of independent drawings, images assembled from independent per-pixel queries, and spatial coherence in self-furnished rooms across viewpoint changes.

Main Findings

• Structure is strong; realism is weak. GPT-6-Astra with Canvas 2D reaches 85.8% overall structural fidelity on AgentDrawBench versus 68.8% for gpt-image-1 (17 percentage points higher) and 90.7% for gpt-image-2. On design and symbolic prompts, DeepSeek V4.1 Flash reaches 89.5%, exceeding gpt-image-1 by 31.4 points, and Astra with Canvas 2D reaches 94.6%, slightly above gpt-image-2's 94.4%.

• Perceptual win rates stay below 50% on naturalistic dimensions. Against GPT-image-1, Astra with Canvas 2D and SVG achieves typography win rates of 77.6% and 74.8%, but all reported configurations remain below 50% on anatomy, perspective, light transport, material, and scene physics (Astra with Canvas 2D anatomy: 15.4%).

• Holistic preference is sharply task-dependent. Astra with Canvas 2D wins 92.0% of design/symbolic comparisons against GPT-image-1, but 13.0% on scenes and 6.0% on people/organic subjects, for an overall 47.7% win rate (ties counted as half a win). Judging uses gemini-3.5-flash with the MMRB2 rubric.

• Visual feedback helps structure only modestly. Blind drawing (execution feedback but no rendered canvas) achieves 83.0% structural fidelity versus 85.6% with visual feedback.

• Reconstructed code is a usable representation for question answering. On 200 WorldBench items sampled across seven domains: 69.0% from SVG source, 65.0% from the rendered reconstruction, 74.0% from the original image. The paper's example shows a model counting four buttons from code while answering three from the photograph.

• Extra test-time compute gives little. Extending refinement to a 4× budget (B = I + 5O, up to 48 model responses versus 12) raises structural fidelity from 85.6% to 86.8% and yields only a 50.5% holistic win rate against the standard run.

• Drawing procedures supplied in context change the look. Procedures distilled from human demonstrations (stroke operations, construction order, value grouping, edge control) let the agent reconstruct reference images in Chinese ink, charcoal, watercolor, and oil painting, producing medium-specific effects. This part is evaluated qualitatively.

• Drawings agree across interfaces. For two different 2D harnesses (Canvas 2D, SVG, Shapes) on the same prompt: mean layout correlation 0.60, palette cosine 0.81, DreamSim 0.19, close to two independent SVG runs (0.66, 0.84, 0.14). The mouse-driven GUI sits inside this cluster (0.56, 0.79, 0.22); Blender is further away (0.34, 0.69, 0.29) but far above the different-prompt null (0.01, 0.31, 0.77). A ten-prompt pilot without visual feedback preserved the ordering (0.42–0.49 versus a 0.03–0.08 null).

• Independently queried pixels assemble into recognizable wholes. Requesting the color of one pixel at a time on a 128×128 canvas (16,384 answers, each in a fresh context with no access to other answers) yields the coarse composition of a Mona Lisa, sunset, red apple, and peace sign — regions and tones agree, fine detail does not.

• Agents keep a room arrangement they chose themselves. In 30 ProcTHOR-10K validation rooms with 7–14 furniture pieces and six turns, deleting the agent's own SVGs after every turn still gives revisit piece-set Jaccard 0.99, relation agreement 0.96, and 99% agreement with its own floor plan (chance 0.31); box IoU is 0.71 versus exactly 1.00 when drawings are kept. Also deleting the furniture list leaves 0.88, 0.79, and 0.99. Fresh calls per view drop to 0.71, 0.61, and 0.24.

Methodology in Plain English

The authors give a frontier coding agent a text prompt plus documentation for one drawing interface, and let it work over multiple turns, optionally seeing its own renders.

• Interfaces (harnesses): direct pixel assignment, ordered geometric shapes, executable Canvas 2D JavaScript, declarative SVG, mouse-and-keyboard control of the JS Paint GUI, and Blender Cycles 3D scene scripts. A separate workflow supplies human-derived stroke procedures for ink, charcoal, watercolor, and oil.
• Benchmark scoring: structural fidelity is measured by checklists of yes/no/unclear questions about objects, attributes, counts, spatial relations, text, and layout, answered by a vision-language model and scored as the proportion of "yes." Perceptual quality uses ten dimensions compared pairwise against GPT-image-1, with positions swapped to cancel position bias (a win requires agreement across both orderings). Holistic quality uses the MMRB2 pairwise judge prompt, also with swapped positions, scoring (W + 0.5T)/(W + T + L).
• Code-as-representation tests: the agent draws blind (execution output only, no RGB canvas), and separately reconstructs benchmark images as SVG, then answers questions from code alone, from the render, or from the original image in fresh contexts.
• Consistency probes: independent drawings of the same prompt across harnesses are compared using a 16×16 grayscale luminance fingerprint (Pearson correlation), an 18-bin saturation-weighted hue histogram (cosine similarity), and DreamSim distance. A per-pixel query protocol asks for one coordinate's color per fresh session and assembles the answers.
• Coherence probe: the agent furnishes a room, draws it from a sequence of headings, and ends with a floor plan; agreement is scored geometrically from its own SVGs, with earlier drawings, the furniture list, or both removed in different conditions.

Why This Matters

Research impact. The paper reframes image generation as program synthesis rather than sampling from a learned distribution, and it introduces behavioral probes (cross-interface consistency, independent pixel queries, spatial coherence) for questions about internal representation that cannot be answered by inspecting weights. It also documents a clear capability split — structural precision without perceptual richness — that motivates work on interfaces and procedures rather than scale alone.

Real-world applications:

• Diagram, chart, logo, and layout generation, where explicit structure and legible typography matter more than photorealism (the design/symbolic category where agents win 92.0% holistically).
• Editable, inspectable graphics: SVG and Canvas output is structured and revisable, unlike a sampled raster image.
• Accessibility-adjacent and low-resource settings where an agent with a code interpreter substitutes for a dedicated image-generation model.
• Art-direction workflows using medium-specific procedures (ink, charcoal, watercolor, oil) for stylized illustration.
• Scene and spatial layout authoring, given the demonstrated ability to hold a furniture arrangement consistent across viewpoints.

Industry relevance. The results suggest coding agents are not drop-in replacements for native image generators overall (47.7% holistic win rate against GPT-image-1, 6.0% on people), but are competitive in constrained design domains — a signal for teams deciding whether to route requests to a generator or to a program-writing agent.

Future Directions

• Broader evaluation across models, interfaces, and transformation types; the authors note most ablations use a single primary agent.
• Stronger human verification, since current evaluations rely largely on automated VLM judgments, and the in-context learning experiments are only assessed qualitatively.
• Improving perceptual realism within code-based drawing — the persistent gap on anatomy, material, light transport, perspective, and scene physics.
• Extending the internal-representation probes: the pixel-query study covers five subjects, and the room experiment involves an agent with retained, opaque reasoning history.
• Testing whether the internal visual language-of-thought hypothesis holds when model weights or provider-internal agent implementations are accessible, which the authors could not do.

Target Audience

Researchers and engineers working on agentic systems, programmatic/vector image generation, and multimodal reasoning; benchmark designers interested in structural versus perceptual evaluation of generated images; and practitioners weighing coding agents against dedicated text-to-image models for design-oriented tasks. Readers should be comfortable with standard text-to-image terminology and basic evaluation methodology.

Authors’ abstract

Modern image generation largely relies on specialized generative models that leave the process of image construction implicit in learned parameters. We explore an alternative: general-purpose coding agents that construct images through executable programs. Our study of zero-shot drawing on AgentDrawBench, a 200-prompt benchmark, reveals a distinctive capability profile. Agents excel at preserving structure and satisfying precise visual constraints, achieving competitive results in domains where compositional accuracy matters more than photorealistic appearance, while remaining limited in perceptual richness. Their visual expressiveness varies with the drawing interface, and a qualitative study on reconstruction shows how procedural skills extracted from human demonstrations support different artistic media. We then examine code as an external visual language of thought: a structured, executable representation that agents can construct and inspect. We test it in visual question answering task and find reconstructed visual code retains substantial information for question answering and, in some domains, supports higher accuracy than the original image. These abilities raise a further question: do agents draw on a shared internal representation of scene content beyond a particular drawing program? We explore an internal visual language-of-thought hypothesis, in which a shared, compositionally structured representation supports visual operations across drawing languages. Cross-harness drawings and images assembled from independent single-pixel queries exhibit related layouts and tones. Agents also maintain self-generated room arrangements across viewpoints after visible drawings are removed. These results demonstrate that the coding agent would maintain a consistent and coherent world inside its imagine even before the harness gets involved and the generation starts.