JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence

AI summary

Overview

Research area: Neural code intelligence — specifically multimodal code generation, where models produce code from text, images, or both, and the code in turn produces visual output (charts, webpages, animations).

Technical level: Advanced. The paper assumes familiarity with LLM/VLM training, instruction tuning, reward modeling, AST-based code analysis, and rendering/execution sandboxes.

Scope in one sentence: The paper introduces a data synthesis toolkit, the JanusCode-800K multimodal code corpus, and the JanusCoder / JanusCoderV model family (7B–14B) that unifies text-centric and vision-centric code generation for visual outputs.

What This Paper Is About

Code intelligence has largely focused on text-only source code, but programs also produce rich visual artifacts — charts, interactive web UIs, and code-driven animations. Progress on connecting code to its visual expression is blocked by two things: existing models are built as narrow specialists (one for chart-to-code, another for WebUI-to-code) that do not generalize, and there is a shortage of high-quality, diverse multimodal code data. This paper attacks both problems at once, releasing a synthesis toolkit and the largest multimodal code corpus to date, and training a unified model family on top of it.

Key Contributions

1. A versatile, released data synthesis toolkit that automatically produces multimodal code data across heterogeneous domains and programming languages — charts, Web UIs, visual artifacts, and code-driven animations — using four strategies (Guided Evolution, Re-contextualization, Reverse Instruction, Bidirectional Translation) plus sandbox execution validation and LLM/VLM reward-based quality control.

2. JanusCode-800K, described as the largest multimodal code intelligence corpus to date, containing task categories not present in prior work such as large-scale animation and artifact data.

3. JanusCoder and JanusCoderV, a unified visual-programmatic interface trained on that corpus, where JanusCoderV uses the full corpus and JanusCoder is trained exclusively on the text-centric portion.

4. A comprehensive evaluation across seven established and newly proposed benchmarks, including DTVBench, a new benchmark for dynamic theorem visualization built from Manim and Wolfram Mathematica, with 102 manually curated tasks (52 Manim animation tasks

Authors’ abstract

The scope of neural code intelligence is rapidly expanding beyond text-based source code to encompass the rich visual outputs that programs generate. This visual dimension is critical for advanced applications like flexible content generation and precise, program-driven editing of visualizations. However, progress has been impeded by the scarcity of high-quality multimodal code data, a bottleneck stemming from challenges in synthesis and quality assessment. To address these challenges, we make contributions from both a data and modeling perspective. We first introduce a complete synthesis toolkit that leverages reciprocal synergies between data modalities to efficiently produce a large-scale, high-quality corpus spanning from standard charts to complex interactive web UIs and code-driven animations. Leveraging this toolkit, we construct JanusCode-800K, the largest multimodal code corpus to date. This powers the training of our models, JanusCoder and JanusCoderV, which establish a visual-programmatic interface for generating code from textual instructions, visual inputs, or a combination of both. Our unified model is a departure from existing approaches that build specialized models for isolated tasks. Extensive experiments on both text-centric and vision-centric coding tasks demonstrate the superior performance of the JanusCoder series, with our 7B to 14B scale models approaching or even exceeding the performance of commercial models. Furthermore, extensive analysis provides key insights into harmonizing programmatic logic with its visual expression. Our code and checkpoints are available at https://github.com/InternLM/JanusCoder.