PrismaCoder: Pushing the Boundaries of Open Multimodal Code Intelligence through Decomposed Rubric Synthesis

AI summary

Overview

Research area: Multimodal code intelligence — generating code (charts, web pages, SVG, LaTeX, animations) that is judged by what it renders, not just by whether it executes.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, reinforcement learning for LLMs (GRPO/GSPO), LLM-as-judge evaluation, and rendering/execution sandboxes.

Scope: The paper introduces PrismaForge, a rubric-synthesis pipeline that acts as both data filter and RL reward; uses it to build the PrismaCoder-2M dataset; and trains and releases the PrismaCoder-9B and PrismaCoder-35B-A3B models, which are benchmarked against open and proprietary systems.

What This Paper Is About

Models are increasingly asked to turn a text description or a screenshot into code whose output is visual — a chart, a webpage, a diagram, an animation. For such tasks, code that merely runs can still produce a wrong-looking artifact, and no existing quality signal captures program correctness, build quality, and visual fidelity at once: execution returns only pass/fail, code similarity does not predict rendered appearance, and task-specific verifiers do not transfer across formats. The paper's goal is to build an open, general-purpose data pipeline and model family that close this gap against proprietary systems.

Key Contributions

1. Decomposed Rubric Synthesis (PrismaForge). A framework that generates one instruction-specific rubric per sample, splitting it into separately scored code rubrics (checked against the generated program) and visual rubrics (checked against the rendered artifact), each combining domain-shared criteria with sample-specific ones, weighted by importance, and sized to an estimated task difficulty level.
2. PrismaCoder-2M. A 2M-sample open dataset for multimodal code intelligence spanning more than 12 categories and 10 programming languages, covering 13 of 15 visual data types tracked in the paper's comparison table (versus 8 for JanusCoder, 2 for VisCodex, 9 for VisCoder2, and 7 for VinciCoder).
3. Two open models. PrismaCoder-9B and PrismaCoder-35B-A3B, post-trained from Qwen3.5-9B and Qwen3.5-35B-A3B, released alongside the code and data.
4. Empirical analyses. Ablations showing rubric-based filtering beats direct holistic judging, that the pipeline transfers to other backbones (Gemma-4-31B-it and Qwen3-VL-30B-A3B-Instruct), and that rubric scores agree with human annotators better than direct scores do.

Main Findings

• Large margins on vision-centric web tasks: On front-end generation, PrismaCoder-9B and PrismaCoder-35B achieve judge scores more than 20 points higher than those of task-specific open baselines. PrismaCoder-9B also scores above proprietary models on DesignBench generation and on the WebCode2M CLIP score.
• One consistent weakness: DesignBench editing is the only web-interface metric on which both PrismaCoder models score below all Claude 4.6 and Gemini 3 variants.
• Chart-to-code leadership: On ChartMimic, PrismaCoder-35B reaches 83.42 (low-level) and 80.69 (high-level), leading all open baselines under both settings; PrismaCoder-9B improves the most over its own base model (58.35 → 81.57 low-level).
• Chemical structures: PrismaCoder-35B parses every output (100.00% validity) and reaches the highest Tanimoto similarity in the table (97.56).
• Long-tail visual languages: On VisPlotBench, PrismaCoder-35B leads open baselines on all three metrics, with a Mean of 62.47 — 11.02 points above the best open baseline and comparable to Claude Sonnet 4.6.
• Interactive plots: On InteractScience's programmatic lens (PFT), PrismaCoder-35B scores 28.00, above the best open baseline and comparable to Gemini 3 Flash; scaling from 9B to 35B raises this score by 3.20 points.
• Rubric filtering beats direct judging: Two equal-sized subsets from the same pool, one filtered by a single holistic score and one by PrismaForge rubrics, produce models where the rubric-filtered one wins on every benchmark — by as much as 24 points on Vega-Lite, while Design2Code moves by a fraction of a point.
• Backbone generalization: Applying the same data and method to Gemma-4-31B-it and Qwen3-VL-30B-A3B-Instruct on Python plotting and HTML, all twelve metrics end above the base model and RL adds to SFT on every one. SFT carries most of the ChartMimic gain; RL drives most of the HTML gains and recovers metrics that dip after SFT alone.
• Human agreement: In a study of 100 sampled training examples rated by human annotators and three proprietary model judges, rubric-based judging agrees substantially more with humans — in both absolute score level and pairwise ordering — than direct scoring.
• Diagnostic example (Figure 3): A generated page whose footer overlaps the Services section and clips a third button passes 5/5 code checks and receives a holistic judge score of 78/100, but the vision rubric scores it 0.18.

Methodology in Plain English

The scoring idea. Instead of asking a judge model for one overall number, PrismaForge writes a custom checklist for each training instruction. Part of the checklist is reusable per domain (e.g., "does the code capture the major visible regions?"), part is written fresh for that specific instruction, and the number of items is scaled to an estimated difficulty level. Half the checklist is answerable from the source code alone; the other half requires looking at the rendered output. Each item gets a weight (the prompt template specifies 0.8–1.0 for critical, 0.4–0.7 for important, 0.1–0.3 for minor), and weighted scores are averaged within each modality then combined into one overall score with weights λ_c and λ_v that sum to 1.

Building the dataset. Seeds come from four complementary groups: public plotting corpora (Python, SVG, Mermaid, Vega-Lite), scientific artifacts (GitHub repositories and arXiv figures, split into self-contained units via an AST-based pass), front-end sources (WebCode2M, the MCD dataset, interactive demos), and symbolic notation (PubChem chemical structures, LaTeX, Asymptote, LilyPond, Manim scenes from 3Blue1Brown video code). Each candidate must compile or render through a per-format sandboxed "gatekeeper." Candidates are then scored and routed: below τ₁ are discarded, between τ₁ and τ₂ are sent back for re-synthesis with the failed rubric items as feedback, and at or above τ₂ are kept. Distinct models are used for rubric generation and for judging so that biases in rubric construction are not reproduced during scoring.

Reinforcement learning. RL uses a held-out set that SFT never sees. For Python, HTML, and SVG, the authors hold out 20K medium-or-hard candidates per domain, sample K = 5 rollouts from the SFT policy, score each rollout with its rubrics, and keep the 5K instances with the highest score dispersion U_i — i.e., prompts where the policy sometimes succeeds and sometimes fails, which give group-relative optimization a usable contrast. Failed rollouts receive reward 0. Optimization uses the standard GRPO clipped objective, with a GSPO objective plus Rollout Routing Replay for the MoE models to stabilize expert training.

Models and pipeline components. PrismaCoder-9B and PrismaCoder-35B-A3B are post-trained from Qwen3.5-9B and Qwen3.5-35B-A3B. Claude-Opus-4.6 serves as the curation model, Gemini-3-Pro as the rubric generator, and Gemini 3 Flash as the judge. Baselines include JanusCoderV, VinciCoder, and VisCoder2-7B. Released data is decontaminated against every benchmark test set by removing training samples whose instructions share a long n-gram with a test instruction.

Why This Matters

Impact on research. The paper releases data, pipeline, and models together, addressing a structural problem in this subfield: open efforts have each covered one corner (plotting, chart-to-code, screenshot-to-page) with their own recipe, while proprietary systems keep data and training closed. PrismaForge also offers a reusable correctness signal for tasks where execution is not a proxy for quality — something that neither binary execution tests nor task-specific verifiers provide.

Real-world applications:

• Automated generation of charts and data visualizations from natural-language or image specifications.
• Screenshot-to-webpage and front-end prototyping, including editing existing pages.
• Document and scientific figure production in LaTeX, Asymptote, LilyPond, and Mermaid.
• Chemical structure code generation, where a single wrong symbol invalidates the molecule.
• Code-driven animation and interactive demos produced from a description.

Industry relevance. The results position a 9B open model above proprietary systems on several generation benchmarks and a 35B-A3B model as competitive with leading proprietary systems, which matters for teams that need visual-coding capability without closed API dependence. The rubric-as-reward formulation is also directly transferable to any RL pipeline where the reward must be judged rather than executed.

Future Directions

• Representation gaps. The coverage table shows PrismaCoder-2M absent on other plotting languages (R, MATLAB, C++, TypeScript) and on Wolfram/Mathematica, where JanusCoder and VisCoder2 have presence — extending coverage is an obvious next step.
• The editing weakness. DesignBench editing is the one web metric where both models trail every Claude 4.6 and Gemini 3 variant; closing this gap is unresolved.
• RL scope and cost. RL is run only on Python, HTML, and SVG, with 20K held-out candidates and 5K retained per domain at K = 5 rollouts; whether the dispersion-based selection transfers to the long-tail formats is not established.
• Judge dependence. The pipeline relies on proprietary models (Claude-Opus-4.6, Gemini-3-Pro, Gemini 3 Flash) for curation, rubric generation, and judging; whether open substitutes preserve the measured gains is an open question the paper does not answer.

Target Audience

Researchers and engineers working on code generation with visual outputs, multimodal LLM post-training, and RL with learned or judged rewards. It is most useful to those building data pipelines and reward models for hard-to-verify outputs, and to practitioners deciding whether an open 9B/35B-A3B model can replace a proprietary system for chart, front-end, or document generation. Readers wanting to reproduce the work will need the appendices, which hold the prompt templates, data sources, and per-format gatekeeper details.

Authors’ abstract

Multimodal code intelligence connects the symbolic world of programs with the visual world their execution creates. Interest in this capability is burgeoning, as models are increasingly asked to produce charts, webpages, animations, and semi-structured documents. Recently, proprietary models have pushed far ahead, producing interactive artifacts even from a text description or a screenshot alone. Yet their data and training remain closed, and open progress is held back by the quantity and quality of available data and by the pipelines and methods for producing it. Quality is the harder of the two, since a sample must be correct as a program, well built, and above all visually right once rendered. No existing signal covers all three. Execution often returns just a pass or fail, code similarity does not map to the visual output, and task-specific verifiers do not generalize. To push open progress further, we start from the data side with PrismaForge, a hierarchical data pipeline centered on a synthesized rubric that decomposes these factors into checkable criteria. The same rubric serves as both filter and reward: it curates the supervised corpus, then drives reinforcement learning on visual-coding tasks. Leveraging PrismaForge for data synthesis and filtering, we build PrismaCoder-2M, the most diverse open dataset for multimodal code intelligence and the largest to date. On the model side, we train PrismaCoder-9B and PrismaCoder-35B on this data and release both. They lead the open state of the art by a wide margin and rival leading proprietary systems. Our analyses further show that the proposed rubric-based pipeline improves multimodal coding performance across model backbones. We release our code and data at https://qiushisun.github.io/PrismaCoder-Home/.