Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

AI summary

Overview

Research area: Computer vision and multimodal large language model (MLLM) evaluation, specifically benchmarks for code-based video generation where a model writes executable programs that render a video rather than synthesizing pixels directly.

Technical level: Advanced. The paper assumes familiarity with MLLM benchmarking, rendering stacks (HTML/CSS, SVG, Canvas, Three.js, Remotion), and automated video-quality metrics.

Scope: Code2Video-Compass is a 2,104-task benchmark with a 12-metric objective and five-metric subjective evaluation suite for code-based text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) generation, used to evaluate 15 proprietary and open-weight MLLMs.

What This Paper Is About

Code-based video generation asks a model to write a complete, runnable program that renders a video, which gives fine-grained control, targeted editing, and component reuse compared with direct video generation. Existing code-based benchmarks cover only narrow niches, such as Manim educational videos, SVG animations, data narratives, or web animation reconstruction, so nobody had measured whether models can build whole videos across many creative genres and input types. The authors build that missing benchmark, spanning text, image, and video inputs, and evaluate how well current MLLMs handle it.

Key Contributions

1. A unified general-purpose formulation. The paper defines T2V, I2V, and V2V tasks in a single setting where an MLLM must emit executable code producing a complete video, with reference images or clips reconstructed through code rather than embedded or replayed.
2. A taxonomy-driven dataset of 2,104 tasks. Tasks span 10 categories and 24 subtypes, each annotated along three complexity dimensions (construction, dynamic animation, and state dependency) with three levels each. The set combines 1,719 paired tasks derived from 573 complete source videos with 385 independent tasks.
3. A dual-level evaluation framework. Twelve objective metrics (execution success, token consumption, technical quality, aesthetic quality, motion smoothness, local and global text–video alignment, and reference-fidelity metrics for I2V and V2V) plus five subjective metrics (instruction following and four holistic quality ratings) are combined into a single protocol.
4. Extensive benchmarking of 15 MLLMs from nine families. The study reports per-setting rankings, a failure-mode analysis over 2,756 error cases, a complexity analysis across all three dimensions, a cost-versus-quality analysis, and a human-validation study of the MLLM judge.

Main Findings

• Claude Opus 5.5 leads all three settings. It achieves the best average rank with 1.60 on T2V, 1.08 on I2V, and 1.15 on V2V, followed by Claude Sonnet 5.5 in each setting; proprietary models hold the top three positions everywhere.
• Open-weight leaders differ by setting. DeepSeek-V4.1-Flash is the strongest open-weight model on T2V, while MiMo-V2.6-Pro leads I2V and V2V.
• Execution reliability varies widely. Execution success ranges from 0.52 to 0.94 on T2V, 0.59 to 0.96 on I2V, and 0.69 to 0.97 on V2V across the 15 models.
• Leaders have different strengths. On T2V, Claude Opus 5.5 leads instruction following (0.91) and content coherence (4.05) plus aesthetic quality and both alignment metrics; Claude Sonnet 5.5 leads motion quality (3.85) and viewer engagement (3.90); GPT-6.1 Sol leads visual design (3.88) and technical quality.
• Reference fidelity does not imply instruction following. On I2V, GPT-6 Astra and GPT-6 Luna have similar perceptual fidelity (0.83 vs 0.82) and structural fidelity (0.65 vs 0.62), but diverge sharply on instruction following (0.86 vs 0.70) and motion quality (3.47 vs 2.61). On V2V, Claude Sonnet 5.5 nearly matches Claude Opus 5.5 on sequence alignment (0.87 vs 0.88) and appearance fidelity (0.77 vs 0.79), yet scores 0.76 instruction following and 3.36 content coherence versus 0.93 and 4.00.
• Quality and reliability trade off. GPT-6 Astra and GPT-6.1 Sol score higher mean holistic quality among successfully rendered videos (4.19 and 4.15) than Claude Opus 5.5 (4.04), but their higher failure rates pull their full-set means down to 3.62 and 3.61, while Claude Opus 5.5 reaches the highest full-set mean of 3.87.
• Failure modes differ by model family. Compilation errors plus missing APIs or packages account for 83.4% of GPT-6 Astra's failures and missing APIs or packages alone for 59.7% of Qwen3.8-Max's, whereas runtime exceptions account for 75.0% of Claude Opus 5.5's failures and over half of those for GLM-5.3-Flash, DeepSeek-V4.1-Flash, and MiMo-V2.6-Pro.
• Performance drops with complexity, but not uniformly. Structural fidelity falls by 14.38% and 12.82% of its score range from level 1 to level 2 under construction and dynamic animation complexity; for these two dimensions most metrics decline more steeply from L1 to L2 than L2 to L3, whereas state-dependency complexity shows larger declines from L2 to L3 across all 15 metrics.
• More tokens and cost do not buy quality. GLM-5.3-Flash uses 90.0k output tokens per request versus 13.9k for GPT-6 Astra yet scores 2.94 versus 3.62; GPT-6.1 Sol and GPT-6 Astra score nearly identical quality (3.61 and 3.62) while GPT-6 Astra incurs roughly 6.1 times the estimated API cost.
• The MLLM judge aligns with humans. On 300 outputs, Gemini 3.8 Flash as judge reaches Pearson's r of 0.796 for instruction following and 0.783 for holistic quality against three human raters, with ICC(A,3) repeat stability of 0.983 and 0.970.

Methodology in Plain English

The authors built tasks two ways. First, an independent set: they drew text prompts from VidProM and Shot2Story, narrowed 54,443 raw candidates down to 145 seed ideas with embedding-based filtering and human review, and separately curated reference images for I2V and YouTube clips for V2V (120 references chosen from 360 candidates per setting, with 2–10 s key segments cropped for V2V). Second, a paired set grounded in real work: they collected 197 YouTube animation records and 2,473 candidate URLs from Bilibili, REDnote, and X, screened them down to 979, removed exact duplicates by normalized URL and SHA-256 hashes and near-duplicates via SSCD frame embeddings indexed with FAISS, scored the remaining 915, and kept 573 complete videos after human review. Each kept video yields a matched T2V, I2V, and V2V task sharing the same creative goal and event progression.

Instructions are written to be self-contained, specifying visual elements, spatial relations, exact text and data, motion, event order, and cross-stage dependencies. Annotators label subjects, temporal events, motion types, and the three complexity dimensions, and write QA checklists of up to 16 yes/no questions per task.

For evaluation, models run once per task in a single API call with no execution feedback or repair. Generated code is rendered through Remotion on a shared backend. Objective metrics then measure whether a video rendered at all, how many output tokens it took, perceptual and aesthetic quality, frame-to-frame smoothness, two scales of text–video alignment, and how faithfully reference images or clips were rebuilt inside the output. Subjective metrics come from a Gemini 3.8 Flash judge that answers the QA checklist and rates four five-point Likert dimensions of holistic quality. Because running everything on 15 models is costly, the team sampled a fixed 1,054-task subset (360 T2V, 347 I2V, 347 V2V) and verified statistically that it does not differ from the full set in complexity distribution or prompt length.

Why This Matters

Impact on research. The paper moves code-based generation evaluation beyond single-purpose niches such as Manim education or SVG animation toward complete videos with narrative progression, and shows that execution reliability, semantic alignment, and reference fidelity are separable axes that can move in opposite directions. The complexity analysis gives a diagnostic ladder (construction, dynamic animation, state dependency) that other benchmarks can reuse.

Real-world applications:

• Automated production of explainer motion graphics, kinetic typography, and title sequences for marketing and media teams.
• Data and infographic animation, where exact numbers, chart states, and cross-stage consistency must survive across scenes.
• UI and product motion mockups generated from a written spec or a single reference screenshot.
• Reference-driven editing and reuse, where an existing clip or still must be rebuilt inside a new, longer composition.

Industry relevance. The token-cost and API-cost analysis gives practitioners a concrete basis for choosing models by budget rather than raw score, and the failure taxonomy tells tooling teams where to invest: sandboxed compilation and dependency resolution for some model families, runtime exception handling for others.

Future Directions

• Closing the reliability gap. Execution success still bottoms out at 0.52 on T2V, and the paper's own failure analysis points to compilation, missing packages, and runtime exceptions as the dominant causes; execution-feedback loops or repair agents are the obvious next lever, since the current protocol allows none.
• Improving long-range state dependency. Declines from L2 to L3 are largest for state-dependency complexity across all 15 metrics, raising the question of how to keep objects, data, and interfaces consistent across multiple intervening changes.
• Better reference integration. Mismatches such as GPT-6 Astra versus GPT-6 Luna on I2V, and Claude Opus 5.5 versus Claude Sonnet 5.5 on V2V, show that reconstructing a reference and following a full-video instruction are distinct skills that need separate attention.
• Efficiency-aware evaluation and judge coverage. The Pareto analysis shows wide cost spreads at equal quality, motivating efficiency-weighted scoring, and the judge validation covers six models on 300 outputs, leaving open how the protocol scales to longer videos and more model families.

Target Audience

Researchers and engineers working on MLLM code generation, programmatic visual creation, and video generation evaluation; benchmark designers who need a template for multi-modal task construction and dual-level scoring; and product teams evaluating which models can reliably generate motion graphics, animated infographics, or reference-driven video edits under real cost constraints. Readers new to the area will find the category taxonomy and metric definitions accessible, but the paper's results assume comfort with benchmark statistics and rendering pipelines.

Authors’ abstract

Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.