VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation

AI summary

Overview

Research area: Computer vision and multimodal agent evaluation — specifically, benchmarks for AI agents that produce videos by writing and executing code rather than by running a diffusion video generator.

Technical level: Intermediate. The paper is readable by anyone familiar with LLM agents, tool use, and standard video/LLM evaluation practice; no diffusion-model mathematics is required, though the three-layer grading protocol takes some care to follow.

One-sentence scope: VibeVideoBench is a 239-task, English-and-Chinese benchmark that gives frontier multimodal agent systems one hour in a sandbox to produce a complete MP4 with sound entirely through code, and grades the results with program checks, roughly 13k task-specific rubric checks, and three VLM judge panels validated against blind human preference.

What This Paper Is About

Diffusion video models generate visually striking clips but cannot reliably render an exact formula, a correct chart, a rule-consistent game, or narration synchronized to on-screen events. An alternative route is for a multimodal agent to write code that renders every frame and synthesizes every sound, driving renderers, headless browsers, TTS and ffmpeg on its own. The paper asks a question existing evaluations do not answer: how good are general-purpose agents at making complete, audio-visual, multi-minute videos this way — and how should such work be graded when no unit test for a video exists?

Key Contributions

1. The benchmark itself. VibeVideoBench contains 239 self-contained tasks across 12 categories and 43 subcategories (34 presentation forms plus 9 knowledge domains), in English (209 tasks) and Chinese (30 tasks), with target runtimes from 8 s to 15 min, optional read-only input assets, a machine-readable delivery spec, a four-channel audio policy (narration, music, sound effects, ambience), and a reproducible sandbox. One pass over the benchmark asks for 11.1 hours of finished video.

2. A hierarchical, evidence-based evaluation protocol. Program checks (L1, 1,449 checks) verify the delivery spec with no model involved; rubric checks (L2, 12,985 checks, about 54 per task) each quote the requirement they enforce, separate process shown from result claimed, and carry a modality tag; quality scores (L3, 1,195 rubrics) rate five dimensions — visual quality (VQ), visual aesthetics (VA), content logic (CL), timing and motion (TM), and audio quality and A/V sync (AQ) — against per-task anchors. Missing videos are handled explicitly by reporting both delivered-only and no-video-as-zero results.

3. A large-scale empirical study of nine agent systems. Nine frontier systems from six labs, each run in its native harness for 2,151 scored rollouts and 6.5k video judgments, plus a run-by-run reading of all 224 failed deliveries, a trajectory-level analysis of renderers, self-inspection and delivery timing, and a blind pairwise human preference study on all 239 tasks.

4. A positioning argument. Table 1 contrasts VibeVideoBench with prior work (VBench, EvalCrafter, VideoArgus, ArtifactsBench, TheoremExplainAgent, Paper2Video, Code2Video-Compass, AgenticVBench) on who makes the video, what is asked, what is delivered and how it is graded; the authors state that only VibeVideoBench has general-purpose agents build complete audio-visual videos from scratch with code across open domains, graded with task-specific rubrics.

Main Findings

• The benchmark is far from solved, and two systems lead. GPT-6 Astra (L3 overall 4.126 on delivered videos, human Elo 1242) and Claude Opus 5.5 (4.074, Elo 1203) rank first and second with both judges and human annotators. Yet even the best system on the strict metric, Opus, satisfies every core requirement on only 76.3% of tasks; strict success ranges across systems from 26.6% to 76.3%.

• Delivery reliability is a second axis, separate from quality. Counting missing videos as zero changes the ranking markedly. Gemini 4 Argon delivers only 184 of 239 videos (77%) and falls from 4th to 7th; GLM-5.3-Flash, which delivers 179 videos and passes the spec on 66.9% of tasks, falls from 7th to 9th, while DeepSeek-V4.1-Flash rises from 5th to 3rd. Opus is the only system to deliver all 239 videos and passes every program check on 99.6% of tasks.

• Videos are lost to how agents use their time. Of the 224 runs without a playable video, 169 (three quarters) are final renders that started too late, ran too slowly, or never started; 45 (20%) are stalled renders, 7 (3%) are outputs the agent destroyed itself, and 3 have other causes. GPT-5.6 Sol's final full renders in failed runs start a median of 9 minutes before the deadline, often with frame-by-frame browser capture needing 5–35 minutes; 65 runs across systems never start a final render at all, 31 of them GLM-5.3-Flash's. Stalled renders occur only in the two Gemini systems: 33 deadlock because the agent's script pipes frames into ffmpeg without reading its error stream, and 12 end the turn while a render runs in the background. Opus, the only system without a failure, finishes a median of 24 minutes before the deadline, against 10–11 minutes for Sol and GLM.

• Agents build their own renderers; self-inspection is universal but not decisive. Every pipeline is "code → frames → ffmpeg" (ffmpeg appears in 96–100% of runs), while Manim appears in 8 of 2,151 runs and MoviePy in 13. Each system keeps a house stack: Claude agents and Gemini 4 Argon draw vectors with cairo (42–54%), GPT-5.6 Sol, Kimi and GLM rasterize with PIL/numpy (43–55%), and Astra, DeepSeek and Gemini 3.8 Flash capture browser canvases and three.js scenes (52–60%). 95% of runs view their own rendered frames, but how much they look does not predict delivered quality (ρ = −0.07, n = 9): Astra looks least (median 3 images per run) and scores best, DeepSeek looks most (median 35). Final reports often claim verifications that do not hold — among playable videos failing spec compliance, the report still claims verification in 10 of 14 cases for Astra and 3 of 11 for Sol.

• Judges rank systems like humans, but do not certify single videos. The three panels rank systems almost identically (system-level Spearman 0.95–0.98), and the three-panel mean matches human Elo at ρ = 0.98. On individual pairs the mean agrees with humans 73.3% of the time — 63% when the panels' score gap is below 0.2 (about a third of pairs) and 86% when it is 0.5 or more. Check-level agreement reaches only κ = 0.42–0.54 despite 91–95% raw agreement, video-level overall scores correlate at r = 0.44–0.50, and same-task orderings agree only 66–70% of the time. On pairs humans label "both bad," the panels give the same scores (3.81) as elsewhere, so scores order videos but do not certify acceptability.

• Costs vary enormously and only weakly track quality. Mean list-price API cost per task ranges from $0.32 (GLM-5.3-Flash) to $15.38 (Claude Opus 5.5), a 49× spread, with prompt-cache writes making up 76% of Opus's cost and 48% of Sonnet's, and fresh uncached input 55% of GPT-5.6 Sol's. Rank correlation between mean cost and overall score is ρ = 0.55 (permutation p = 0.13). Three systems sit on the cost–score Pareto frontier: GLM-5.3-Flash ($0.32, 2.79), DeepSeek-V4.1-Flash ($0.47, 3.91) and GPT-6 Astra ($8.19, 4.11).

Methodology in Plain English

Building tasks. Real requests are vague and often presuppose material that was never supplied, so they cannot be graded as written. The authors therefore use real sources only as seeds: 484 prompts collected from users of code-based video tools, dense captions of real videos (508,993 caption records scanned), real asset packs such as 3D scans and game sprites, and lecture notes and research papers. LLM drafters with cross-review rewrite each seed under a shared "contract" that pins down every fact, number and object identity the video depends on, splits the task into stages with dwell times, and states a four-channel audio policy, while leaving creative freedom intact. Humans then select checkable tasks over open briefs, verify facts, and strip all authoring traces. Most web prompts were removed because they presupposed missing material (125 of 332 removals) or were content-free style templates (99); selectors kept 88% of music-and-sound drafts but only 17% of story and comedy drafts and 15% of game drafts.

Grading. A video has no unit test, a holistic score cannot be traced to evidence, and many VLMs cannot hear. So grading has three layers with different executors. L1 is decided by code alone: does an MP4 decode, is the resolution and frame rate right (frame rate ±0.01), is the duration within [t_min − 0.5, t_max + 0.5] s, is the source project delivered, is an exact frame count met where required (54 tasks), is audio audible (peak > −50 dBFS), plus narration coverage via VAD, narration language via ASR, and absence of long black or frozen stretches for lessons and papers. L2 is 12,985 atomic pass/fail rubric checks judged by VLMs; each check quotes its basis, requires that a process be shown rather than a result claimed, and carries a modality tag. L3 asks for 1–5 scores on five dimensions against anchors written per task.

Judges. Three independent panels see the task prompt and rubric but never the system identity. J1 (Gemini 4 Argon) and J2 (Gemini 3.8 Flash) watch the full MP4 with sound; J3 routes visual checks to GPT-5.6 Sol and audio checks to Qwen3.8-Omni-Flash, so audio checks never reach a judge that cannot hear. Panel metrics are computed separately and then averaged.

Running agents. Each model runs fully autonomously in its vendor's native harness at the highest effort the provider accepts, with all tool permissions pre-approved, one fixed instruction, and no follow-up turns, in identical Docker containers under a one-hour wall-clock budget. No image or video generation model is available; the only generative service is a TTS command-line tool backed by Qwen3-TTS-Flash. Evaluation covers complete agent systems rather than separating model from harness.

Validation. A blind pairwise human study covered all 239 tasks, with annotators choosing A, B, tie or both-bad between two playable videos; Bradley–Terry strengths were fit on decisive votes with 2,000-sample bootstrap CIs.

Why This Matters

Impact on research. The paper argues that video is a perceptual, temporal and audio-visual outcome for which no unit test exists, and therefore probes something text-only agentic benchmarks cannot: whether an agent can check its own output across picture, sound and time. It also supplies a caution for the field's habit of using VLM judges as reward signals — per-video verdicts and per-video rewards need margins, multi-panel consensus or human checks before they can be trusted, whereas system-level rankings over hundreds of tasks are well supported.

Real-world applications (as framed by the task categories):

• Classroom lessons and paper explainers, where exact formulas, notation and multi-minute structure must be correct rather than merely plausible.
• Data stories, charts and statistical comparisons, where numbers and labels must be exactly right.
• Product walkthroughs, software and device tutorials, and making/assembly sequences that must be rule-consistent and revisable.
• Music-driven pieces, games and rhythm content, where sound and visuals must be synchronized to events.

Industry relevance. The results tie evaluation to deployment conditions: users adopt a model together with its harness, so the benchmark scores native agent systems end to end. The delivery-timing failures — late final renders, frame-by-frame capture, deadlocked ffmpeg pipes — are engineering lessons for anyone shipping an autonomous media agent. The cost spread of $0.32 to $15.38 per task, combined with a cost–score correlation of only ρ = 0.55, gives a concrete efficiency picture for product teams, and the report-verification gaps (agents claiming "ffprobe" checks that did not hold) are directly relevant to anyone relying on agent self-reports.

Future Directions

• Making per-video judgments trustworthy. The panels agree with humans on 73.3% of pairs and only 63% when their score gap is below 0.2, and they assign the same scores (3.81) to pairs humans call "both bad." Margins, multi-panel consensus, or targeted human checks are open problems, especially if these scores are to be used as per-video training rewards.

• Teaching agents to plan their hour. 169 of 224 failures are final renders that started too late, ran too slowly, or never started. Since slower systems recover some of these videos given a longer budget (App. I.3), how to schedule, checkpoint and guard a long render is a concrete capability gap rather than a raw model-quality gap.

• Understanding why self-inspection does not help. 95% of runs look at their own frames, yet looking volume does not predict quality (ρ = −0.07). Turning frame inspection into genuine error correction — and closing the gap between claimed and actual verification — is unresolved.

• Broadening coverage and renderer analysis. The current set is 209 English and 30 Chinese tasks; extending language, category and domain coverage, and disentangling renderer choice from system identity (the paper notes renderer and system are coupled, so renderer–quality links are associations only), are natural next steps.

Target Audience

Researchers and engineers working on multimodal agents, agentic benchmarks and evaluation methodology; teams building automated video, educational-content or media-production tools; and practitioners who use LLM or VLM judges as evaluators or reward models and need evidence on where those judges are and are not reliable. Readers interested primarily in diffusion video generation will find the paper's framing deliberately orthogonal to that line of work.

Authors’ abstract

Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a rule-consistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox with no image or video generation model, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, scored by three VLM judge panels. Across nine frontier agent systems and 2,151 scored rollouts, GPT-6 Astra and Claude Opus 5.5 rank first and second with both the judges and human annotators, and the strongest system satisfies every core requirement on 76.3% of tasks. Delivery is a second axis: counting missing videos as zero changes the ranking, and three quarters of the lost videos are final renders that started too late, ran too slowly or never started, a failure of how agents plan their hour. Agents build their own rendering pipelines and inspect their own frames, yet how much they look does not predict quality. The judges' ranking of the systems closely matches blind human preferences.