CodeCap-Bench: Can Code Serve as a Good Video Caption?

AI summary

Overview

Research area: Computer Vision — fine-grained video captioning, video representation, and multimodal large language model (MLLM) evaluation.

Technical level: Advanced.

Scope: This paper proposes and benchmarks "Code as Video Caption," a paradigm in which executable HTML/CSS/JavaScript programs, rather than natural language, serve as compact, renderable descriptions of structured 2D videos, and it introduces CodeCap-Bench (323 videos, five categories) to measure how faithfully seven MLLMs can produce such code.

What This Paper Is About

Natural language video captions struggle to express fine-grained spatial layouts, motion trajectories, and coordinated state changes without becoming long and ambiguous. The authors ask whether executable code — built from graphical primitives, layout rules, and functions of time — can act as a complementary caption that is precise, unambiguous, and directly checkable by rendering it back into a video.

The paper has two goals: first, to test whether code captions actually preserve more video information than natural language captions (via question answering and video reconstruction); second, to systematically measure whether today's MLLMs can generate faithful code captions at all, which is what the CodeCap-Bench benchmark is built for.

Key Contributions

1. A new captioning paradigm. The paper formalizes "Code as Video Caption": given a reference video of duration T, a model must emit code whose execution through a rendering engine reproduces the video's visual elements, spatial structure, and temporal dynamics — in a pure-code setting where no reference frames, screenshots, external images, textures, or fonts may be reused.

2. CodeCap-Bench, a 323-video benchmark. Videos span five categories of structured 2D content (desktop and application interfaces; 2D games; visual design and media animation; data and scientific visualization; icons and abstract motion graphics), collected from Bilibili, YouTube, Behance, and Vimeo, filtered for 2D layouts, and annotated with focal points and object-level spatio-temporal boxes under human review.

3. A three-level evaluation framework. Fidelity is scored by (a) global visual similarity using V-JEPA 2.1 ViT-Gigantic-384 features, (b) object-level spatio-temporal IoU using GPT-6-sol-guided segmentation with SAM 3.1, and (c) fine-grained visual information retention on a 0–5 scale, split into static and dynamic sub-scores.

4. Evidence that code captions beat natural language captions, plus a systematic study of seven MLLMs across agentic and non-agentic settings, thinking-effort levels, cost, tool-use behavior, and failure modes.

Main Findings

• Code captions support better question answering. On 100 videos with QA annotations drawn from DVBench, VideoMathQA, and GUI-World, a Gemini-3.8-flash answerer scored higher on code-rendered videos than on natural language captions for all three caption generators: GPT-5.6-sol 76.42 vs 73.58, Opus-5 83.02 vs 72.64, and Qwen3.8-max 81.13 vs 78.30. Answering from the original videos gives 83.96, so Opus-5's code-rendered videos nearly match the originals. Gains range from 2.83 to 10.38 percentage points.

• Code captions reconstruct videos more faithfully. With the same three caption models, code rendering beats caption-to-video generation (via Seedance-2.5) on all three reconstruction metrics: Appearance Similarity improves by 0.20–0.23 and Keyframe Structural Similarity by 0.18–0.22, with smaller gains in Motion Pattern Similarity.

• GPT-6-astra leads the benchmark. With the Codex harness at medium thinking effort it reaches 99.4% video production success, 0.9735 visual similarity, 75.37 object-level spatio-temporal IoU, and 73.24 overall fine-grained retention — 14.40 points above the next-best configuration, Opus-5 with Pi Agent (58.84).

• High rendering success does not mean faithful captions. GPT-6-sol with Codex produces a video in 100.0% of tasks but scores only 51.34 overall retention, versus 73.24 for GPT-6-astra at 99.4% success.

• Dynamic content is consistently harder than static content. Every evaluated configuration preserves motion and temporal relationships less faithfully than appearance and layout: GPT-6-astra with Codex scores 80.72 static vs 68.61 dynamic, and the gap is larger for GPT-6-sol with Codex (64.04 static vs 43.92 dynamic).

• Agentic workflows help reliability more than fidelity. Qwen3.8-max with Pi Agent rises from 82.0% to 93.2% success and from 44.67 to 48.83 overall retention versus non-agentic generation. But GPT-6-sol and GPT-5.6-sol score lower with both tested agent harnesses than in the non-agentic setting.

• Open-source models lag substantially. Qwen3.8-max is the strongest non-proprietary entry in the main table, at 48.83 overall retention with Pi Agent and 44.67 non-agentic; DeepSeek-v4.1-flash reaches 44.29 with Pi Agent.

• Thinking effort pays off unevenly. For GPT-6-sol, moving from medium to xhigh lifts spatio-temporal IoU from 50.42 to 67.13 and retention from 47.53 to 63.29, while average cost rises from USD 0.315 to USD 1.409. Qwen3.8-max gains most between low and medium effort (IoU 36.05 to 48.90; retention 42.02 to 48.83), and moving to xhigh adds only 2.67 and 2.72 points while cost rises from USD 0.922 to USD 1.129.

• Cost, tokens, and time diverge. GPT-6-astra scores 72.87 overall retention at a substantially lower estimated cost than Opus-5's 58.84, yet Qwen3.8-max and DeepSeek-v4.1-flash take 42.1 and 50.1 minutes per video respectively, against 12.5 minutes for GPT-6-astra.

• More tool calls are not better. GPT-6-astra reaches 72.87 retention with 47.4 tool calls per video, while Gemini-3.8-flash scores 47.73 with 99.6 calls. Tool-call mix also differs: GPT-6-sol and GPT-5.6-sol spend roughly half their calls viewing reference frames, Gemini-3.8-flash allocates 46.8% to extracting and analyzing frames, and Qwen3.8-max has the highest share of rendering and comparison calls at 30.9%.

• Category difficulty varies. All four models plotted across categories achieve their highest object-level spatio-temporal IoU on "Data and Scientific Visualization," and overall retention is higher on "Desktop and Application Interfaces" and "Data and Scientific Visualization" than on "Visual Design and Media Animation" or "Icons and Abstract Motion Graphics."

• The automatic metric tracks humans well. Across 50 reference videos (10 per category), GPT-6-sol judge scores correlate with human focal-point scores at a Pearson r of 0.942. A second judge (Gemini-3.1-pro-preview) gives lower absolute scores than GPT-6-sol but preserves the same overall ranking between GPT-6-sol and DeepSeek-v4.1-flash outputs.

Methodology in Plain English

The authors start from an intuition: a sentence like "the seat map scrolls downward" is vague, whereas code can specify the grid spacing, the scroll interval, the displacement, the easing curve, and the condition under which a seat changes color. Executing that code produces a video you can put side by side with the original.

To test the idea, they collect videos that are mostly flat, structured 2D content — interfaces, games, animations, charts, icon motion — and discard anything dominated by 3D viewpoints, heavy textures, or dense pixel-level motion. Annotators mark "focal points" (specific static features or dynamic processes, with timestamps) and track key objects frame by frame. All annotations are human-reviewed.

Evaluation is deliberately indirect: instead of grading the code text, they render it at 24 fps and compare the resulting video to the reference on three levels — overall look and motion via learned video features, per-object spatial overlap over time via segmentation, and a 0–5 rubric for whether specific annotated details survived, judged by GPT-6-sol. Models are run in two modes: a single-pass "non-agentic" mode (frames sampled at 2 fps, no feedback) and an "agentic" mode using Pi Agent, Codex, or Claude Code, where the model can inspect frames, write a single self-contained index.html exposing a seek(seconds) interface, screenshot its own render, and iterate. Tasks that fail to render score zero everywhere. Finally, to validate the whole premise, they run head-to-head comparisons of code versus natural language captions on question answering and on reconstruction fidelity.

Why This Matters

Impact on research. The paper reframes captioning as a representation problem rather than a fluency problem, and supplies the first benchmark where caption quality is measured by re-executing the caption. It also gives the video-understanding community a stress test that isolates temporal reasoning: nearly every model, proprietary or not, loses more points on dynamic content than on static content, which is a cleaner signal than aggregate captioning scores.

Real-world applications:

• UI and app reproduction — turning screen recordings into working HTML/CSS prototypes, with built-in spatio-temporal fidelity checks.
• Animation and motion-graphics authoring — converting a reference clip into editable, parameterized code instead of a fixed video.
• Educational and scientific content — regenerating chart and visualization walkthroughs as code, where values and timing are explicit and inspectable rather than described in prose.
• Accessible and searchable media — a programmatic caption is machine-checkable and editable in ways free text is not.

Industry relevance. The cost, runtime, and failure analysis is directly actionable for teams building agentic video-to-code pipelines: proprietary GPT models stay under 1% failure and finish in 12.5 minutes per video on average, while cheaper open-source models are far slower (42.1–50.1 minutes) and account for 73.1% of all failed tasks, with distinct failure signatures (HTML delivery for Qwen3.8-max, generation timeouts for DeepSeek-v4.1-flash). Generation timeouts, incomplete HTML delivery, and rendering timeouts together make up 83.6% of failures — infrastructure problems as much as modeling ones.

Future Directions

• Closing the dynamic-fidelity gap. Static retention is well ahead of dynamic retention in every configuration tested; the paper does not offer a method for fixing this, only evidence of its pervasiveness.
• Making agentic workflows reliably help. Agent harnesses raised success rates but sometimes lowered fidelity (GPT-6-sol and GPT-5.6-sol both scored below their non-agentic runs); what makes rendering feedback beneficial in some cases and harmful in others remains open.
• Extending beyond structured 2D content. The benchmark deliberately excludes 3D viewpoints, detailed textures, and dense pixel-level dynamics, so whether code captions generalize to natural, real-world footage is untested.
• Better cost-efficiency and evaluation scaling. Returns on thinking effort are strongly model-dependent, and the human-correlation study covers 50 videos scored by one judge model; broader calibration and cheaper reliable judging are natural next steps.

Target Audience

Researchers and engineers working on video captioning, video-language models, and multimodal agent evaluation; teams building video-to-code or screenshot-to-code systems; and practitioners who need a rigorous, execution-grounded way to measure whether a model's understanding of a video is actually faithful rather than merely fluent. Readers should be comfortable with benchmark design, MLLM agent harnesses, and standard vision metrics such as IoU and SSIM.

Authors’ abstract

Video captioning aims to preserve visual information through compact descriptions, predominantly using natural language. However, precisely expressing fine-grained spatial structures and temporal dynamics in natural language often leads to lengthy and potentially ambiguous descriptions. In this work, we explore Code as Video Caption, a paradigm where executable programs serve as a complementary representation of visual content through graphical primitives, layout rules, and functions of time. To examine whether code captions effectively preserve video information, we compare them with natural language captions on downstream question answering and video reconstruction tasks. Our results show that code captions achieve superior reasoning accuracy and reconstruction fidelity. Despite these advantages, realizing the potential of code captioning depends on whether current multimodal large language models (MLLMs) can generate faithful code captions. To systematically assess this capability, we introduce CodeCap-Bench, which comprises 323 videos spanning five categories of structured 2D content, along with fine-grained annotations and a multi-level evaluation framework. Extensive experiments reveal that while leading proprietary MLLMs demonstrate promising code captioning capabilities, they still struggle to faithfully capture complex temporal dynamics, with open-source models lagging substantially behind.