Video-Coder: Reinforcing Agentic Coding for Video Generation

AI summary

Overview

Research area: Code-driven video generation, multimodal coding agents, and agentic reinforcement learning (computer vision / generative AI).

Technical level: Advanced. The paper assumes familiarity with diffusion-based video generators, Blender scripting, supervised fine-tuning, and GRPO-style reinforcement learning.

Scope: Video-Coder trains a 9B multimodal language model to write, execute, and iteratively debug Blender Python programs that build a dynamic 3D scene, whose rendered "draft video" then guides a downstream video generator toward better physical and spatial consistency.

What This Paper Is About

Current video generators match the pixel-level statistics of video but do not explicitly model the 3D world, so they often miscount objects, break occlusion, produce implausible trajectories, or violate physical laws when prompts involve precise spatial relations or multi-object interactions. End-to-end generated videos are also hard to edit or reproduce, forcing users into prompt trial-and-error. Video-Coder reframes part of the problem as an agentic programming task: a model writes executable code that constructs and animates a 3D scene, renders it into a structure-and-motion-only draft video, and a fixed downstream generator turns that draft plus the prompt into the final realistic video.

Key Contributions

1. A code-driven video generation agent. The authors describe Video-Coder as the first attempt to train a multimodal coding agent for video generation, where the agent autonomously writes, executes, and revises code to build a dynamic 3D scene that supplies structure and motion signals to a downstream generator.
2. A four-stage data construction pipeline and two datasets. Prompts are collected from public video-generation datasets and rewritten by an LLM into scene-centric descriptions; trajectories are generated with Gemini 3.8 Flash inside the harness; ground-truth videos are synthesized with MiniMax-H3 from prompt plus draft video; samples are filtered on execution validity, draft-video quality, and prompt–video consistency (judged by Seed2.0). This yields Video-Coder-SFT-6k (6k prompt + full trajectory + reference video) and Video-Coder-RL-2k (2k prompt–reference pairs).
3. The Video-Coder Harness. A unified interactive environment combining sandboxed code execution, 3D scene rendering (untextured gray clay shading, fixed resolution and frame rate), and rich execution feedback, used both for trajectory synthesis and for RL training/inference.
4. A two-stage training recipe with dual rewards and reward annealing. SFT on trajectories followed by harness-native agentic RL with GRPO, combining an execution-based reward (0/1 for tool-call correctness and successful execution passing all environment checks) and a VLM-judged video-based reward normalized to [0, 1], with the execution term decayed over training by $e^{-\text{steps}/T}$ and $\alpha = 0.5$.

Main Findings

• PhyGenBench gains: Video-Coder-9B + Minimax-H3 reaches the best average of 0.67, versus 0.61 for Minimax-H3 alone, and leads all four categories: Mechanics 0.68, Optics 0.79, Thermal 0.60, Material 0.61. For reference, Wan2.2-TI2V-5B scores 0.54, HunyuanVideo 0.33, Gen-3 0.51, and Kling 0.49.
• VBench-2.0 gains: The same combination reaches 78.19% average plausibility versus 54.58% for Minimax-H3 and 77.04% for CogVideoX-1.5, with Mechanics at 88.67% (vs. 67.59% for Minimax-H3 and 80.80% for CogVideoX-1.5) and Thermotics at 70.88%.
• One dimension still lags: On VBench-2.0 Material, Video-Coder reaches 75.02%, improving on MiniMax-H3 (48.94%) but remaining below CogVideoX-1.5 (83.19%); the authors attribute this to material phenomena depending on fine-grained appearance, which the draft video deliberately omits.
• Transfer to another generator: Although RL rollouts used Minimax-H3, the agent also improves OmniWeaving from 0.48 to 0.51. (The text calls this "an increase of 16.28 points," which is inconsistent with the 0.48 → 0.51 values in Table 1.)
• SFT helps, RL matters more: Ablation on PhyGenBench: MiniMax-H3 0.61 → + SFT 0.63 → + full RL 0.67, a 6-point total improvement over the vanilla generator.
• Both rewards are necessary: Removing the video reward drops the score to 0.64; removing the execution reward drops it to 0.63.
• Annealing is important: Fixed-weight reward combination scores only 0.65; the authors suggest constant execution reward encourages executable but conservative programs that mask fine-grained video-quality differences.
• Insensitivity to α: Performance dips slightly at α = 0.4 and α = 0.7, while α = 0.5 and α = 0.6 give similar favorable results.

Methodology in Plain English

The pipeline starts with captions from existing video-generation datasets. Because those captions emphasize texture, lighting, and style, an LLM rewrites them into scene-centric descriptions that spell out which objects exist, how many, where they sit, how they move, and what the camera does—and discards prompts that depend on fine appearance or facial expression.

A teacher agent (Gemini 3.8 Flash) is then plugged into the harness and asked to produce code for each prompt. The harness gives it two tools: a fast static check (check_script) and a full execution-and-render call (run_blender). After each call the agent sees error messages, automatic content warnings (empty frames, static video, subject out of frame), and a few sampled frames as images. It loops write–run–revise until the draft matches the prompt or the turn limit is hit. Submissions are re-executed from scratch in a clean directory and accepted only if they pass again, ensuring reproducibility from code alone.

Each accepted draft video is paired with the prompt and passed to MiniMax-H3, which produces a realistic reference video whose structure and motion follow the draft and whose appearance follows the prompt. Three filtering passes clean the data, after which the split is fixed.

Training uses Qwen3.5-9B. Stage one is supervised fine-tuning on the recorded multi-turn trajectories, teaching scene planning, tool invocation, feedback interpretation, and revision. Stage two is GRPO-based agentic RL on prompt/reference pairs: the model rolls out several trajectories per prompt, receives a binary execution reward plus a VLM-judged video reward, and the execution term is exponentially decayed so late training is driven mainly by video quality. The video generator itself is frozen throughout—only the coding agent is optimized.

Why This Matters

Research impact. The paper offers an alternative to "scale the video model" thinking: it shows that a relatively small 9B coding agent can raise the physical-consistency scores of several different frozen video generators. It also contributes infrastructure—an executable, verifiable harness with dense structured feedback—that is reusable for other agentic code-generation research, and it releases data, models, and code.

Real-world applications:

• Controllable content creation: scripted object counts, layouts, and camera moves for advertising or previsualization, where prompt-only control is unreliable.
• Reproducible/editable generation: because a program, not a prompt, defines the scene, a shot can be tweaked by editing a few lines and re-rendering.
• Physics-aware simulation-style clips: falling, collision, rolling, and bouncing sequences where naive generators violate mechanical laws.
• Previsualization and storyboarding: a cheap gray-clay draft communicates structure and motion before money is spent on final rendering.

Industry relevance. The approach is generator-agnostic and retrofits onto existing commercial or open video models (Gen-3, Kling, MiniMax-H3, OmniWeaving were all evaluated), making it a plausible add-on layer rather than a replacement for a video-generation stack.

Future Directions

• Closing the appearance gap. The Material dimension remains behind CogVideoX-1.5; enriching the draft beyond structure and motion (e.g., coarse material or texture cues) is an open question.
• Scaling data and reward quality. The video reward depends on a VLM judge (Qwen3.8-omni-flash); better or more fine-grained judges, and larger filtered datasets, are natural extensions.
• Broader scene-programmability. The pipeline deliberately discards prompts about fine-grained appearance, facial expression, and anything not constructible from geometric primitives and keyframed motion—expanding that envelope is unresolved.
• Generalization beyond Blender and beyond physics benchmarks. Only PhyGenBench and VBench-2.0 physical dimensions are measured; whether the agent transfers to other 3D toolchains or to longer, more complex scenes is not reported.

Target Audience

Researchers and engineers working on video generation, controllable generation, and multimodal agents; practitioners who want structured control over diffusion-based video models; and agentic-RL researchers interested in long-horizon, tool-using environments with verifiable execution feedback. Readers without background in diffusion models, 3D scene scripting, or policy-gradient RL will find the method sections demanding.

Note on internal inconsistencies in the source: the datasets are named Video-Coder-SFT-6k / Video-Coder-RL-2k in the abstract and method, but the training-details and conclusion sections refer to Video-Coder-SFT-10k / Video-Coder-RL-4k, and the OmniWeaving "16.28 points" figure does not match Table 1. These are reported as stated rather than reconciled.

Authors’ abstract

Recent years have witnessed remarkable progress in video generation in terms of visual quality and temporal consistency. However, generating videos with precise spatial structure, coherent motion dynamics, and controllable scene composition remains challenging. In this paper, we introduce Video-Coder, the first attempt to train a code-driven video generation agent, which autonomously writes and executes code to construct a dynamic 3D scene (Draft Video) that provides…