CineRecon: Benchmarking Frontier Models for Agentic Cinematic Reconstruction

AI summary

Overview

Research area: Computer Vision, specifically agentic 3D scene reconstruction, benchmark design, and automated evaluation of multimodal coding agents (arXiv:2610.16577v1 [cs.CV]).

Technical level: Advanced. The paper assumes familiarity with 3D scene construction in Blender, vision-language model judging, agent harnesses, and consistency-based evaluation protocols.

Scope: The paper introduces a benchmark of 50 reference videos and 360 annotated evaluation clips that measures how well autonomous agents reconstruct complete cinematic videos as editable dynamic 3D Blender scenes under three levels of representation detail.

What This Paper Is About

Reconstructing a reference video as an editable dynamic 3D scene gives creators direct control over layout, subject actions, and camera behavior, and the rendered result can then serve as a visual proxy to guide video generation. The open question the authors pose is whether agents can autonomously build such scenes from a complete video and faithfully reenact its content and events, without supplied scene assets or camera parameters. CineRecon is built to measure that joint capability, covering scene content, subject actions and interactions, camera motion, and shot transitions at once.

Key Contributions

1. A new benchmark. CineRecon comprises 50 reference videos and 360 annotated evaluation clips for evaluating agents that reconstruct complete dynamic videos as editable Blender scenes. The reference videos span film, real-world footage, and games.

2. Three representation settings. The authors define Proxy, Simplified, and Detailed settings that increase geometric and appearance requirements while holding requirements for spatial relations, camera behavior, motion, interactions, and event timing constant, and they evaluate camera, static, and dynamic consistency using clips and checklists that underwent multiple rounds of human review and refinement.

3. Systematic model evaluation. They analyze 12 multimodal coding models (13 configurations) using their corresponding agent harnesses and identify shared deficits in spatial relations, specific action events, and camera changes.

4. Cross-setting and ablation analysis. Cross-setting comparisons show that simplifying appearance requirements improves performance for most models, while substantial errors in action timing, contact relations, and coordinated subject motion persist under Proxy. An Opus 5.5 ablation isolates the effect of iterative refinement and reasoning effort.

Main Findings

• Failures appear in all three dimensions. Evaluating 12 models with their native agent harnesses, the authors find failures across camera, static, and dynamic consistency. Subject motions and interactions remain difficult to reproduce even under the Proxy setting.

• Specific actions are the hardest category. The highest model scores on action events are only 42.7 in Proxy, 46.6 in Simplified, and 48.6 in Detailed, indicating that even stronger models incompletely reproduce specific actions and interactions.

• Consistent category-level weaknesses. Across all three settings, each of the 13 configurations scores higher on subject motion than on action events, higher on object presence than on spatial relations, and higher on camera targets than on rotation.

• Wide spread across configurations. In the Detailed setting, the median score across configurations on camera rotation is 25.2 and the highest is 74.9; the corresponding subject-motion scores are 27.3 and 70.3.

• Model-to-model gaps are uneven across requirements. Comparing Opus 5.5 with GPT-6 Astra: object-presence scores are 89.9 versus 79.6 (a gap of 10.3 points), subject-motion scores are 70.3 versus 44.5 (a gap of 25.8), camera-rotation scores are 74.9 versus 42.1 (a gap of 32.9), and action-event scores are 48.6 versus 27.2. Opus 5.5's larger advantages occur in motion, actions, and viewpoint changes.

• Simpler appearance helps most models, but not all. Proxy scores exceed Detailed scores for 12 of 13 configurations on camera consistency and 10 of 13 on dynamic consistency. Averaged equally across the 13 configurations, camera and dynamic scores increase by 7.4 and 4.1 points respectively. GPT-6 Astra scores 78.5, 79.1, and 84.1 on camera consistency in Detailed, Simplified, and Proxy, and 38.1, 40.6, and 44.6 on dynamic consistency. Opus 5.5 changes little across settings, with Proxy-minus-Detailed differences of −0.8 for camera and −0.5 for dynamics.

• Higher scores still leave dynamics unsatisfied. GPT-6 Astra scores 44.6 on dynamic consistency under Proxy, and Opus 5.5 scores 60.8.

• Iteration and reasoning effort matter. Opus 5.5 Full (high effort plus iterative refinement) achieves 76.6 overall, compared with 66.4 for No Refine and 49.5 for Low Effort, gaps of 10.2 and 27.1 points. Both variants score lower across all three dimensions, with the largest reductions in dynamic consistency: 11.6 and 34.6 points.

• Automatic evaluation tracks human judgment. The study comprises 1,440 comparisons across 12 models, with 480 each for camera, static, and dynamic consistency. Overall win rates yield a Spearman correlation of 0.951 and a Pearson correlation of 0.975 across the 12 models. Human and automatic evaluation both rank Opus 5.5 and GPT-6 Astra first and second, with three models shared between their top four.

• Dataset scale. The 50 reference videos total 763.59 seconds, with a mean duration of 15.27 seconds and a range of 4.90–24.04 seconds. Annotations record camera motion in 94% of videos, subject interaction in 58%, and shot transitions in 18%. The 360 evaluation clips span 3–13 clips per video, average 1.41 seconds, and last at most two seconds (range 0.42–2.00 seconds).

• Annotation volume. Proxy, Simplified, and Detailed contain 9,647, 20,021, and 20,000 checks respectively, totaling 49,668 setting-specific checks; merging shared records yields 35,836 records in the released checklists. Camera and dynamic checks cover the same categories, subjects, and temporal intervals across settings, with 2,376 camera checks and 703 dynamic checks per setting.

Methodology in Plain English

An agent receives one complete reference video and a generic, setting-specific prompt. It must analyze the video, build an editable scene in Blender 5.2 through Blender MCP plus file, shell, and media-inspection tools, animate it, render it, inspect the rendered result, and revise. The delivered artifacts are scene.blend, reconstruction.mp4, a synchronized side-by-side comparison video, a comparison contact sheet, and a short implementation and verification report. The output must match the reference resolution, frame rate, frame count, and duration. External assets, asset-generation services, and using reference imagery as backgrounds or projected surfaces in place of 3D modeling are prohibited.

Three representation settings control how much geometry and appearance is required. Proxy allows a small number of geometric primitives or simple combinations, preserving subject counts, coarse proportions, and the movable parts needed for the observed actions. Simplified requires recognizable categories, overall silhouettes, and major structural components, with fine details and textures allowed to be simplified. Detailed requires the visible structure and appearance of the specific reference instances, including distinctive local shapes and fine appearance features. Spatial relations, camera behavior, motion, interactions, and event timing are held constant across all three.

Evaluation is checklist-based. Each of the 360 clips spans a contiguous interval of at most two seconds and captures a salient scene state, camera change, subject motion, or interaction. Each checklist statement ties an observable property or event to an identifiable entity and a specified timestamp or interval, across 11 categories. All clips and checklist statements were initially annotated by Codex powered by GPT-6 Astra, followed by multiple rounds of human review and refinement.

The 11 categories fall into three dimensions. Camera Consistency covers translation, rotation, zoom, position, target, and shot transitions. Static Consistency covers object presence, object appearance, and spatial relations. Dynamic Consistency covers subject motion and action events (including interactions, deformation, and state changes). Scores are averaged within a clip, then averaged across clips with equal weight, and the overall score is the arithmetic mean of the three dimension scores; equal weighting prevents dimensions with more annotations from dominating.

A vision-language judge, GPT-5.6 Sol at high reasoning effort, compares timestamp-aligned reference and rendered frames and rates each item unsatisfied, partially satisfied, or fully satisfied, mapped to 0, 0.5, and 1. Results are reported on a 0–100 scale. Evaluation clips and checklists are withheld during reconstruction. Each model runs through its own agent harness: Codex CLI 0.153.4 for the GPT models, Claude Code for the Opus models, Antigravity CLI 1.2.11 for both Gemini models, plus Qwen Code 0.24.1, Kimi Code CLI 2.0.2, Grok Build 1.0.41, ZCode CLI 0.16.9, and DSH 0.1.5-rc.3. The Qwen3.8-Max and Qwen3.8-Flash configurations use xhigh reasoning effort, the other ten use high, and Opus 5.5 (Low) is included at low effort, giving 13 configurations. The output limit is 32,768 tokens per model response, and the agent chooses its own refinement rounds.

Why This Matters

Impact on research. CineRecon shifts graphics-agent evaluation from static reconstruction and scene editing toward joint reconstruction of geometry, actions, and camera behavior from complete cinematic videos. It provides a setting-controlled way to separate errors caused by appearance demands from errors caused by dynamic-event demands, and its automatic judge shows strong alignment with human preferences (Spearman 0.951, Pearson 0.975), which is a prerequisite for scalable benchmarking. It also links directly to the upstream needs of 3D-guided video generation, where rendered scenes act as control proxies.

Real-world applications.

• Filmmaking and previsualization: creators could rebuild a reference shot as an editable scene to adjust layout, action timing, and camera trajectories explicitly.
• Controllable video generation: reconstructions can serve as renderable visual proxies that guide generative video pipelines.
• Game and virtual production: gameplay footage is one of the reference categories, so the same workflow maps onto reusable, editable game scenes.
• Sports and broadcast analysis: the reference set includes sports and racing content, where camera behavior and coordinated subject motion matter directly.

Industry relevance. The benchmark tests frontier multimodal coding models operating through production-style agent harnesses (Codex CLI, Claude Code, Antigravity CLI, and others) inside a real 3D authoring application, with generation cost tracked against performance in the reported Pareto comparison. That makes the results directly relevant to teams building agentic tools for 3D content pipelines, including both model providers and studios.

Future Directions

• Downstream generation effects. The authors state that evaluating how these reconstructed proxies affect downstream video generation is a next step.
• Fixing persistent dynamic failures. Action timing, contact relations, and coordinated subject motion remain incorrect under Proxy, so methods that improve event-level fidelity without extra appearance detail are needed.
• Narrowing the camera gap. Camera rotation and shut-transition reproduction show wide spreads across configurations (median 25.2 versus a high of 74.9 in Detailed), leaving room for viewpoint-control improvements.
• Instance-specific appearance. Both compared models miss details such as weathered plaster, graffiti, and stone bases, indicating open problems in recovering instance-specific appearance under the Detailed setting.

Target Audience

Researchers and engineers working on 3D scene reconstruction, agentic graphics systems, multimodal coding agents, and controllable video generation. It is also useful for benchmark designers interested in checklist-based, entity- and time-grounded evaluation with vision-language judges, and for practitioners in film previsualization, virtual production, and game content pipelines who want to understand the current limits of autonomous reconstruction.

Authors’ abstract

Reconstructing reference videos as editable dynamic 3D scenes is an important path toward controllable video generation for filmmaking: these scenes provide modifiable, renderable visual proxies that enable explicit control over scene layout, subject actions, and camera behavior. To evaluate agents' ability to reconstruct editable dynamic 3D scenes from dynamic reference videos, we introduce CineRecon, a benchmark comprising 50 reference videos and 360 annotated evaluation clips. The reference videos span film, real-world footage, and games. Given a complete video, an agent uses tools to construct, animate, render, and refine its reconstruction. We evaluate this task under three representation settings: Proxy, Simplified, and Detailed. These settings increase geometric and appearance requirements while retaining the same requirements for spatial relations, camera behavior, and subject dynamics. This design tests whether agents can reproduce the events in a video when detailed appearance is unnecessary, and how performance changes as representation requirements increase. For evaluation, we construct checklists that associate observable properties and events with specific entities and times. A vision-language judge compares timestamp-aligned reference and rendered frames across 11 categories, grouped into camera, static, and dynamic consistency dimensions. We evaluate 12 models using their corresponding agent harnesses and find failures across all three dimensions. Subject motions and interactions remain difficult to reproduce even under the Proxy setting.