DyCraft: An Agentic System for Long-Horizon 4D Scene Generation

AI summary

Overview

Research area: Computer vision and computer graphics — specifically text-to-4D (3D + time) scene generation, where LLM-based coding agents author animated 3D content as code for Blender.

Technical level: Advanced. The paper assumes familiarity with 3D content creation pipelines (Blender, rigging, inverse kinematics, B-rep CAD), LLM coding-agent harnesses (Codex, Claude Code), and agentic tool-use design.

Scope in one sentence: DyCraft is an agentic system that restructures long-horizon 3D animation authoring into prompt planning, 3D storyboarding of verified key poses, and per-frame physical verification, so that a general coding agent (GPT-6-Astra in Codex) can produce minute-long, physically plausible animated scenes from text.

What This Paper Is About

Frontier coding agents can already build detailed static 3D scenes and articulated assets, but they fail at coherent long-horizon animation: when handed a general-purpose harness they produce objects that interpenetrate, float without support, or move unnaturally, and they cannot easily repair these errors because they script the entire animation in one pass. The paper's goal is to give such agents an animation workflow rather than better coding ability — expanding a sparse prompt into a plan, building a static scene, storyboarding key poses as individually checkable 3D states, authoring motion only between verified states, and running numerical physical checks on every frame.

Key Contributions

1. DyCraft, a hybrid agent harness for 4D generation. It extends general coding-agent harnesses (Codex, Claude Code) with a custom system prompt and custom operations for generating, probing, and verifying scenes and motion, while leaving authoring autonomy to the agent. The output is an editable Blender project and a final rendered film, produced entirely through code. The authors argue this beats specialized harnesses that prescribe rigid pipeline stages.
2. 3D storyboarding as the decomposition strategy. Drawing on pose-to-pose animation practice, the system requires the agent to author key moments as static 3D scenes, render and verify them through the film's own cameras, and only then author motion between verified start and end states — turning entangled spatiotemporal problems into cheap static checks (e.g., is the doorway wide enough, is the workspace in reach).
3. Domain-specific authoring tools plus a numerical verifier suite. A humanoid motion skill (footstep planning, avoiding backwards-bending knees and elbows), guidance for baking Blender's native rigid-body and liquid simulation instead of hand-keying, a B-rep CAD interface (build123d / Open CASCADE) for precise solid geometry and articulated joints, and verifiers for interpenetration, physical support (graph connectivity), joint limits, and agent-declared assertions, returning actionable diagnostics such as penetration depth and separating axis.
4. DyBench and an evaluation of long-horizon 4D generation. A new benchmark of 24 text-to-4D prompts split equally across three embodiments (humanoid robots — Unitree G1 — human/animal characters, and physical mechanisms), plus human-preference and automated physical-plausibility results against a bare agent and VIGA.

Supporting infrastructure: a rigged character library built from animated subsets of Objaverse and TexVerse (106,976 records; 86,942 distinct shapes, of which 62,205 carry usable motion; 65,430 skinned assets; 68% of 45,692 character rigs map onto a known skeleton family), and two read-only motion-capture libraries — the CMU Graphics Lab Motion Capture Database (2,548 clips, 113 subjects, 120 fps) and dog captures from Zhang et al. 2018 (51 clips, 41 minutes, 60 fps, split into 1,065 labelled spans).

Main Findings

• Large reduction in interpenetration: across all 24 delivered films, the penetration rate (fraction of frames with a dynamic mesh penetration above a 5mm tolerance) falls to 13.0% for DyCraft, versus 80.1% for the bare agent and 74.9% for VIGA.
• Large reduction in floating objects: the floating rate (dynamic body-frames lacking a support path to the ground) is 2.7% for DyCraft, versus 15.7% (bare agent) and 26.1% (VIGA).
• Human raters prefer DyCraft: against VIGA, DyCraft wins 89.7% overall preference, with 92.0% on overall 4D realism, 88.5% on physical plausibility, and 88.5% on motion quality. Against the bare agent it wins 72.3% overall, with 75.6%, 70.1%, and 71.0% on those three aspects.
• User study scale: 109 raters recruited on Prolific, each evaluating five video pairs with 15 binary-choice questions; 1,538 judgments recorded, with judgments discarded where either film played for less than 1 second.
• Residual failures are localized: remaining problems are mainly character–held-object contact (a dog's mouth sinking into a newspaper) and layered parts (stitches on a weight belt), rather than global layout failures.
• Ablations are qualitative: removing prompt-rewriting planning, 3D storyboarding, domain-specific motion tools, or external verifiers all produce planning or physical-plausibility errors (Fig. 8); the paper does not report numeric ablation scores.
• Bare agents default to kinematic scripting: characters keyframed without physical constraints, objects falling at a constant rate rather than accelerating under gravity, and liquids represented as static cylinders; they also build scenes from primitives and hand-written vertex/face lists.
• VIGA fails differently: it produces short animations that omit several key events described in the prompts, whereas the bare agent performs the events but in simpler scenes with lower-quality motion.
• Per-prompt variation exists: preference is not uniform — for example, on the weight-driven clock DyCraft is preferred over VIGA in only 55% (4D realism), 36% (physical plausibility), and 45% (motion quality) of judgments, and on the automated drawbridge physical plausibility drops to 45%.

Methodology in Plain English

The researchers keep a strong off-the-shelf coding agent (GPT-6-Astra running in Codex) and change the process around it.

First, planning: two separate "prompt rewriters" with fresh context expand the user's terse brief — one writes a detailed scene description (layout, what the place accumulates over time, how objects are actually manufactured), the other writes how the performance is physically carried out (grasps, supports, state transitions, required clearances). This matters because bare agents rarely invent detail beyond what the prompt states.

Second, scene construction: the agent builds the static scene, using a CAD-style solid-geometry interface for precise parts and articulated joints, while retaining native Blender tools for deformable characters and fluids, and optionally retrieving a rigged character and motion-capture clips from the provided libraries.

Third, storyboarding: instead of writing the whole animation at once, the agent lays out key poses as static 3D scenes, renders them through the film's cameras, checks them with the verifiers, and only after they pass does it author continuous motion between them using the humanoid motion skill, inverse kinematics, and baked physics simulation.

Fourth, verification: compiling the scene and motion produces a diagnostic report on penetrations, unsupported bodies, and joint-limit violations; a failing check blocks clean completion, and the agent iterates. The agent can also render arbitrary camera views for visual self-review, and can delegate object, scene, or motion subtasks to subagents.

Evaluation pits this against (a) a bare agent given the same harness plus a condensed natural-language statement of the quality standards, and (b) VIGA, on the 24 DyBench tasks, using a human preference study and two automated plausibility metrics.

Why This Matters

The paper argues that the bottleneck for agentic 4D generation is the workflow, not raw coding skill — a finding that generalizes to other long-horizon, constraint-heavy generation tasks where intermediate states can be checked more cheaply than final outputs. It also offers a benchmark (DyBench) and an explicit automated protocol for measuring penetration and floating in generated animations, and it demonstrates that code-based 3D representations remain more controllable than pixel-space video generators for scenes requiring geometric consistency.

Real-world applications:

• Previsualization and animation prototyping for film and games, where a director needs a rough but physically coherent minute-long sequence before committing production resources.
• Game cutscene and level prototyping, since outputs are editable Blender projects rather than opaque video.
• Robot learning and simulation content, generating scenes with humanoid (Unitree G1), human, animal, and mechanism actors for training or evaluation environments.
• Technical and educational visualization, such as the paper's own examples of bottling lines, drawbridges, marble runs, and weight-driven clocks.

Industry relevance: the approach sits directly on top of widely used tools (Blender, Open CASCADE/build123d) and existing coding-agent products, so it is adoptable without a new generative model, and its verifier-first design targets the physical-plausibility failures that block professional use of generative 3D.

Future Directions

• Closing the remaining contact failures. Penetration and floating are reduced but not eliminated (13.0% and 2.7%); character–held-object contact such as grasping and mouth contact, and thin layered geometry, are the named weak points.
• Extending verification coverage. The penetration metric explicitly excludes liquid and effect meshes, persistent static overlaps, directly connected structural parts, and consumed food — suggesting verified handling of fluids, cloth, and deformables as open work.
• Better automated evaluation. The authors state that reliable automated metrics for dynamic scene generation are difficult to design and rely mainly on human preference; standardized, perceptually aligned metrics remain open.
• Scaling horizon, parallelism, and tool generality. The delegation mechanism (object/scene/motion subagents) and the minute-long target raise questions about how far the approach stretches in duration and scene complexity, and whether the same harness design transfers beyond Blender and beyond a single base model.

Target Audience

Researchers and practitioners in 3D/4D generative modeling and computer graphics who care about controllability and physical plausibility; LLM-agent researchers interested in harness design, tool decomposition, and verifier-driven iteration for long-horizon tasks; and technical artists or pipeline engineers in animation, games, and robotics simulation who want to understand where agent-authored Blender content is currently reliable and where it still fails. Readers seeking new learned 4D generative models or new training objectives will not find them here — the contribution is system and workflow design.

Authors’ abstract

While frontier coding agents can generate high-quality 3D scenes and articulated assets, they struggle to create coherent long-horizon animations. When equipped with general-purpose harnesses, they frequently produce physically implausible outputs, generating objects that interpenetrate, lack physical support, or move unnaturally. Such bare agents struggle to repair these errors because they default to scripting the full animation in one pass, making it difficult to verify…