Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generation

AI summary

Overview

Research area: Computer vision / generative video — specifically long-form, multi-shot agentic video generation driven by screenplays, with explicit spatial and temporal memory.

Technical level: Advanced. The paper is written for readers comfortable with video diffusion/generation systems, 3D scene reconstruction (point clouds, Gaussian Splatting-style representations), camera control, and agentic plan–act–observe loops. The core idea, however, is explained in reasonably plain language.

One-sentence scope: The paper proposes ATAS ("Any Time, Any Space"), a framework that turns a screenplay into shots by maintaining a persistent, validated world state of static scene geometry and time-indexed event evidence, so that later shots can return to an earlier space or replay an earlier event.

What This Paper Is About

Existing agentic video systems keep continuity between shots using recent frames, text summaries, or retrieved reference images. Those cues work for neighbouring shots but provide no shared spatial frame for revisiting a location and no time-indexed record for replaying an earlier event from a new camera. ATAS addresses this by compiling the screenplay into explicit shot requests (with location, camera, and event-time references), retrieving relevant evidence from a scene memory and an event memory before generation, and only writing generated clips back into that memory when they pass validation.

Key Contributions

1. A world-state-centric framework (ATAS) for screenplay-driven video generation built around persistent scene and event memories, so later shots can query evidence from earlier observations "in any space and any time."
2. Two complementary memory designs — a spatially registered scene memory holding accepted static geometry in a shared coordinate frame, and a time-indexed event memory holding dynamic observations keyed by scene, event, and time — plus a validation-controlled update mechanism that separates film acceptance from admission of new evidence into persistent state.
3. A two-query protocol for evaluating persistence: spatial revisit (return to a previously observed space and camera) and historical event recapture (regenerate an earlier event from a requested camera using source-grounded evidence rather than a synchronized target video).
4. Empirical validation across four video generators (MiniMax-H3, Wan 2.7, HappyHorse 1.0, Wan 3.0) against five agent baselines (I2V Extension, VGoT, Mora, MovieAgent, ReCA) on VBench, ST-Bench, ViStory-Self, ViStory-Cross, spatial-revisit metrics, event-recapture criteria, and human ratings.

Main Findings

• Cross-shot consistency improves on every reported measure. ATAS leads every reported aggregate on MiniMax-H3, Wan 3.0, and HappyHorse 1.0. On Wan 3.0 it reaches 0.877 on VBench, 0.719 on ST-Bench, 0.756 on ViStory-Self, and 0.584 on ViStory-Cross, for an average of 0.734, compared with 0.648 for ReCA on the same backbone. With Wan 2.7 it stays ahead on ST-Bench and both ViStory consistency measures despite trailing on VBench.
• Target-view spatial revisit is the strongest result. ATAS records 15.968 PSNR_C, 0.385 SSIM_C, and 0.284 LPIPS_C under a common-visibility mask, ahead of the 3D-guided methods ViewCrafter (15.255 PSNR_C, 0.375 SSIM_C, 0.293 LPIPS_C) and FlexWorld (14.740 PSNR_C, 0.343 SSIM_C, 0.315 LPIPS_C) and the agent baselines.
• Event recapture leads on three of five criteria. ATAS scores 0.750 on event identity, 0.688 on temporal order, and 0.650 on viewpoint realization, ties ReCA on character and object consistency (0.917), and is slightly lower on action-state preservation (0.688 versus 0.700 for ReCA). The paper attributes the remaining gap to exact action-state reproduction rather than retrieval of the correct occurrence or view.
• Human raters rank ATAS first on all six criteria. On a five-point anonymized protocol across six methods, ratings are highest for script faithfulness (4.40), character consistency (4.22), and visual appeal (4.10), while background consistency (3.62) and narrative coherence (3.77) are lower.
• Memory is what drives the gains, and recursive compilation amplifies them. In the component ablation with Wan 3.0, adding scene and event resolution raises ViStory average from 0.478 to 0.612 and the event score from 0.441 to 0.508; recursive compilation pushes the event score to 0.711 and C-Top10 to 0.684; observation validation and the state-update check reach 0.670 ViStory average, 0.719 C-Top10, and 0.739 event recapture.
• Scene and event memory contribute differently. In the memory ablation, C-Top10 rises from 0.536 with no memory to 0.719 with full state, the event score from 0.452 to 0.739, and LPIPS_C falls from 0.341 to 0.284. Adding static scene evidence lowers LPIPS_C by 0.030, updating the scene raises C-Top10 by 0.041, and event memory gives the largest recapture gain.
• Persistence cuts retries at a measurable compute cost. Retries fall from 1.00 to 0.54 per shot and retry-adjusted delivery time drops from 359 to 339 seconds per shot, but per-story runtime rises from 51.9 to 64.9 minutes, memory overhead reaches 98 seconds per shot, and throughput falls from 7.28 to 6.21 shots per hour.
• Film-review rejection falls as evidence accumulates. Rejection declines monotonically from 52.6% with no memory to 36.8% with full state, with intermediate settings following the same direction.
• State update and film delivery are decoupled. A clip can be kept in the final film even when registration or admission is rejected, so unaligned observations never condition later shots.

Methodology in Plain English

The input is a high-level screenplay, not a list of finished prompts. A compiler recursively splits it into bounded shot requests, each carrying a local description, a scene identifier, a spatial anchor, a target camera, a requested time interval, and an optional event identifier. Before generation, the system queries two memories: a scene memory that stores accepted static geometry as a point cloud in a shared coordinate frame and can be rendered from the requested camera, and an event memory that stores dynamic observations as records keyed by scene, event, source time interval, and frame-level RGB-D, mask, and dynamic point-cloud evidence. The rendered scene reference and any retrieved event reference are combined into a context-dependent prompt that conditions a short-video generator.

After generation, the clip is treated as a candidate observation rather than trusted state. Three checks gate the update: film review decides whether the clip is delivered, registration decides whether the reconstruction aligns to the scene coordinate frame, and a state-update check requires valid provenance, scene scope, and matching parent-state version. The world state is replaced only when all three pass; otherwise it is unchanged and the clip remains available only as film. Evaluations use official inputs and evaluators for VBench, ST-Bench, ViStory-Self, and ViStory-Cross, with methods in each backbone block sharing the generator, prompt package, duration budget, and aggregation protocol. Spatial revisit is measured against target images under a common-visibility mask; event recapture uses a fixed source-grounded VLM judge with a versioned rubric, mapping binary decisions to 0 or 1 and ordinal ratings from 1 to 5 to (r−1)/4, with no synchronized target video.

Why This Matters

Impact on research. The paper reframes multi-shot video generation from an editorial problem (what prompt or reference image comes next) into a state problem (what is known about the world, and is that knowledge trustworthy). It also makes an explicit design separation between generating a film and admitting evidence into a model's memory, and contributes an evaluation protocol for event recapture that avoids requiring a synchronized target video.

Real-world applications:

• Interactive filmmaking, where a director can return to a set later in the story and keep its layout consistent.
• Episodic content creation, where recurring locations and events must stay recognizable across episodes.
• Previsualization, letting creators inspect a scene or replay an event from a new camera before committing to production.
• Camera-controlled reshooting of an already established scene, which the paper situates against dynamic view-synthesis and 3D-guided baselines such as ViewCrafter and FlexWorld.

Industry relevance. Because ATAS improves consistency across both open-source and proprietary backbones, its gains appear to come from the persistent state rather than from one renderer — an important property for studios and tool vendors who switch or mix generation models. The efficiency numbers also matter commercially: fewer retries per shot

Authors’ abstract

Long-form agentic video generation must turn a screenplay into shots that remain spatially and temporally consistent across nonconsecutive requests. Passing prompts or recent frames between shots does not provide a shared spatial frame for revisiting a scene or a time-indexed record for replaying an earlier event. We present Any Time, Any Space (ATAS), which compiles a screenplay into shot requests with explicit location, camera, and event-time references. Before generation, it retrieves relevant evidence from a scene memory of registered static geometry and an event memory of dynamic observations indexed by scene, event, and time. After generation, ATAS reconstructs and registers the clip. It updates memory only when review and state checks accept the observation, while film delivery remains independent of memory admission. Across four video generators, ATAS improves all reported cross-shot consistency measures and achieves the highest aggregate benchmark score for each backbone. It also outperforms the compared agents and 3D-guided methods on target-image spatial revisit, leads three event-recapture criteria, and receives the highest human ratings on all six criteria. The video results and code are available on our project page: https://lhmd.top/atas/.