PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control

AI summary

Overview

Research area: Controllable generative video — specifically physics-grounded, interactive image-to-video synthesis with autoregressive (streaming) diffusion models.

Technical level: Advanced. The paper assumes familiarity with diffusion models, autoregressive video generation, latent-space conditioning, and monocular depth/segmentation estimators.

Scope: PhysStream is an autoregressive image-to-video model that lets users push individual objects in a multi-object tabletop scene by injecting sparse velocity changes mid-generation, while an online "scene memory" derived from the model's own past frames keeps the simulated physics coherent.

What This Paper Is About

Existing controllable video generators mostly force users to specify the entire control schedule before generation begins, and they typically use pixel-space signals (drag points, dense tracks, masks) that dictate where an object should go rather than how it should physically move. This paper asks whether a model can instead accept a small, physically meaningful nudge — a velocity change applied to one object at one moment — and then autonomously produce the resulting collisions, tumbles, and chain reactions across a whole scene. The goal is interactive, physics-grounded, scene-level control where the user watches frames appear and intervenes whenever they like.

Key Contributions

1. First end-to-end interactive physics-grounded control at scene level. PhysStream is presented as the first method enabling direct, end-to-end interactive control in multi-object tabletop rigid-body scenes, where both the user's physical input and the model's memory operate on the generated video itself — without an external 3D reconstruction or physics simulator at inference.

2. Structured scene memory. A new conditioning mechanism for autoregressive video generation consisting of a normalized positional map (per-pixel 3D camera-frame coordinates from monocular depth) and an object-tracking map (per-object colored masks from SAM2), both updated online after each generated frame and shown to improve geometric consistency and physical plausibility.

3. A 100k-video synthetic dataset. A curated corpus of indoor tabletop scenes rendered with Blender and simulated in PyBullet, containing complex multi-object rigid-body motion, collisions, and multi-frame velocity perturbations, intended as a resource for improving physical correctness in video generation.

4. A two-stage training recipe. A bidirectional backbone first learns velocity-increment control alone; a causal autoregressive model is then trained on top of it to learn scene memory and causal attention jointly, keeping each distribution shift small while avoiding future-frame leakage.

Main Findings

• Motion distribution distance drops sharply. On synthetic test set (ii) with multi-object interactive control, PhysStream reaches FVMD 787.0 versus 1183 for RealWonder and 1463 for Tora, corresponding to roughly a 33% reduction over the strongest baseline.

• Trajectory tracking improves. Traj-ADE falls to 40.24 (from 45.67 for FlashMotion and 60.91 for RealWonder), and the median variant to 32.00 — roughly a 12% trajectory-error reduction over the strongest baseline — with the lowest failure rate at 43.15.

• Human evaluators strongly prefer it. On in-the-wild comparisons, PhysStream wins 91.8% of physical-plausibility judgments, 88.4% on motion accuracy, and 91.2% on visual quality; MLLM scores are also highest (SA 5.00, PC 4.15 versus 4.85/3.65 for the next best).

• Scene memory is the key ablation delta. Adding both positional and tracking maps (row e) gives the best FVMD (879.8) and photometric consistency (80.37) compared with velocity-only (941.3) or either map alone.

• Positional maps matter most for depth-changing objects. Grouping test objects by ground-truth depth displacement, the positional-map gain grows from +2.1% overall to +15.5% for the top 10% of objects by depth change, indicating the map helps most when 3D structure actually shifts.

• Generalization beyond the training distribution holds. On OCID real cluttered scenes the model scores SA 5.00 / PC 4.06, and on the Physics-IQ solid-mechanics subset it achieves 47.86 with SA 4.80 — even modeling a rolling-ball collision correctly despite appearance drift on out-of-distribution input.

• The paradigm extends to deformables. Finetuned models handle elastically bouncing balls (SA 4.20, PC 4.20) and fluttering cloth (SA 5.00, PC 5.00) using the same control and memory scheme.

• Consistency metrics can be misleading. The authors show DragStream produces a nearly static scene yet earns the highest consistency score, while FlashMotion scores comparably to PhysStream despite visible artifacts — so consistency numbers are reported only for completeness.

• Teacher forcing beats distillation here. Under equal budget, Teacher-Forcing training outperformed Diffusion-Forcing and Self-Forcing, largely because rollout-based objectives would require running the online estimators inside every training rollout.

Methodology in Plain English

The researchers start from a pretrained bidirectional video diffusion model (Wan2.2-TI2V-5B) and teach it to accept a new kind of control signal: a velocity-increment map. When a user wants to shove an object, they pick that object and specify a 3D velocity change, which gets "painted" onto the object's mask as a color-coded signal. Neutral gray means no change; other colors encode direction and magnitude along each axis. Crucially, this signal is anchored to the object's position in the first frame rather than its current position — a deliberate training-time convention that prevents the control channel from secretly leaking the object's future trajectory.

Training happens in two stages. Stage 1 finetunes the bidirectional model on velocity conditioning alone, with all frames denoised jointly. Stage 2 converts the model to causal, frame-by-frame generation with KV caching, and adds two more conditioning streams: a positional map (3D coordinates per pixel, computed by running Depth-Anything-3 on the last few generated frames) and a tracking map (each tracked object painted a distinct palette color by SAM2). These memory maps are never given to the bidirectional stage, because under bidirectional attention they would leak future scene state.

At inference, the model generates one latent frame at a time. After each is committed, the frames are decoded to pixels, the two estimators run on the new output only (using cached features, so per-step cost is constant regardless of video length), and the updated maps are encoded and fed back as conditions for the next frame. All conditioning enters through channel-wise concatenation after a one-frame temporal shift, so under causal attention each frame only ever sees information derived from strictly earlier content. New conditioning branches are zero-initialized so the augmented model starts numerically identical to the pretrained backbone.

Why This Matters

Impact on research. The paper reframes controllable video generation around physical quantities rather than positional targets: instead of telling the model where things should be, you tell it what force to apply and let it work out the consequences. It also shows that a generative model's own outputs can be recycled as structured feedback (depth plus tracking) to improve physical consistency — a feedback-loop idea that could transfer to other autoregressive video systems. The released 100k-video benchmark gives the physics-grounded video community a training and evaluation resource with ground-truth trajectories.

Real-world applications.

• Creative and VFX tooling: animators could nudge objects in a shot and have collisions, toppling, and chain reactions resolved automatically rather than keyframed by hand.
• Robotics and manipulation planning: a model that predicts how objects respond to applied velocity changes can serve as a learned forward-dynamics prior for tabletop manipulation.
• Interactive world models for games and simulation: streaming generation with mid-flight user intervention maps naturally onto playable environments where scene state must persist.
• Content creation at scale: single-image-to-video with sparse interaction lowers the authoring cost for e-commerce, advertising, and social media clips of product or object motion.

Industry relevance. The work comes from a university–industry collaboration (University of Pennsylvania, Snap Inc., KAUST) and targets SIGGRAPH Asia, indicating direct relevance to visual-effects, AR, and consumer content pipelines. The fact that inference needs no external simulator — only lightweight depth and segmentation estimators running incrementally — makes deployment realistic on commodity hardware in a way that simulator-in-the-loop approaches are not. The trade-off is that the model generalizes from synthetic tabletop rigid-body data, so domain shift on messy real footage remains a practical hurdle.

Future Directions

• Extending beyond rigid-body tabletop scenes. The paper's own non-rigid experiments (bouncing balls, cloth) are promising but preliminary, and articulated objects, fluids, and free camera motion are untouched.

• Scaling to longer horizons and heavier scenes. Structured memory is designed for constant per-step cost, but long videos with many objects may still accumulate error or drift; how the memory degrades over hundreds of frames is unresolved.

• Reducing reliance on ground-truth masks and synthetic supervision. Training uses rendered ground-truth instance masks and simulated dynamics; replacing these with self-supervised or foundation-model-derived signals would broaden applicability.

• Beyond velocity: richer physical controls. The current interface only exposes velocity increments. Forces, torques, friction, and material properties are natural extensions that would give users finer physical authorship.

• Better evaluation of physical plausibility. The authors explicitly critique consistency metrics as gameable by degenerate near-static outputs; the field still lacks a robust automatic measure of physical correctness, which this benchmark partially addresses but does not solve.

Target Audience

This paper is best suited for graduate students and researchers working on video diffusion, controllable generation, or physics-aware generative models, as well as graphics and VFX practitioners interested in interactive authoring tools. Roboticists studying learned forward dynamics and world models will also find the velocity-conditioning formulation relevant. Readers need a working understanding of diffusion sampling, latent video representations, and causal attention to follow the method sections, though the high-level framing and results are accessible to anyone familiar with generative video.

Authors’ abstract

Interactive control for video generation is moving from coarse prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Yet existing controllable methods either require the full control schedule before generation starts, or use pixel-space signals that dictate object positions rather than physical dynamics. To address these limitations, we propose PhysStream, an autoregressive model for physics-grounded image-to-video synthesis that incorporates structured scene memory---positional maps and object tracking maps derived online from previously generated frames---and supports fine-grained motion control via sparse velocity-increment signals that encode physical quantities, letting the model learn the underlying dynamics. We train our model in two stages: a bidirectional model is first finetuned with motion-control conditioning, then a causal autoregressive model is trained with additional structured scene memory, further improving physical consistency. PhysStream enables interactive, mid-generation control over multi-object tabletop rigid-body scenes---a capability not supported by prior methods---reducing motion distribution distance (FVMD) by 33% and trajectory error by 12% over the strongest baselines on synthetic benchmarks, and is preferred by human evaluators in over 85% of in-the-wild comparisons. Please check our website for more details: https://czzzzh.github.io/PhysStream