PhysVideoGenerator: Towards Physically Aware Video Generation via Latent Physics Guidance

AI summary

Overview

Research area: Computer Vision — generative video modeling, specifically physics-aware latent video diffusion and representation alignment with predictive world models.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiT), latent diffusion, cross-attention conditioning, JEPA-style predictive architectures, and multi-task optimization.

Scope: A proof-of-concept feasibility study that embeds a learnable physics prior (regressed from V-JEPA 2 features) into the temporal attention layers of a Latte-1 video diffusion backbone, validating joint-training stability over 50 epochs without performing generative quality evaluation.

What This Paper Is About

Current video generation models produce visually pleasing videos but often fail to capture real-world physics, producing artifacts such as unnatural object collisions, inconsistent gravity, and temporal flickering. Existing fixes rely either on massive scale or on expensive external simulators, neither of which offers a tractable general-purpose path. This paper asks whether a small auxiliary network can predict high-level physical representations directly from noisy diffusion latents and feed them back into the generator, and whether training such a system jointly with the diffusion objective is stable.

Key Contributions

1. Architectural design of PhysVideoGenerator. A framework that integrates a physics predictor branch (PredictorP) into a DiT-based video generator, regressing V-JEPA 2 embeddings from noisy latents, and injecting the resulting physics tokens into the temporal attention layers of Latte via a dedicated cross-attention mechanism.

2. Implementation of a joint training pipeline. A training loop that simultaneously optimizes the diffusion noise-prediction loss and the physics regression loss, requiring solutions for tensor expansion and memory management on A100-class hardware, plus a streaming dataset pipeline that converts raw videos into aligned latent, visual, and textual embeddings without storing intermediate video files.

3. Feasibility validation over 50 training epochs. Demonstration that the physics loss converges alongside the diffusion noise loss without oscillation or divergence, confirming that diffusion latents contain sufficient information to recover V-JEPA 2 physical representations.

4. A proposed evaluation protocol for future work. A planned suite covering optical flow consistency, VideoCLIP-based motion consistency, T-LPIPS perceptual temporal consistency, and VideoPhy / VideoPhy-2 semantic adherence and physical correctness.

Main Findings

• Both losses decreased simultaneously without divergence. Over 50 epochs on a subset of OpenVid-1M, the diffusion loss and physics prediction loss both fell, which the authors describe as evidence that the multi-task objective is well-posed rather than suffering from competing gradients.

• Diffusion loss followed a standard trajectory. The noise prediction loss decreased steadily over the first 10 epochs before entering a slower convergence regime, suggesting the added physics conditioning branch does not fundamentally disrupt diffusion learning dynamics.

• Physics loss converged faster than diffusion loss early on. This suggests the latent-to-physics mapping is learnable even from partially denoised representations.

• Memory optimizations were necessary and effective. Gradient checkpointing reduced peak memory by approximately 40 percent; BF16 mixed precision reduced memory footprint while maintaining numerical stability; freezing the VAE, text encoder, and spatial attention blocks focused trainable parameters on the physics-relevant modules.

• Latte was chosen over CogVideoX-2B for resource reasons. Attempts to build on CogVideoX-2B produced persistent out-of-memory failures given access to at most two A100 NVIDIA GPUs; Latte's factorized spatial–temporal attention was more memory-efficient.

• No generative quality was measured. The authors explicitly state they did not run inference to assess whether the trained model produces physically plausible videos, did not train at sufficient scale (50 epochs on a data subset is described as insufficient), and did not perform ablations or baseline comparisons.

• Hardware reporting is inconsistent across the paper. Section 3.1 describes training on "1 NVIDIA A100 (40GB) GPU provided on Google Cloud Burst" and Section 4.1 states "a single NVIDIA A100 (40GB) GPU," while Section 4.3 states the optimizations "enabled training on a two 40GB A100 GPUs." The paper does not reconcile these statements.

Methodology in Plain English

The team starts from Latte-1, a diffusion transformer that already generates video from text and splits its attention into spatial and temporal parts. They leave the spatial side alone and modify the temporal side.

Their added piece is a small network called PredictorP. During training, a video is compressed into a compact latent by a pretrained VAE. Noise is added to that latent, and PredictorP is asked to look at the noisy latent, the text embedding, and the timestep, and then guess what V-JEPA 2 would have said about the clean video. V-JEPA 2 is a pretrained model that predicts how scenes evolve, so its internal features are treated as a stand-in for "physics."

PredictorP does this in three stages: a small 3D convolution encodes the noisy latent, a transformer encoder fuses that with the text and timestep embeddings, and a transformer decoder with 2048 learnable queries cross-attends to the fused representation to produce predicted physics tokens.

Those predicted tokens are then plugged into the generator's temporal blocks through a new physics cross-attention layer, placed right after temporal self-attention. Every spatial patch's motion across frames is conditioned on the same predicted physics tokens, so physics acts as a global constraint on how things move.

Training minimizes two things at once: the usual diffusion error (how badly the model predicts the noise that was added) and the physics error (how far PredictorP's output is from the real V-JEPA 2 tokens), with the physics term weighted by λ = 0.1. The VAE, text encoder, and spatial transformer blocks are frozen; only PredictorP and the new temporal cross-attention layers are trained, using AdamW with learning rate 1e-5 and weight decay 0.01.

The data setup: OpenVid-1M video-text pairs, processed at 256×256 resolution with 16 frames. Videos are encoded by the Latte-1 VAE from a 3×16×256×256 tensor down to a 4×16×32×32 latent (spatial downsampling factor of 8). Text is encoded by T5-v1.1-XXL at a fixed sequence length of 226, giving 226×4096 embeddings. Physics features come from V-JEPA 2's final encoder layer as 2048×1408 tokens, where 2048 corresponds to flattened spatiotemporal patches (16×16 spatial times 8 temporal).

At inference time, the pipeline starts from Gaussian noise, and at each reverse diffusion step runs the predictor on the current latent to produce physics tokens, which are injected with a learned gate controlling conditioning strength, then denoises with an epsilon-prediction DDPM sampler and decodes with the VAE. A "dry-run" mode skips checkpoint loading and runs a single denoising step on one prompt for debugging.

Why This Matters

Impact on research. The paper's central claim is deliberately narrow: it does not show improved video quality, but it does show that diffusion latents carry enough information to recover high-level predictive representations, and that generative and predictive objectives can be optimized together without destabilizing training. That is a precondition for any future physics-aware generation work, and the paper frames itself as a methodological foundation rather than a results paper. It also positions itself against two existing extremes — implicit scaling (hoping physics emerges from data) and explicit simulation (binding generation to rigid physics engines) — and against general representation alignment work such as Video-REPA, arguing that aligning specifically to a predictive world model targets physical commonsense (gravity, object permanence, interaction dynamics) rather than mere temporal smoothness.

Real-world applications (the paper names the first two explicitly; the others follow from the stated goal of physically consistent video generation):

• Cinematic production, where physically implausible motion breaks immersion.
• World simulation, where generated environments must obey consistent dynamics.
• Any downstream setting requiring temporal coherence and long-horizon prediction, which the authors identify as the setting where current models' lack of physics structure becomes visible.
• Applications where physical consistency matters but external physics engines do not generalize, since the paper notes simulators struggle on "in-the-wild" scenes.

Industry relevance. The paper's engineering narrative is about accessibility: the authors argue that their combination of gradient checkpointing, BF16 mixed precision, component freezing, and the choice of Latte over CogVideoX-2B makes the approach tractable on academic compute budgets rather than frontier-scale clusters. The authors report access to at most two A100 GPUs, and the report documents the memory failures that forced the backbone switch — a practical datapoint for teams with limited hardware. They also release the project repository at https://github.com/CVFall2025-Project/PhysVideoGenerator. The honest negative framing — a predictor plus cross-attention module bolted onto an existing generator, trained and monitored but never evaluated — is also useful as a realistic account of what a proof-of-concept at this scale can and cannot establish.

Future Directions

1. Run inference and evaluate generative quality. The authors state that assessing whether the trained model produces physically plausible videos requires substantial additional compute and systematic evaluation, and remains future work. They outline a protocol using Farnebäck optical flow, VideoCLIP embedding cosine similarity between consecutive frames, T-LPIPS with a VGG backbone, and VideoPhy semantic adherence and physical correctness, extending to VideoPhy-2 for longer temporal horizons and complex interactions.

2. Ablation studies to attribute any effects. The paper proposes three variants: the original Latte backbone without physics injection, PredictorP trained but with cross-attention disabled, and a fully frozen backbone ("w/o LoRA"), plus comparisons against OpenSora, VideoCrafter2, and HunyuanVideo on prompts designed to test physical interactions.

3. Make the physics representation cheaper. The current 2048×1408 V-JEPA 2 token representation is described as memory-intensive; the authors propose investigating compressed or pooled representations to improve scalability.

4. Train at scale and on other backbones. The authors acknowledge that 50 epochs on a data subset is insufficient to fully train either the predictor or the adapted diffusion layers, and suggest revisiting more capable backbones (such as CogVideoX) with larger compute budgets, along with inference-time physics guidance variants that modulate conditioning strength during sampling.

Target Audience

Researchers and graduate students working on video diffusion, physically grounded generative models, or representation alignment between generative and predictive architectures will get the most from this paper. It is also useful for practitioners with constrained compute budgets, since a substantial portion of the report is about what failed on limited A100 hardware and which engineering choices (backbone selection, gradient checkpointing, BF16, freezing) made joint training possible. Readers looking for benchmark numbers, state-of-the-art comparisons, or demonstrated improvements in video quality will not find them here — the authors explicitly record those as absent.

Authors’ abstract

Current video generation models produce high-quality aesthetic videos but often struggle to learn representations of real-world physics dynamics, resulting in artifacts such as unnatural object collisions, inconsistent gravity, and temporal flickering. In this work, we propose PhysVideoGenerator, a proof-of-concept framework that explicitly embeds a learnable physics prior into the video generation process. We introduce a lightweight predictor network, PredictorP, which regresses high-level physical features extracted from a pre-trained Video Joint Embedding Predictive Architecture (V-JEPA 2) directly from noisy diffusion latents. These predicted physics tokens are injected into the temporal attention layers of a DiT-based generator (Latte) via a dedicated cross-attention mechanism. Our primary contribution is demonstrating the technical feasibility of this joint training paradigm: we show that diffusion latents contain sufficient information to recover V-JEPA 2 physical representations, and that multi-task optimization remains stable over training. This report documents the architectural design, technical challenges, and validation of training stability, establishing a foundation for future large-scale evaluation of physics-aware generative models.