PanFlow: Decoupled Motion Control for Panoramic Video Generation

AI summary

Overview

• Research area: Computer Vision — motion-controllable 360° panoramic video generation using diffusion models.
• Technical level: Advanced. The paper assumes familiarity with diffusion transformers, optical flow, spherical (equirectangular) geometry, SLAM, and LoRA fine-tuning.
• Scope: The paper introduces a framework that generates temporally coherent panoramic videos from a single image plus a user-supplied motion signal, by separating camera rotation from the remaining motion (translation and object dynamics) and by warping noise across panorama boundaries.

What This Paper Is About

Panoramic video generation has advanced through diffusion models, but existing methods lack explicit motion control and struggle with large, complex motion. Panoramic video is harder than ordinary video because content exiting one edge must reappear seamlessly at the opposite edge, and any mismatch produces visible seams. The goal of PanFlow is to let a user specify motion precisely for a 360° video, while keeping the left-right boundary and the poles visually and motion-wise consistent.

Key Contributions

1. PanFlow framework: The authors describe it as the first motion-controllable panoramic video generation framework capable of synthesizing temporally coherent 360° videos from a single image while following a user-defined motion flow.
2. Decoupled motion control: A mechanism that isolates camera-induced rotation flow from translation and object motion, so the diffusion model is conditioned only on a "derotated" residual flow, with rotation reapplied afterwards. A spherical noise warping strategy accompanies it, propagating noise across panorama boundaries to preserve loop consistency.
3. A curated large-scale dataset: Roughly 150k motion-rich panoramic video clips (3–10 s each) with frame-level optical flow and camera pose annotations, built from the 360-1M corpus.
4. Strong empirical results: Experiments across two benchmarks (360-1M and WEB360) report that PanFlow outperforms prior methods in motion fidelity, motion magnitude, temporal coherence, and visual quality.

Main Findings

• Best overall scores on both benchmarks: On the 360-1M test set, PanFlow reaches FVD 298.39, FID 23.01, CLIP 27.14, Flow EPE 2.011, LPIPS 0.243, PSNR 21.21, and SSIM 0.723. On WEB360 it reaches FVD 195.77, FID 19.44, CLIP 26.84, Flow EPE 0.486, LPIPS 0.185, PSNR 27.46, and SSIM 0.842 — the best value on every listed metric in both tables.
• Outperforms the strongest motion-conditioned baseline: GoWithTheFlow scored FVD 477.13 and Flow EPE 3.297 on 360-1M, and FVD 319.01 and Flow EPE 0.710 on WEB360, both behind PanFlow.
• Human-aligned quality also improves: On 360-1M, PanFlow records Q-Align Image Quality 0.5552, Image Aesthetic 0.4397, and Video Quality 0.6196; on WEB360 these are 0.7014, 0.6165, and 0.7515 — the highest among the compared methods in both tables.
• Panoramic baselines trail substantially: On 360-1M, 360DVD reports FVD 1663.61 and FID 152.04, DynamicScaler 1360.94 and 85.25, and HoloTime 1013.32 and 59.59; MotionClone reports FVD 1435.66 and FID 87.60.
• Decoupled motion control works as a plug-in: Injecting it into GoWithTheFlow improves most metrics, including End Continuity (0.1389 to 0.0276) and Flow EPE (5.754 to 4.818). Removing it from the full PanFlow model degrades every metric in the ablation table.
• Each loop-consistency component matters: In the ablation, removing circular padding sharply raises the End Continuity error (0.0264 to 0.0512) while other metrics stay similar; removing latent rotation (0.0287) or spherical noise warping (0.0286) raises continuity error less but harms Flow EPE and PSNR, and removing spherical noise warping also worsens FVD (441.59) and FID (26.04).
• Decoupling also shifts the seam: The authors note that decoupled motion control rotates the seam away from the boundary in the generated derotated video, which they say presumably "hacks" the continuity metric — so they evaluate loop-consistency variants on the ablated model without decoupling.
• Beyond generation, the method transfers and edits: Application examples include motion transfer from a source video onto a different target scene, and video editing where ChatGPT 4o edits the first frame and prompt before regenerating with the source motion.

Methodology in Plain English

The system takes a single panoramic image, a text prompt, and a motion signal taken from a reference video.

1. Separate rotation from everything else. A 360° panorama is a spherical scene flattened onto a rectangle. On a sphere, camera rotation produces a predictable, scene-independent flow pattern around the rotation axis. The authors compute that part analytically and subtract it from the estimated optical flow, leaving a "derotated" flow that contains only camera translation and object motion. The video model then only has to learn the derotated motion, and the rotation is re-applied afterwards by rotating each generated frame back.
2. Warp the noise instead of embedding the flow. Rather than feeding flow as a separate conditioning embedding, they warp the initial random Gaussian noise of the diffusion process from frame to frame using the flow. Because the noise determines the content in diffusion models, this gives both motion control and temporal coherence. Pixels that leave through a boundary wrap around to the opposite side, and pole-crossing pixels are handled by reflecting coordinates.
3. Keep the loop intact in three places. They roll the latent map along longitude by a fixed angle at each denoising step (accumulated and reversed at the end) so the seam is blended inside the frame instead of sitting at the edge; they add circular padding to the 3D-VAE encoder (8 pixels) and decoder (1 latent pixel); and the spherical noise warp itself wraps across boundaries.
4. Train on motion-rich data. They filter 360-1M down to equirectangular, watermark-free, high-motion clips, segment them with scene detection and panoramic SLAM, and annotate poses and captions. They fine-tune a CogVideoX-5B-I2V model (49 frames, 480×720) with LoRA rank 128, batch size 32 for 15k steps, AdamW at learning rate 2×10⁻⁵ on 8 NVIDIA A100 GPUs for about 2 days. Inference uses DDIM with 30 steps and a 40° rotation angle, with output resized to 480×960.

Why This Matters

• For research: It reframes panoramic motion control as a decomposition problem — analytic rotation plus learned residual — rather than asking a single model to absorb all motion types. It also adds a motion-rich, annotated panoramic dataset and a boundary-aware noise-warping technique that other spherical generation methods can adopt.
• Virtual reality: Generating 360° footage from a single image with controllable camera and object motion supports immersive headset content with far less capture effort.
• Immersive storytelling and film: Directors can specify camera movement directly and repurpose motion from existing footage onto new scenes.
• Motion transfer: A dynamic camera move from one video can be applied to an unrelated panorama while preserving the target scene's geometry.
• Video editing: Editing a single first frame and prompt (as demonstrated via ChatGPT 4o's image generation) and regenerating the panorama under source motion makes content revisions cheaper.
• Industry relevance: The pipeline targets production of immersive media and virtual environments, and is relevant to companies building VR content pipelines, synthetic-data generators, and 3D reconstruction tools. The authors state they plan to release the code, dataset, and models.

Future Directions

• Replace SLAM-based rotation estimation: The authors identify small-parallax conditions as a weakness of their SLAM-based rotational estimation and suggest a learned derotation module as a fix that would also reduce reliance on reference videos.
• Close the metric gap on loop consistency: Since the rotated seam can artificially improve the end-continuity metric, better evaluation and generation methods for seam quality are still open.
• Scale and diversity of the dataset: With 53% natural, 22% urban, 9% indoor, and 16% other content, extending coverage to new scene categories and longer clips is a natural extension.
• Broader conditioning modalities: The current design uses image, text, and flow; whether pose, depth, or higher-level scene controls can be integrated alongside the decoupled motion remains unexplored.

Target Audience

This paper suits graduate students and researchers working on video diffusion models, panoramic or spherical content generation, and motion transfer; engineers building VR, immersive media, or synthetic video pipelines; and practitioners who need controllable camera and object motion for content creation. Readers should already be comfortable with diffusion models, optical flow, and equirectangular projection to get the most from the method section, though the high-level ideas and the empirical tables are accessible to anyone tracking generative video progress.

Authors’ abstract

Panoramic video generation has attracted growing attention due to its applications in virtual reality and immersive media. However, existing methods lack explicit motion control and struggle to generate scenes with large and complex motions. We propose PanFlow, a novel approach that exploits the spherical nature of panoramas to decouple the highly dynamic camera rotation from the input optical flow condition, enabling more precise control over large and dynamic motions. We further introduce a spherical noise warping strategy to promote loop consistency in motion across panorama boundaries. To support effective training, we curate a large-scale, motion-rich panoramic video dataset with frame-level pose and flow annotations. We also showcase the effectiveness of our method in various applications, including motion transfer and video editing. Extensive experiments demonstrate that PanFlow significantly outperforms prior methods in motion fidelity, visual quality, and temporal coherence. Our code, dataset, and models are available at https://github.com/chengzhag/PanFlow.