GimbalDiffusion: Gravity-Aware Camera Control for Video Generation

AI summary

Overview

• Research area: Computer Vision — controllable text-to-video generation using diffusion/flow-based models, with a focus on camera pose conditioning and 3D geometry.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching video backbones, camera extrinsics and intrinsics, Plücker ray encodings, and positional-encoding schemes such as RoPE and UCPE.
• Scope: A method (GimbalDiffusion) plus a data pipeline and two evaluation benchmarks for generating videos whose camera orientation is specified in a gravity-aligned absolute world frame, including extreme pitch and roll.

What This Paper Is About

Existing text-to-video models can be steered with camera trajectories, but those trajectories are usually encoded relative to the first frame, which leaves camera orientation ambiguous and makes extreme rotations (a 180° turnaround, a barrel roll, looking straight up or down) unreliable. The authors want camera control expressed in a physically grounded, gravity-aligned absolute coordinate system, so that a user can specify pitch and roll unambiguously and the model obeys even when the prompt text conflicts with the requested view. They build a training pipeline from 360° panoramic video that covers the whole viewing sphere, plus a captioning trick to stop the prompt from overriding the camera command.

Key Contributions

1. A pipeline for generating diverse camera trajectories from 360° panoramic video, which provides exhaustive coverage of possible camera orientations rather than the forward-facing, near-zero-roll bias of natural video datasets.
2. Null-pitch conditioning, a captioning strategy that disentangles text from absolute camera angle by captioning upright forward-facing crops (pitch and roll set to 0°, yaw unchanged) rather than the actual rotated crops used for the reconstruction loss.
3. Two new evaluation benchmarks: one for video generation with extreme camera angles, built by rebalancing the pitch distribution of the SpatialVID-HQ dataset (plus artificially added roll), and one for measuring entanglement between the input prompt and camera orientation, built from PolyHaven panoramas.
4. Demonstration that the approach is architecture- and encoding-agnostic, applied to CogVideoX-2B with Plücker encoding, WAN-2.1 1.3B with Plücker encoding, WAN-2.2 5B with Plücker encoding, and WAN-2.1 1.3B with UCPE encoding.

Main Findings

• Best absolute orientation accuracy: On the SpatialVID-extreme benchmark, GimbalDiffusion on WAN 2.2 5B + Plücker achieves PitchErr 8.36 and GravityErr 10.54, the best of all reported methods, versus 39.09 / 44.13 for AC3D + cam. text. and 29.16 / 33.48 for PreciseCam+WAN-I2V-CC.
• Consistency across backbones and encodings: GimbalDiffusion on CogVideoX-2B scores PitchErr 19.67 / GravityErr 23.31, on WAN 2.1 1.3B + Plücker 10.50 / 13.05, and on WAN 2.1 1.3B + UCPE 13.95 / 16.95 — all lower than the corresponding baselines.
• Null-pitch conditioning matters a great deal: Ablating it raises pitch error by 37% in the "WAN 2.1 1.3B + UCPE encoding" variant (13.95 to 19.13) and by 72% in the "WAN 2.2 5B + Plücker camera encoder" variant (8.36 to 14.39), with similar degradation in gravity error.
• Absolute conditioning is essential: Removing absolute rotations (making camera poses relative to the first frame) causes a 3.67× increase in pitch error and a 3.36× increase in gravity error.
• Trade-off in relative rotation and prompt alignment: The method is competitive or better on RotErr (15.05 to 25.84 depending on variant) and achieves FID scores similar to baselines (110.6 to 116.0 for the WAN variants), but shows a slight drop in CLIP score (e.g. 20.6 for WAN 2.2 5B Plücker versus 22.3 for PreciseCam+WAN-I2V-CC and 22.5 for UCPE), which the authors attribute to prompt-camera entanglement.
• Qualitative superiority at extreme angles: Baselines struggle with a "backflip" trajectory (forward, straight up, backwards, down, back to start) and with a "down" starting view, generating prompt-consistent content that contradicts the requested camera orientation; the ceiling in one example is only correctly rendered by the proposed method when the camera points up.
• Entanglement benchmark results: In the PolyHaven-based benchmark of 20 scenes and 380 prompt/pitch pairs, null-pitch conditioning produces substantially lower pitch-angle error across all pitch values, particularly at ±90°, and improves CLIP similarity against ground-truth frame captions at extreme pitch values.
• Comparison to panorama-based generation: The authors note three downsides of cropping a field of view out of a generated 360° panorama: most pixels are discarded causing resolution degradation, desired prompt concepts can be cropped out, and current text-conditioned panorama methods do not provide camera translation control.
• Training data scale: Filtering the PanoVid "YouTube videos" subset by shortest-side resolution (at least 900 pixels) and by pose-estimation quality reduces the training set from 7797 to 4798 videos (38% discarded).

Methodology in Plain English

The authors start from 360° equirectangular videos. For each video they recover camera motion by cutting six perspective crops (front, back, left, right, top, bottom) and running a pose-estimation method on the resulting video, giving relative poses. To ground everything in gravity, they estimate the camera up vector at the first frame from eight perspective views sampled every 90° of yaw, average the resulting up vectors, and use that to rotate all poses into a shared gravity-aligned frame. Each sequence is initialized at null yaw, so pitch and roll are well defined while yaw stays free.

From each panorama they then sample a random camera rotation path: random initial pitch (uniform over −90° to 90°), roll (uniform over −90° to 90°), and yaw (uniform over 0° to 360°), followed by up to three additional random rotations placed at random times along the sequence, each with a random rotation axis and a random angular displacement drawn from a Beta(1.0, 5.0) distribution scaled by 720° times the rotation duration. Field of view is varied between 35° and 100° using cubic splines with randomized boundary derivatives. This acts as an infinite data augmentation: trajectories and prompts are regenerated in the dataloader each iteration.

Because the actual rotated crops can show only ground (or only sky), captioning on them teaches the model to assume prompt text determines camera angle. So they also render a version of each trajectory with pitch and roll set to 0° and caption that upright view instead — the null-pitch conditioning. A third set of videos looking straight down (−90° pitch) is captioned and this "look-down" caption is appended to the null-pitch caption 50% of the time, used at inference as a negative prompt to suppress unwanted artifacts such as tripods or selfie sticks.

The camera control is fed to the video model either as Plücker rays or through UCPE, which injects camera rays through lightweight trainable self-attention layers alongside rotary positional embeddings. Training uses 49-frame clips at 720×480 resolution, with InternVL-3-2B captioning six evenly spaced frames.

Why This Matters

• Research impact: It reframes camera control as an absolute, gravity-anchored problem rather than a relative one, and provides paired benchmarks and metrics (PitchErr, GravityErr, RotErr, TransErr, CLIP, FID, FVD) that the community previously lacked for extreme camera angles. The finding that relative camera representations fundamentally cap orientation accuracy is broadly relevant to any camera-conditioned generative model.
• Real-world applications:
  • Filmmaking and previsualization: generating plates with deliberate Dutch angles, top-down shots, barrel rolls, or full 360° pans from a text description.
  • Virtual production and visual effects: producing background footage with a specified camera orientation relative to a known horizon.
  • Architectural and real-estate visualization: specifying exact pitch and roll for interior or aerial walkthroughs.
  • Robotics and simulation: synthesizing gravity-aware training footage for agents that must reason about which way is up.
• Industry relevance: The method is demonstrated on multiple existing backbones (CogVideoX-2B, WAN-2.1 1.3B, WAN-2.2 5B), so it is presented as a training strategy that can be layered onto current commercial and open video models rather than a replacement architecture. The authors state the test datasets will be released upon publication.

Future Directions

• Extending the gravity-aligned absolute representation to full 6-DoF control including translation in metric world units, since trajectories are currently normalized for evaluation and translation accuracy varies across variants.
• Reducing the observed CLIP score penalty, i.e. closing the remaining gap between strict camera adherence and faithful rendering of prompt content at extreme angles.
• Handling unwanted objects that appear below the camera in real 360° training data more generally, beyond the "look-down" negative-prompt workaround.
• Applying the pipeline to dataset sources beyond the PanoVid "YouTube videos" subset and evaluating whether the pole-biased sampling strategy transfers to other video domains.
• Investigating whether gravity-aligned conditioning improves downstream tasks such as novel-view synthesis with dynamic scene motion, which the GEN3C baseline cannot handle at all.

Target Audience

Researchers and engineers working on controllable video generation, diffusion/flow-based generative models, and 3D-aware content creation. It is most useful to readers who already understand camera extrinsics, Plücker embeddings, and positional encoding schemes, and who want either a training recipe for absolute camera control or a benchmark for evaluating whether a model truly obeys camera commands. Practitioners in VFX and previsualization who need extreme camera moves may also find the framing useful, though the paper itself is written for a technical audience.

Authors’ abstract

Recent progress in text-to-video generation has achieved remarkable realism, yet fine-grained control over camera motion and orientation remains elusive, especially with extreme trajectories (e.g., a 180-degree turnaround, or looking directly up or down). Existing approaches typically encode camera trajectories using relative or ambiguous representations, limiting precise geometric control and offering limited support for large rotations. We introduce GimbalDiffusion, a framework that enables camera control grounded in physical-world coordinates, using gravity as a global reference. Instead of describing motion relative to previous frames, our method defines camera trajectories in an absolute coordinate system, allowing accurate, interpretable control over camera parameters. Using panoramic 360-degree videos for training, we cover the full sphere of possible viewpoints, including combinations of extreme pitch and roll that are out-of-distribution of conventional video data. To improve camera control, we introduce null-pitch conditioning, a strategy that prevents the model from overriding camera specifications in the presence of conflicting prompt content (e.g., generating grass while the camera points toward the sky). Finally, we propose new benchmarks to evaluate gravity-aware camera-controlled video generation, assessing models' ability to generate extreme camera angles and quantify their input prompt entanglement.