PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation

AI summary

Overview

Research area: Computer vision — panoramic (360°) video depth estimation, built on video diffusion models and synthetic data generation.

Technical level: Advanced. The paper assumes familiarity with latent video diffusion models, denoising U-Net backbones, equirectangular projection (ERP), cubemap projection, and standard depth metrics (AbsRel, δ₁).

Scope: The paper introduces a large-scale synthetic panoramic RGB-D video dataset (PanoCARLA) and an end-to-end model (PVDepth) that adapts a pretrained perspective-domain video depth model to panoramic video via two geometry-aware modifications.

What This Paper Is About

Estimating temporally consistent depth from 360° video is hard for two reasons: there is almost no large-scale panoramic RGB-D video data to train on, and the equirectangular format used to store panoramas badly distorts geometry near the poles while making smooth motion look erratic there. The authors build a synthetic dataset to break the data bottleneck, then adapt an existing perspective video depth diffusion model (DepthCrafter) so that it copes with these panoramic distortions instead of just being fine-tuned on them.

Key Contributions

1. PanoCARLA dataset. A large-scale synthetic RGB-D panoramic video dataset captured in the CARLA simulator, containing 117 long video clips and 126.7K frames covering over 50 km of roaming distance across Towns 01–07 and Town 10, with natural vehicle trajectories plus drone-like roaming at altitudes varied from 2m to 20m. The authors state it is the first panoramic RGB-D dataset in their comparison table to include the video dimension.

2. A two-stage, decoupled data acquisition pipeline. Trajectories are recorded by driving an autonomous vehicle so motion follows the simulator's navigation policies, then replayed for a "ghost rig" of six synchronized RGB-D cameras detached from the vehicle to avoid self-occlusion, with six-view collision detection to discard clipping artifacts. Six-view cubemaps are stitched into ERP with 120° camera fields of view and feather blending for RGB, and planar Z-depth is converted to Euclidean distance before stitching for depth.

3. Progressive Sphere-aware Noise Initialization (PSNI). Instead of initializing the diffusion process with standard i.i.d. Gaussian noise, the model samples noise in 3D and projects it onto the ERP, then blends it with planar noise via a variance-preserving schedule (w_s = sqrt(1 − w_n²)) where the planar weight decays linearly from 1 to a small residual, reported as w_final = 0.1.

4. Cube-rectified Temporal Modeling (CRTM). An auxiliary branch projects ERP features into cubemap faces, applies temporal layers (weight-sharing with the main branch) where motion is near-linear, projects back to ERP, and fuses the correction through a zero-initialized projection added to the primary branch's output.

Main Findings

• Main benchmark gains. On the PanoCARLA cross-scene test set (197 clips, Towns 02 and 10 held out), PVDepth reports AbsRel / δ₁ of 0.211 / 0.789 at 2 fps (50 frames), 0.205 / 0.787 at 10 fps (90 frames), and 0.181 / 0.818 at 20 fps (110 frames). The fine-tuned DepthCrafter baseline reports 0.228 / 0.752, 0.252 / 0.729, and 0.216 / 0.781 on the same three settings.

• Fine-tuning matters more than zero-shot transfer. Zero-shot DepthCrafter reports 0.340 / 0.498, 0.287 / 0.563, and 0.244 / 0.640 on the three settings; a "Cube-based" variant of DepthCrafter is much worse (1.480 / 0.257, 0.712 / 0.337, 0.630 / 0.390), which the authors attribute to destroying global context. Fine-tuning on PanoCARLA produces a substantial jump.

• Post-processing and image-based methods lag. Matrix-3D reports AbsRel 0.601 / 0.512 / 0.369 across the three settings (limited to static scenes by its reprojection assumption); ViPE reports 0.684 / 0.679 / 0.648. Image-based methods such as DA-2 report frame-level values (in gray in the paper) of 0.781, 0.759, 0.731 AbsRel, but video-level AbsRel of 1.367, 1.125, 1.021.

• Both modules contribute, and they are complementary. Averaged over the three settings, the baseline scores 0.248 AbsRel / 0.757 δ₁ (0.232 / 0.754 when trained for 40k steps to match the full model's budget). PSNI alone gives 0.228 / 0.782, CRTM alone 0.208 / 0.779, and the full model 0.199 / 0.798. Simply training the baseline longer yields only marginal gains or degradation.

• Annealing the noise schedule is necessary. Directly using sphere-projected noise improves AbsRel from 0.248 to 0.235, adding the progressive annealing reaches 0.228, and combining annealing, a residual of normal noise, and CRTM reaches the best 0.199 / 0.798 (versus 0.203 / 0.795 without the residual).

• Naive fusion of the cube branch degrades performance. Compared with the ERP-only baseline (0.248 / 0.757), naive averaging gives 0.267 / 0.704 and naive addition gives 0.356 / 0.649, while the learned projection-plus-add fusion gives 0.214 / 0.772; conditioning attention keys/values on cube features rather than ERP features improves it further to 0.208 / 0.779.

• Two-stage training with frozen spatial layers is best. One-stage joint training reaches 0.221 / 0.763; two-stage with trainable spatial layers 0.215 / 0.769; two-stage frozen with normal noise 0.204 / 0.775; two-stage frozen with PSNI 0.199 / 0.798.

• Resolution sensitivity is mild. Trained at 640×320, PVDepth improves from AbsRel 0.455 (δ₁ 0.738) at 384×192 to 0.199 (0.798) at 1024×512, plateauing around 768×384 (0.212) and 896×448 (0.207) in between.

• Cost is higher than the baseline. At 1024×512 on 110-frame clips, PVDepth has latency 0.482 s/frame, 30.7 G GPU memory, and 2.553 PFLOPs, versus DepthCrafter's 0.355 s/frame, 30.6 G, and 1.986 PFLOPs. UniK3D is far cheaper (0.05 s/frame, 3.84 G, 0.317 PFLOPs) but much less accurate (AbsRel 0.592 vs 0.199).

• Qualitative generalization on real footage. On Web360 open-world panoramic videos (no depth annotations), the authors show temporal profiles where per-frame methods such as DA-2 and UniK3D produce zigzag flicker and PVDepth produces smooth transitions.

Methodology in Plain English

The starting point is DepthCrafter, a video diffusion model that generates depth by denoising a noisy depth latent conditioned on a video latent, so the work is mostly about adapting that pretrained model rather than training from scratch.

Two adaptations do the heavy lifting. The first changes what "noise" means. Diffusion models begin from random Gaussian noise, which assumes every pixel carries roughly the same amount of information — false for a panorama, where the poles are stretched across many pixels but contain little detail. The authors instead generate noise on a sphere and unfold it into the equirectangular image, so its information density matches the projection. Because swapping distributions abruptly would destabilize a pretrained model, they mix the spherical noise with the original planar noise using weights that keep total variance at 1, and gradually reduce the planar share during training — a curriculum from flat to spherical. A small residual of planar noise (w_final = 0.1) is kept because it works better than removing it entirely.

The second adaptation handles motion. Standard video models attend only along time, which assumes objects move roughly in straight lines in pixel space. Near the poles of a panorama, a straight-line motion in the world traces a strongly curved pixel path, so this assumption breaks. Since cubemap faces are distortion-free, motion there looks near-linear again, so the model runs an extra temporal branch on cubemap-projected features, shares weights with the main branch, and maps the result back to equirectangular space through a zero-initialized projection that is added to the main branch's output. The zero initialization means the cube branch starts as a no-op and is learned in gradually.

Training proceeds in two stages: 20K steps fine-tuning the spatial layers with the new noise scheme, then 20K steps adding the cube branch while updating only the new projection layers. The model is trained at 640×320 with frame rates randomly sampled from 1 to 20, a learning rate of 5×10⁻⁶, batch size 8, and a 5-step denoising schedule with sliding-window inference for long videos; evaluation is at 1024×512 with affine-invariant global alignment (one scale and shift per clip) so that temporal consistency is actually reflected in the scores.

Why This Matters

Impact on research. The paper argues that the field's bottleneck is data, not architecture, and supplies both a dataset and an end-to-end alternative to the dominant "predict per frame, then optimize for consistency" paradigm. Because PVDepth needs no test-time optimization, it offers a faster and more stable route than iterative alignment methods, and it gives the panoramic video depth community a reproducible cross-scene benchmark split.

Real-world applications:

• Virtual reality: converting 360° footage into 3D for immersive playback and interaction.
• World models and generative simulation, which the authors name as a motivating application.
• Robot navigation, which the authors also cite as a driver of panoramic vision.
• Aerial/drone roaming reconstruction, reflected in the dataset's 2m–20m altitude variation.

Industry relevance. Depth from consumer 360° cameras is a plausible pipeline stage for VR content capture, spatial computing, and any product that turns panoramic video into geometry; the paper's emphasis on end-to-end inference without per-scene optimization speaks directly to deployment cost, even though the authors report that PVDepth is still slower than single-pass methods.

Future Directions

• Cheaper projections. The authors explicitly attribute the extra FLOPs and latency to the bi-directional ERP-to-cube projection and suggest exploring more efficient projection algorithms.
• Broader scene coverage. PanoCARLA contains only outdoor scenes, so the authors flag limited generalization to indoor settings and do not report quantitative indoor results because no annotated indoor panoramic video dataset is publicly available.
• Reflective surfaces. The paper reports qualitative failures on strong specular reflections such as water, and proposes collecting more diverse panoramic RGB-D video as a remedy.
• Beyond synthetic data. Zero-shot generalization is shown only qualitatively on Web360, with a reprojection-based geometric consistency protocol deferred to the appendix; extending supervised or weakly supervised training to real panoramic video is an open step.

Target Audience

Graduate students and research engineers working on monocular/video depth, 360° vision, or diffusion-based generative perception; practitioners building VR, spatial computing, or world-model pipelines who need temporally stable geometry from panoramic capture; and dataset builders interested in the two-stage decoupled acquisition recipe for synthetic embodied data. Readers without background in latent diffusion models and spherical projections will find the method sections dense, though the problem framing in the introduction is accessible.

Authors’ abstract

Panoramic video depth estimation is pivotal for applications such as Virtual Reality and World Models. However, advancements in this field are impeded by two primary obstacles: the scarcity of large-scale training data and the unique spatiotemporal challenges of Equirectangular Projection (ERP), which hinder the direct transfer of perspective models. In this paper, we first present PanoCARLA, a large-scale synthetic RGB-D panoramic video dataset, featuring natural motion trajectories and drone-like roaming perspectives. Building on this foundation, we propose PVDepth, an end-to-end framework adapted from perspective video depth models. To tackle ERP-specific geometric distortions and consequent non-linear temporal dynamics, we introduce two core mechanisms: (1) A Progressive Sphere-aware Noise Initialization strategy that anneals the noise distribution from planar to spherical, guiding the model to adapt to non-uniform information density; and (2) A Cube-rectified Temporal Modeling module that incorporates an auxiliary cubemap temporal branch to rectify non-linear temporal dynamics in the ERP domain. Extensive experiments demonstrate that PVDepth achieves superior performance, generating geometrically accurate and temporally consistent depth sequences. Code and data will be released at https:// github.com/ChuanxinSong/PVDepth.