360Anything: Geometry-Free Lifting of Images and Videos to 360°

AI summary

Overview

Research area: Computer vision / generative modeling — specifically diffusion-transformer-based panoramic (360°) image and video generation, with connections to camera calibration and 3D scene reconstruction.

Technical level: Advanced. The paper assumes familiarity with diffusion models, flow matching, latent VAEs, diffusion transformers (DiTs), and equirectangular projection geometry.

One-sentence scope: The paper introduces 360Anything, a geometry-free DiT framework that lifts in-the-wild perspective images and videos into seamless, gravity-aligned 360° panoramas without any camera metadata, and shows that it beats prior methods that do use ground-truth camera information.

What This Paper Is About

Turning a normal photo or video into a full 360° panorama normally requires knowing the camera's field-of-view and orientation so that the input can be mathematically projected into the panorama's equirectangular (ERP) space and pixel-aligned with the output. That metadata is usually missing or noisy in real-world ("in-the-wild") media. The paper's goal is to remove that requirement entirely: treat the perspective input and the panorama target as two sequences of tokens and let a pre-trained diffusion transformer learn the geometric mapping purely from data. A second goal is to eliminate the visible "seam" where the left and right edges of a generated panorama meet, which the authors trace back to the training-time VAE encoder rather than the sampling process.

Key Contributions

1. 360Anything, a geometry-free DiT architecture for in-the-wild perspective-to-canonical-panorama generation. It implicitly infers camera intrinsics and extrinsics through sequence concatenation instead of explicit geometric projection, eliminating camera calibration from the pipeline.

2. Diagnosis and fix of panorama seam artifacts. The authors identify zero-padding in the convolution-based VAE encoder as the root cause of boundary discontinuities in the panorama latent space, and propose Circular Latent Encoding — circular padding applied before encoding, with the padded latents dropped afterward — which adds no training or inference overhead.

3. State-of-the-art results without camera metadata. 360Anything outperforms baselines on panoramic image and video generation even though those baselines have access to extra camera information, and it removes the inference-time seam tricks those baselines rely on.

4. Zero-shot camera estimation and 3D reconstruction. The model's implicit camera understanding is evaluated on FoV and orientation benchmarks, and generated panoramic videos are distilled into 3D Gaussian Splats to demonstrate geometric consistency.

Main Findings

• Image outpainting quality: On Laval Indoor, 360Anything scores FID 8.0, KID (×10²) 0.22, CLIP-FID 4.6, FAED 9.8, and CLIP-score 29.21, versus CubeDiff's 9.5, 0.32, 3.2, 18.4, and 27.02. On SUN360 it scores FID 22.4, KID (×10²) 1.27, CLIP-FID 7.3, FAED 3.8, CS 28.07, versus CubeDiff's 25.5, 1.33, 8.1, 7.6, and 25.00. It only marginally trails CubeDiff on CLIP-FID for Laval Indoor (4.6 vs. 3.2). FAED error is reduced by nearly 50% relative to the previous state of the art; FAED is the only metric computed on the whole panorama.

• Video outpainting quality: Against Imagine360, Argus, Argus*, and ViewPoint*, 360Anything leads on every metric in both camera-trajectory subsets. On real camera trajectories it reaches PSNR 25.75, LPIPS 0.0468, FVD 483.4, and VBench Imaging/Aesthetic/Motion scores of 0.5515 / 0.5427 / 0.9885. On simulated trajectories it reaches PSNR 23.64, LPIPS 0.0846, FVD 432.9, and 0.5489 / 0.5394 / 0.9891. The better PSNR/LPIPS is notable because the model does not use pixel-aligned conditioning the way the baselines do.

• Seam elimination: Measured by discontinuity score (DS), Circular Latent Encoding reduces the image score to 3.87 (versus 9.92 for vanilla and 5.29 for blended decoding) and the video score to 13.28 (versus 35.52 vanilla and 19.84 blended). Blended decoding removes the visible discontinuity but introduces gray line-like artifacts.

• Camera augmentation helps rather than hurts: Training with randomly sampled FoV, pitch, and roll improves results on all metrics, even though the standard evaluation protocol uses a fixed (90°, 0°, 0°) conditioning view. The average FID degradation across view variations is +0.98 for 360Anything, close to the +1.17 of a channel-concatenation model that uses ground-truth cameras, while a version trained without camera augmentation degrades by +6.48.

• Robustness to camera parameters: All metrics improve as input FoV increases (larger FoV means more information about the panorama). Changing pitch and roll causes degradations all below 1.0.

• Zero-shot FoV estimation: Mean/median errors (degrees) are 3.90/3.17 on NYUv2, 5.68/4.22 on ETH3D, and 5.21/4.04 on iBims-1, an average error of 4.93 degrees. This beats several supervised baselines but trails DUSt3R (listed at 4.06) and MoGe (2.91) by roughly 1–2 degrees. Over 90% of training images are indoor, creating a domain gap with the outdoor ETH3D and iBims-1 sets.

• Zero-shot camera orientation estimation: On MegaDepth it reports roll 0.87 and pitch 2.56 degrees; on LaMAR, roll 0.68 and pitch 1.23. It outperforms most supervised baselines and lags behind GeoCalib by about 0.5 degrees in both roll and pitch.

• Canonicalization matters for visual quality: Training on gravity-aligned canonical videos (at a reduced 256×512 resolution for the ablation) lowers FVD from 559.5 to 470.8 on real trajectories and from 527.0 to 449.8 on simulated ones, and raises VBench Imaging/Aesthetic/Motion scores, at the cost of lower PSNR and higher LPIPS on the conditioning region. Non-canonical training makes reconstruction of the conditioning frame easier but degrades overall fidelity.

• 3D reconstruction: Distilling generated panoramic videos into 3D Gaussian Splats produces consistent scenes that allow fly-through exploration. This is tested only on static indoor scenes because vanilla 3DGS cannot handle dynamic subjects.

Methodology in Plain English

Base models. The image model fine-tunes FLUX.1-dev, a text-to-image diffusion transformer, using the Adam optimizer at a learning rate of 5×10⁻⁵, batch size 512, and 50k steps, generating ERP images at 1024×2048. The video model fine-tunes Wan2.1-14B with Adam at 1×10⁻⁵, batch size 64, and 20k steps, producing 512×1024 videos with 81 frames. Both use flow matching, and inference runs 50 sampling steps with timestep shifting of 3.16 (images) and 3.0 (video).

Sequence concatenation instead of projection. Prior work unprojects the perspective input into ERP space, puts it in a separate channel of the input tensor, and relies on it being pixel-aligned with the target. That only works if the camera parameters are known. 360Anything instead encodes the perspective input to latents, appends them to the noisy panorama latents along the sequence dimension, and lets global self-attention in the DiT learn the spatial correspondence. No projection, no camera metadata.

Canonical coordinate target. Because no pose is supplied, the generated panorama could come out in any orientation. The model is instead trained to always output an upright, gravity-aligned panorama, which forces it to infer the input's pose and place it correctly on the canvas. Training panoramas are made consistent first: per-frame camera poses are estimated with COLMAP and frames rotated to match the first frame, then GeoCalib estimates the gravity direction and the video is rotated so gravity points along the vertical axis.

Data and augmentation. For fair comparison with CubeDiff, the image model trains on the same panorama datasets (Polyhaven, Humus, Structured3D's empty/simple/full subsets, and Pano360), with captions from Gemini 2.5 Flash; about 90% of that data is synthetic indoor renderings. Perspective conditioning crops are sampled with FoV in [30°, 120°], pitch in [−60°, 60°], and roll in [−15°, 15°], plus horizontal roll augmentation on panoramas. The video model uses Argus's training data plus camera trajectories extracted from real-world videos to increase trajectory diversity.

Circular Latent Encoding. The VAE encoder's convolutions zero-pad at image boundaries, leaving a discontinuity in the latent that shows up as a seam after generation. The fix is to crop w′ = W/8 columns from the left and right of the panorama, concatenate them on the opposite sides, encode, then discard the latents belonging to the padded regions. The token sequence length is unchanged, so this costs nothing at train or inference time.

Evaluation. Image results use Laval Indoor and SUN360 with FID, KID, CLIP-FID, FAED, and CLIP-score. Video results use the 101 testing videos from Argus with PSNR, LPIPS, FVD, and VBench Imaging Quality, Aesthetic Quality, and Motion Smoothness. Camera parameters are recovered by exhaustively searching over FoV, pitch, and roll to minimize the error between the input image and the panorama re-projected back to a perspective view.

Why This Matters

Impact on research. The paper challenges a widely held assumption in panorama generation — that explicit geometric inductive biases (unprojection, spherical convolutions, dual-branch architectures) are necessary. It argues that a plain transformer over token sequences, with sufficient data, learns the same geometry, and that U-Net-based designs have been a limiting factor for the field. It also reframes seam artifacts as a training-time representation problem rather than a sampling-time one, which is a transferable insight for any work using convolution-based VAEs on data with wrap-around boundaries.

Real-world applications:

• AR/VR and immersive media: Converting ordinary phone photos and videos into 360° content for headset viewing, without requiring calibrated capture rigs.
• Gaming and virtual world building: Turning narrow-view footage into full spherical environments that can be explored from any direction.
• 3D scene reconstruction and digital twins: The generated panoramas provide enough geometric cues to optimize 3D Gaussian Splats from a single monocular video, enabling fly-through of partially captured rooms.
• Robotics and simulation: Panoramic scene context for navigation, planning, or training simulated agents, derived from in-the-wild footage rather than specialized hardware.

Industry relevance. The work is done at Google DeepMind and builds on open-weights models (FLUX.1-dev, Wan2.1-14B), which makes the approach directly relevant to teams building content pipelines on top of existing generative foundation models. Removing the camera-estimation step simplifies production pipelines: no separate calibration tool, no accumulated error from an external estimator, and fully end-to-end optimization that benefits from scaling model and data.

Future Directions

• Closing the gap with dedicated geometry models. 360Anything trails DUSt3R and MoGe by roughly 1–2 degrees on FoV estimation and GeoCalib by about 0.5 degrees on orientation, despite those methods being trained specifically for 3D understanding. Whether scaling the outpainting objective closes this gap is an open question.

• Reducing the domain gap. Over 90% of the image training data is indoor, and the paper attributes weaker outdoor performance on ETH3D and iBims-1 to this. Broader and more diverse training data is an obvious next step.

• Handling dynamic scenes in 3D. The 3D Gaussian Splatting reconstruction is restricted to static indoor videos because vanilla 3DGS cannot handle dynamic subjects. Extending the reconstruction pipeline to moving content is left open.

• Scaling and architecture choices. The paper argues that transformer-based, prior-free learning is the right direction and that U-Net designs limit the field, but its own ablations cover only seam handling, camera augmentation, and canonicalization. How far the approach scales with model size and data volume is not reported.

• Wider use of the inferred geometry. The paper notes the estimated FoV and pose are competitive on benchmarks but only demonstrates them as a byproduct; using them explicitly for downstream vision tasks is suggested rather than developed.

Target Audience

Researchers and practitioners working on generative models for panoramic or immersive content, diffusion transformers, and latent-space representation design. It is also relevant to computer-vision engineers interested in camera calibration and 3D reconstruction from in-the-wild media, and to applied teams building AR/VR, gaming, or robotics pipelines who need panoramic generation without calibrated capture hardware. Readers without a background in diffusion models, latent VAEs, and equirectangular projection geometry will find the method sections demanding.

Authors’ abstract

Lifting perspective images and videos to 360° panoramas enables immersive 3D world generation. Existing approaches often rely on explicit geometric alignment between the perspective and the equirectangular projection (ERP) space. Yet, this requires known camera metadata, obscuring the application to in-the-wild data where such calibration is typically absent or noisy. We propose 360Anything, a geometry-free framework built upon pre-trained diffusion transformers. By treating the perspective input and the panorama target simply as token sequences, 360Anything learns the perspective-to-equirectangular mapping in a purely data-driven way, eliminating the need for camera information. Our approach achieves state-of-the-art performance on both image and video perspective-to-360° generation, outperforming prior works that use ground-truth camera information. We also trace the root cause of the seam artifacts at ERP boundaries to zero-padding in the VAE encoder, and introduce Circular Latent Encoding to facilitate seamless generation. Finally, we show competitive results in zero-shot camera FoV and orientation estimation benchmarks, demonstrating 360Anything's deep geometric understanding and broader utility in computer vision tasks. Additional results are available at https://360anything.github.io/.