OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes

AI summary

Overview

Research area: Computer vision and generative AI for 3D scene construction, specifically panoramic (360°) generation and perception and the lifting of 2D generative priors into graphics-ready 3D assets.

Technical level: Advanced. The paper assumes familiarity with flow matching, DiT architectures, LoRA adapters, rotary position embeddings (RoPE), equirectangular panorama projections, inverse rendering, and physically based rendering (PBR).

Scope in one sentence: OmniX is a single framework built on a pre-trained 2D flow matching model that performs panorama generation, panorama perception of geometry and PBR materials, and panorama completion, and that pipeline is used to build PBR-ready 3D scenes from images.

What This Paper Is About

Automatic 3D scene construction is currently dominated by procedural generation (assembling scenes from asset libraries) and 2D lifting (using 2D generative models to produce 3D content). Panorama-based 2D lifting is the more promising of the two because it uses strong 2D generative priors to produce immersive and diverse environments, but existing methods focus on appearance and fall back on off-the-shelf depth estimators for geometry, ignoring textures and PBR materials. The goal of this paper is to repurpose a 2D generative model so that it also perceives intrinsic scene properties — geometry, albedo, roughness, metallic, normals — and thereby produce 3D scenes that can be relit and simulated in standard graphics pipelines.

Key Contributions

1. OmniX, a unified framework. A single system that repurposes pre-trained 2D flow matching models for panorama generation, perception (RGB to X), and completion (masked X to X), with a generalized formulation that takes multiple conditioning inputs and produces multiple target outputs.
2. Circular synchronization. A training-free mechanism that enforces circular translation equivariance in the model's spatial operators — circular padding for convolutions and token padding plus attention masking for RoPE-based attention — to remove the seam discontinuities that plague equirectangular panorama generation.
3. Modality-specific adapters. A cross-modal adapter design that assigns separate LoRAs to different input modalities, which the authors report outperforms shared-branch and shared-adapter alternatives and can be extended to new modalities with minimal disruption to the model's weight distribution.
4. PanoX, a synthetic panorama dataset. A dataset rendered with Unreal Engine 5 covering eight large-scale 3D environments (five indoor, three outdoor), containing more than 10,000 instances and 60,000 panoramic images with pixel-aligned distance, world normal, albedo, roughness, and metallic annotations, plus text descriptions extracted by Florence 2. The authors state it is the first panorama dataset covering both indoor and outdoor scenes with dense geometry and material annotations.

Main Findings

• Panorama generation leads on several metrics. On Laval Indoor, OmniX reaches FID 7.4, CLIP-FID 2.5, FAED 5.2, and CLIP Score 28.47, with KID 0.41 (×10²). On SUN360 it reaches KID 0.66, CLIP-FID 6.5, and CLIP Score 27.54, with FID 39.4 and FAED 8.7. The authors describe the KID result on Laval Indoor and the FID/FAED results on SUN360 as competitive rather than best; the strongest baseline, CubeDiff, records KID 0.32 on Laval Indoor and FID 25.5 with FAED 7.6 on SUN360.
• Strong intrinsic decomposition results. Against RGB↔X, MGNet, IDArb, IID, and DiffusionRenderer, OmniX reports on PanoX-OutDomain an albedo PSNR of 17.76 with LPIPS 0.344, roughness PSNR 16.21 with LPIPS 0.398, and metallic PSNR 18.87 with LPIPS 0.254. On Structured3D, albedo reaches PSNR 20.35 and LPIPS 0.174.
• Best normal estimation, second-best depth. On geometry, evaluated on PanoX-OutDomain, OmniX records distance AbsRel 0.158, δ-1.25 0.787, MAE 1.68, RMSE 6.83, and normal mean angular error 27.14 with median 14.88, with 0.155 percent of pixels below 5° and 0.663 percent below 30°. The depth-only method MoGe achieves the lower AbsRel (0.106) and RMSE (5.35), but the paper notes MoGe uses vastly more depth annotations (9.0M versus 0.087M) and requires multi-view inference and stitching for panoramas.
• Adapters and pre-trained priors both matter. In ablation, Separate-Adapter (the OmniX design) reaches albedo PSNR 21.68 / LPIPS 0.260, roughness PSNR 18.16 / LPIPS 0.329, and distance δ-1.25 0.808 / AbsRel 0.154 / RMSE 4.755 / MAE 1.110. Training the same design from scratch drops to albedo PSNR 17.74 and roughness PSNR 14.83, confirming the value of initializing from 2D generative pre-trained weights.
• Completion beats a strong inpainting baseline. OmniX-Fill on PanoX-OutDomain RGB reports FID 16.14, KID 0.211, CLIP-FID 2.41, and FAED 1.49, versus Flux-Fill at FID 30.09, KID 0.975, CLIP-FID 5.74, and FAED 2.53.
• Seamless boundaries without fine-tuning. Circular synchronization is described as producing near-seamless transitions across the horizontal wrap-around, while the Flux.1-dev baseline shows clear discontinuities and circular blending and latent rotation only partially alleviate artifacts.
• Graphics workflows verified in Blender. Generated scenes support free exploration, PBR-based relighting with an animated point light, and physical simulation with an elastic ball given an initial horizontal velocity.
• Acknowledged weaknesses. The authors report that Euclidean distance prediction is not accurate enough, producing bumpy reconstructed 3D surfaces that affect PBR rendering, and that the metallic prediction model (OmniX-Pano2Metallic) generalizes poorly, partly due to scarce panoramic PBR material training data.

Methodology in Plain English

The starting point is an existing image generator (Flux.1-dev) that already knows how to produce realistic imagery. Rather than train a new model from scratch, the authors wrap it with small adapter modules — multiple LoRAs, one per input modality — so the same backbone can take in different kinds of conditioning inputs (an RGB image, a camera ray map, a masked target map) and output different kinds of panoramas. The objective is the standard flow matching loss, extended to a multi-input, multi-output version, and only the target output contributes to the loss even though conditioning inputs are fed into the DiT.

To make outputs wrap around seamlessly at the left-right edge of an equirectangular panorama, the authors do not modify trained weights or blend latent features. Instead they change the spatial operators themselves: convolutions use circular padding, and attention layers with rotary position embeddings pad the key and value tokens circularly and mask attention so each query sees a circular, spatially uniform window. This is a structural constraint applied at inference, so it costs nothing during training.

To teach the model about intrinsic scene properties, the authors built PanoX, rendering eight Unreal Engine 5 environments into panoramas with pixel-aligned distance, normal, albedo, roughness, and metallic maps and Florence 2 text captions. They split the first six scenes 8:1:1 into train, validation, and test, and held the remaining two scenes out as PanoX-OutDomain for generalization testing. Training combined PanoX with Pano360, Structured3D, Poly-Haven, and Humus for generation, and with Structured3D for perception; roughness and metallic used PanoX only because Structured3D lacks PBR materials.

For 3D scene building, the pipeline runs in three stages: generate a panorama and then its intrinsic maps, reconstruct a mesh by projecting each distance-map pixel into 3D space as a vertex and connecting vertices based on pixel neighbors and relative distances (with albedo, normal, roughness, and metallic assigned to faces via spherical UV unwrapping), and then iteratively complete the scene. For completion, masks are built by sampling a random 3D displacement and using ray intersection against the distance map to find occluded regions, and the adapters are fine-tuned with mask inputs as OmniX-Fill. Training used PyTorch on four NVIDIA L40S GPUs, AdamW at a learning rate of 1e-4 with no decay, 512 × 1024 panoramas, batch size 1 per GPU, and 12 adapter models.

Why This Matters

Impact on research. The paper reframes inverse rendering as a generative modeling task within the same framework that generates panoramas, rather than treating perception as a separate bolted-on problem. It also argues that boundary consistency in panoramas should be a structural property of the operators rather than an external constraint applied through overlapping fields of view, which is a different angle from multi-view aggregation approaches. PanoX addresses a stated scarcity of panoramic data with dense geometry and material annotations.

Real-world applications:

• Virtual production and gaming, where explorable, relightable indoor and outdoor environments can be produced from a single image.
• Robotics and embodied agents, since the introduction motivates near-zero-cost simulation learning for intelligent agents or robots.
• Architecture, urban planning, and digital twins, domains the paper cites as users of procedural generation, and which benefit from PBR-ready assets that plug into standard 3D tools.
• Graphics content pipelines, where the output can be imported into Blender for relighting, novel-view rendering, and physical simulation.

Industry relevance. The explicit goal of producing outputs usable in modern graphics pipelines — with albedo, roughness, and metallic maps rather than only color and depth — targets a real gap between generative 3D research and the PBR workflows used in production. The involvement of authors affiliated with Kuaishou Technology and Tencent is consistent with that applied framing.

Future Directions

• Improving distance prediction accuracy, which the authors identify as the cause of bumpy reconstructed surfaces and degraded PBR rendering.
• Improving PBR material estimation and generalization, particularly metallic prediction, which the paper attributes in part to scarce panoramic PBR material training data and to the gap between neural rendering and PBR rendering.
• Efficiency, since OmniX inherits the slow training and inference of the underlying pre-trained 2D flow matching models.
• Scaling data and modality coverage, including expanding the panorama perception benchmark and adding new input modalities, which the modality-specific adapter design is claimed to support flexibly.

Target Audience

Researchers and graduate students in computer vision and generative modeling working on 3D scene generation, panoramic vision, or inverse rendering; graphics engineers who need PBR-ready assets from images or panoramas; and practitioners in gaming, virtual production, simulation, and digital-twin development who want to understand how generative models can feed standard rendering pipelines. Readers need a solid background in diffusion or flow matching models and 3D representations to follow the technical sections.

Authors’ abstract

There are two prevalent ways for automatic 3D scene construction: procedural generation and 2D lifting. Among these, panorama-based 2D lifting has emerged as a promising technique, leveraging powerful 2D generative priors to produce immersive, realistic, and diverse 3D environments. In this work, we advance this technique to generate graphics-ready 3D scenes suitable for physically based rendering (PBR), relighting, and simulation. Our key insight is to repurpose 2D generative models for panorama perception of geometry, textures, and PBR materials. Unlike existing 2D lifting approaches that emphasize appearance generation and neglect the perception of intrinsic properties, we present OmniX, a versatile and unified framework for panorama generation, perception, and completion. Built upon cross-modal adapter structure and cyclic spatial operators, OmniX effectively repurposes pre-trained 2D flow matching priors for joint modeling of multimodal, seamless equirectangular representations. Furthermore, we construct a large-scale synthetic panorama dataset comprising high-quality multimodal panoramas from diverse indoor and outdoor scenes. Extensive experiments demonstrate the effectiveness and generality of OmniX as a unified framework for panorama generation and perception across geometry, lighting, and semantics, enabling graphics-ready 3D scene generation and opening new possibilities for immersive and physically realistic virtual world creation.