Orbia: Do Generated Videos Form a Consistent 3D World?

AI summary

Overview

Research area: Computer vision — generative video world models and 3D scene consistency evaluation.

Technical level: Advanced. The paper assumes familiarity with video diffusion/generation models, camera-pose conditioning, depth estimation, and Gaussian-splatting reconstruction, though its central question is stated in plain terms.

Scope: Orbia is a benchmark of 720 evaluation instances (480 short cases plus 240 paired minute-long extensions) that measures whether videos produced by interactive world models and video generators depict one coherent, persistent 3D world, scored along three dimensions — input preservation, generated-content persistence, and 3D self-consistency — across 27 models and six exploration-and-revisit protocols.

What This Paper Is About

Video generators are increasingly pitched as explorable "worlds," but a video can look realistic frame by frame while contradicting itself in 3D — objects drift, walls change shape, and newly invented scenery is different when the camera comes back. Existing benchmarks test only pieces of this problem (local geometry, perceptual quality, or vision-language judgments of "memory"), so there is no systematic, quantitative test of 3D consistency across viewpoints and over long exploration. Orbia addresses this by building controlled exploration-and-revisit trajectories with registered RGB-D references, annotated landmarks, and revisits to model-generated regions, then scoring 27 text-, action-, and camera-conditioned models on whether their outputs behave like a single persistent 3D world.

Key Contributions

1. A dedicated 3D-consistency benchmark. The authors introduce Orbia, described as the first benchmark dedicated to systematic, quantitative evaluation of 3D consistency in interactive world models across viewpoints and exploration horizons. Table 1 compares it against twelve representative benchmarks (WorldScore, WorldOlympiad, WBench, WorldMark, iWorld-Bench, MIND, WRBench, WorldRoamBench, LoopBench, R2M-Bench, CycleBench, MemoBench) and shows Orbia as the only entry covering all five consistency tests: local 3D self-consistency, global 3D self-consistency, input-object preservation, revisit consistency, and long-term drift.

2. A three-part decomposition of consistency. Rather than a single "consistency" score, the benchmark separates input preservation (what the input image constrains), generated-content persistence (what the model freely invents and must then keep), and 3D self-consistency (whether generated views agree with each other in 3D), each with its own reference and metrics.

3. A unified construction and annotation pipeline. Combining public datasets, self-captured scenes, and Unreal Engine simulation, the pipeline produces registered RGB-D pairs, verifies anchors and cross-view surface support, synthesizes geometrically valid revisit trajectories, and applies human curation — yielding 720 instances and 859 anchor evaluation targets from over 110k source samples and over 300 hours of annotation and curation.

4. A 27-model evaluation with interface adapters. The authors evaluate 19 camera- or action-conditioned world models and eight TI2V (text/image-to-video) models through native-interface adapters that translate the same trajectories into text instructions, discrete actions, or camera poses, and relate 3D consistency to control accuracy, visual quality, and exploration duration.

Main Findings

• No model leads every dimension. SolarWM-H3 ranks first overall at 76.32, with the best generated-content persistence (70.28) and temporal stability (84.05) among world models; Lyra2 (74.77) and EVOKE (72.70) rank second and third overall. But Gen3C leads input preservation (75.23), HY-WorldPlay leads 3D self-consistency (70.34), EVOKE leads visual quality (76.48), and Lyra2 leads control (81.64).

• Explicit 3D memory is associated with strong results. Three of the top five world models — Lyra2, EVOKE, and Gen3C — use explicit geometry. Lyra2 retrieves relevant past frames using per-frame geometry and establishes correspondences with target views; EVOKE stores observed geometry in an external world state bank and renders it from the current pose; Gen3C maintains a 3D cache whose renderings guide generation. The authors suggest these mechanisms let models reuse content after it leaves the recent video context.

• Camera control is the largest gap between model families. World models average 57.29 on control versus 32.48 for TI2V models, a gap of 24.80 points. Even the best TI2V model, Seedance2.5 (43.23), falls far below Lyra2 (81.64). Control also shows the largest standard deviation across world models.

• TI2V models are competitive on visual quality but not on control. Seedance2.5 leads the TI2V group overall at 67.44 and has the group's highest input preservation (72.84), while Cosmos3-Super leads the group on visual quality (79.04), generated-content persistence (69.75), and 3D self-consistency (74.55). Kling3.0 (65.23) and Cosmos3-Super (63.72) follow Seedance2.5 overall. The pattern suggests strong image quality does not imply an executable, controllable world.

• Input preservation and generated-content persistence fail differently. The authors highlight (Figure 2) that a model can look acceptable at return-to-start while producing inconsistent content in regions it invented itself, which motivates scoring the two separately rather than averaging over the frame.

• Different consistency dimensions do not track each other. For example, Cosmos3-Nano has strong generated-content persistence (66.29) but the lowest control in the world-model group (35.70), while HY-WorldPlay has the best 3D self-consistency (70.34) but control of 45.26 — evidence that these are separable capabilities rather than one underlying skill.

Methodology in Plain English

Building controlled revisits. The team starts from real RGB-D captures — four public datasets (ScanNet++, SpatialVID, Sekai, MiraData), their own recordings (85 recorded, 60 retained), and 48 Unreal Engine environments (365 candidate videos rendered, 180 cases retained). For each case they select two registered viewpoints: an input view P0 and a query view Pq. A vision-language model (Qwen) proposes distinctive static landmarks ("anchors"); SAM3 segments them in both views; the annotations are filtered and tracked to confirm the same object appears in both; and back-projecting with depth verifies the anchor is actually visible and geometrically supported in the query view.

Synthesizing the path. A camera trajectory is fitted to pass through both reference poses and return to the start, checked against scene geometry for clearance. Six templates vary the spatial demand: forward and backward, lateral return, yaw return, arc return, orbit around an anchor, and peek and return (in which the camera moves past an anchor into unseen territory, comes back, and revisits it a second time). Revisit pairs are chosen for regions outside the input view, with the first generated observation serving as the reference — the model is free to invent, but must keep what it invented.

Scoring. Depth and camera poses are recovered from the generated video with DA3. Input preservation compares return/query views against registered RGB-D using DINO similarity, PSNR, SSIM, LPIPS, and depth AbsRel, scoring each anchor separately so that a missing small landmark is not washed out by a scene-wide average. Generated-content persistence projects first-visit depth into the revisit view to match the same physical surfaces, then compares appearance and depth, plus a bidirectional nearest-point F-score between point clouds. 3D self-consistency uses cross-view depth and feature correspondence, dispersion of depth ratios over a 4×4 grid for regional scale agreement, and a held-out rendering test in which DA3 predicts a Gaussian scene from 64 context frames and 16 withheld views are compared against the corresponding generated frames. Long-horizon behaviour is tracked in five-second windows; the final score weights short- and long-video means at 30% and 70%, and the overall score weights control (25%), visual quality (25%), the three consistency dimensions (10% each), and temporal stability (20%).

Standardization. Cases are standardized at 720p and 16 FPS; short trajectories last about 22 seconds with one revisit, and minute-long extensions add three revisits. The dataset covers 120 indoor and 360 outdoor short cases, with 300 from real-world observations and 180 from simulation.

Why This Matters

Impact on research. The benchmark turns "does this video model a world?" into a set of measurable, separately diagnosable quantities. Because input preservation, generated-content persistence, and 3D self-consistency are scored apart, the results localize failure modes — a model may follow camera instructions yet disagree in depth, or produce coherent geometry along the wrong trajectory. The finding that explicit 3D memory co-occurs with the strongest content-preservation results gives the field a concrete design signal rather than a leaderboard number.

Real-world applications:

• Interactive games and virtual environments, where a player returning to a location expects the same layout and objects.
• Generative rendering and virtual production, where surfaces revealed from new angles must be mutually compatible for compositing and shot design.
• Embodied simulation and robotics, where an agent reuses previously observed spatial relationships for planning and navigation.
• Autonomous driving and sensor-world modeling, where consistency across a driven route underpins simulation fidelity.

Industry relevance. The 24.80-point control gap between world models and TI2V models indicates that scaling visual quality alone does not yield an explorable world; teams building interactive products need to invest in pose/action controllability and persistent spatial memory. The benchmark, adapters, and protocol also give a common yardstick for comparing proprietary and open models on the same trajectories.

Future Directions

• Closing the control gap. The largest measured deficit is accurate execution of requested camera motion, especially for TI2V models (best score 43.23 versus 81.64 for the best world model). Whether control can be added post hoc to strong generators is an open question.
• Scaling explicit 3D memory. The correlation between geometry-backed memory (Lyra2, EVOKE, Gen3C) and strong preservation results invites work on cheaper, longer-lived, or learned world-state stores.
• Pushing past minute-long horizons. Orbia's longest sequences run to one minute with three revisits; drift beyond that, and under many more returns, is not covered.
• Reducing dependence on estimated geometry. Scoring relies on DA3 for depth, poses, and Gaussian reconstruction, so estimator error and static-scene assumptions bound what can be measured — dynamic content, and any failure that the estimator itself smooths over, remain weakly tested.

Target Audience

Researchers and engineers working on video generation, interactive world models, and 3D/4D scene modeling; evaluation and benchmarking teams at labs and studios building generative environments; and robotics or autonomous-driving practitioners who need to know whether a generative simulator can be trusted to stay spatially consistent across a rollout. Readers without a background in 3D reconstruction will find the problem framing accessible, but the metric definitions and results tables assume graduate-level familiarity with the area.

Authors’ abstract

Video generators serve as interactive worlds for exploration and simulation, yet realistic and visually appealing frames do not necessarily depict a consistent 3D world. Existing benchmarks assess selected geometric properties or use vision-language models to judge memory and consistency, leaving systematic 3D evaluation underexplored. We introduce Orbia, the first benchmark dedicated to a systematic and comprehensive evaluation of 3D consistency in interactive world models. Through controlled exploration and revisit, we organize the evaluation into three dimensions: input preservation, generated-content persistence, and 3D self-consistency. For input preservation, registered RGB-D references and annotated anchors support appearance, geometry, and spatial stability measurements. For generated-content persistence, we compare revisits to regions absent from the input image with their first generated observations to test whether models preserve what they create. For 3D self-consistency, cross-view depth and feature correspondence, regional scale agreement, and held-out Gaussian rendering test whether generated views describe the same 3D scene. Minute-long exploration and revisits further test long-term consistency. Our construction and annotation pipeline brings together public datasets, self-captured scenes, and simulation, yielding 720 instances across indoor and outdoor environments and six exploration-and-revisit protocols. Evaluation spans 27 text-, action-, and camera-conditioned models through native-interface adapters. Our Orbia benchmark provides a systematic basis for diagnosing 3D inconsistency and developing more persistent video world models.