Wanderland: Geometrically Grounded Simulation for Open-World Embodied AI

AI summary

Overview

Research area: Computer vision and Embodied AI — specifically real-to-sim simulation, 3D reconstruction, novel view synthesis, and visual navigation benchmarking.

Technical level: Advanced. The paper assumes familiarity with SLAM, 3D Gaussian Splatting (3DGS), neural rendering, and reinforcement-learning-based navigation.

Scope: The paper introduces Wanderland, a real-to-sim framework and dataset that uses handheld multi-sensor capture and LiDAR-inertial-visual (LIV) SLAM to build geometrically grounded, photorealistic simulation environments for open-world embodied navigation.

What This Paper Is About

Closed-loop evaluation in embodied AI (such as visual navigation) is hard to reproduce because there is no trusted open-world simulator. Recent video-3DGS methods can build visually convincing environments from casual videos, but the authors argue these suffer from inaccurate 3D reconstruction, non-metric scale, unreliable collision geometry, and degraded rendering for viewpoints off the capture path. Wanderland addresses this by capturing real urban scenes with a handheld multi-sensor scanner, reconstructing them with LIV-SLAM, and integrating the resulting mesh and 3DGS model into Isaac Sim so that navigation policies can be trained and evaluated on geometry that is metrically accurate and collision-consistent.

Key Contributions

1. The authors identify fundamental limitations of video-3DGS pipelines for embodied AI — inaccurate reconstruction, unreliable geometry, and extrapolated view degradation — and introduce a real-to-sim framework that overcomes them.
2. They introduce the Wanderland framework and dataset (530 scenes, over 420,000 frames, more than 100 hours of recording, covering more than 3.8 million square meters), designed for systematic analysis of key factors in simulation construction and as a testbed for embodied navigation.
3. They show that vision-only reconstructions remain significantly less accurate than LiDAR-enhanced ones, producing unreliable simulation for embodied AI.
4. They release rich raw sensor data (synchronized RGB fisheye images with intrinsic calibrations, globally consistent camera poses, colorized metric point clouds, optimized 3DGS models, extracted collision meshes, and USD scenes) to benchmark 3D reconstruction and novel view synthesis methods.

Main Findings

• Vision-only pose estimation falls far short of LIV-SLAM. On scenes spanning less than 100 meters, even the "best of all" vision-only result reaches only meter-level metric accuracy. After ground-truth scale alignment, the best-of-all average error is still 30 cm and 5 degrees. Individual methods are far worse: the best single-method scaled T-ATE medians in Table 2 come from COLMAP with ground-truth calibration (4.8 m mean / 0.30 m median) and DA3 (6.0 / 2.3), versus the best-of-all 2.8 m mean and 0.30 m median.

• Vision-only input prevents accurate mesh extraction. Meshes from Vid2Sim and GaussGym show significant noise, fragmentation, and incompleteness because of inaccuracies in their underlying neural representations, undermining collision reliability. Wanderland instead extracts meshes from the globally consistent LiDAR point cloud.

• Geometric grounding improves novel view synthesis. On the same train/validation split, Wanderland achieves 20.37 PSNR / 0.688 SSIM / 0.327 LPIPS on interpolated views and 17.92 / 0.591 / 0.445 on extrapolated views. For comparison, GaussGym reaches 12.17 / 0.440 / 0.738 and 12.63 / 0.436 / 0.725; Vid2Sim reaches 17.20 / 0.549 / 0.399 and 16.49 / 0.573 / 0.371; COLMAP-reconstruction baselines (3DGS, 2DGS, 3DGUT) range from 16.81 to 18.29 PSNR across the two view types.

• Rendering quality affects downstream perception. In a semantic consistency check using DINOv3 and Grounded SAM 2, GaussGym's fragmented renderings prevent reliable segmentation, Vid2Sim's DINOv3 features diverge significantly from ground truth, and Wanderland's renderings maintain structural integrity and semantic consistency.

• RL training is sensitive to simulation fidelity. Models generally deteriorate after RL training in Vid2Sim-built environments. For example, CityWalker shows 26.40 m NE (+24%), 0.17 SR (-21%), 0.17 SPL (-19%) after training in Vid2Sim, but improves to 19.02 m NE (-10%), 0.24 SR (+14%), 0.23 SPL (+14%) in the Wanderland environment. NoMaD trained in Wanderland reaches 17.99 m NE (-2%) and 0.26 SR (+8%); MBRA reaches 23.76 m NE (-4%) and 0.23 SR (+5%).

• Evaluation is unreliable without grounded geometry. When evaluated in Vid2Sim-built environments, all models show a much lower success rate and higher intervention rate than in Wanderland environments.

• A large open-world navigation gap remains. In the benchmark over the full dataset, no model reaches a success rate over 50%. VLN models generally outperform point-goal and image-goal models, and outdoor navigation is harder than indoor: NaVILA scores 5.13 m NE, 0.47 SR, 0.47 SPL, 0.41 IR indoors versus 13.2 m NE, 0.31 SR, 0.31 SPL, 0.68 IR outdoors.

• Depth regularization beats freezing Gaussian centers. Using depth loss gives 20.37 / 0.688 / 0.327 on interpolated views and 17.92 / 0.591 / 0.445 on extrapolated views; freezing Gaussian centers gives 20.39 / 0.703 / 0.327 and 17.10 / 0.558 / 0.456.

Methodology in Plain English

The team captures real urban spaces with the SkylandX MetaCam Air, a compact handheld 3D scanner that combines a Livox Mid-360 non-repetitive LiDAR with a built-in IMU, an RTK-GNSS antenna, and two synchronized 4K fisheye cameras with over 180° field of view. The LiDAR is mounted at a tuned inclination to capture ground-level detail and maximize overlap with the cameras. Scene sizes are chosen to be 5,000–10,000 square meters, and RGB frames are triggered by movement distance or rotation angle rather than a fixed frame rate, so viewpoints are spread more uniformly. Data was collected in New York City and Jersey City across residential buildings, business districts, public streets, plazas, and university campuses, at different times of day and under varying weather.

Raw data is processed by MetaCam Studio, which implements a LiDAR-inertial-visual-GNSS fusion pipeline producing dense, metric-scale point clouds and globally consistent camera trajectories. Images are masked in two stages — Egoblur for faces and license plates, then an object detector for people, animals, and vehicles — and raw fisheye images are cropped to 120° and undistorted into perspective views for 3DGS training.

3D Gaussians are initialized from the dense colorized point cloud (raw spacing 5–10 mm, roughly 10–50 million points per scene, uniformly downsampled to around 5 million points per scene with one Gaussian per point) using the gsplat framework at 800×800 resolution for 15,000 training steps. Instead of monocular depth as pseudo ground truth, the authors project the initialized Gaussians into each camera pose to produce ground-truth depth for a depth loss, combined with a photometric loss. They also use Difix3D+ to augment training views with cleaner novel views, gradually expanding the sampled viewpoints away from the training trajectories over training steps.

For simulation, the collision mesh is extracted by voxelizing the global point cloud into an occupancy grid and applying marching cubes, then removing geometry far from the collection trajectory and filtering small fragments. Because the mesh and the 3DGS model share the same coordinate system, both are combined in a Universal Scene Description (USD) scene — the mesh as the physics/collision layer, the 3DGS model as the renderer — and loaded directly into Isaac Sim.

Navigation tasks come from the mesh geometry: it is imported into Unity, the NavMesh baking API extracts a triangulated navigable surface, and a pathfinding module generates collision-free expert trajectories for point-goal and image-goal tasks. For vision-language navigation, each trajectory is replayed into an egocentric video, a VLM generates natural language instructions from that video, and humans verify them.

Why This Matters

Impact on research. The paper argues that geometric grounding is not optional for the next generation of embodied systems. It provides metric-scale camera poses, dense LiDAR point clouds, and carefully designed extrapolated views as ground truth for foundational vision geometry models and as a benchmark for novel view synthesis under realistic off-trajectory conditions. It also addresses the scarcity of large-scale metric benchmarks for outdoor 3D vision.

Real-world applications:

• Last-mile delivery robots operating on sidewalks and streets.
• Campus-scale wayfinding systems for pedestrians and service robots.
• Service robots traversing lobbies, hallways, plazas, and sidewalks.
• Long-range depth estimation and geometric learning research that requires metric ground truth outdoors.

Industry relevance. The work targets a standardized, reproducible testbed for evaluating navigation policies, which matters to anyone deploying embodied systems outside controlled indoor settings. It also benchmarks core 3D vision components (reconstruction and view synthesis), which are shared infrastructure across robotics, simulation, and content creation pipelines built on 3D Gaussian Splatting.

Future Directions

• Higher capture frame rate. The current capture system operates at 1 FPS due to hardware constraints, producing sparser viewpoint sampling than ideal and limiting rendering quality in highly complex scenes; the authors plan hardware upgrades.
• Modeling urban dynamics. The current work focuses on geometric reconstruction and static environment simulation, leaving moving pedestrians, vehicles, and traffic patterns for future work, which would require behavior prediction and interactive simulation.
• Dataset expansion. The authors state an active development roadmap targeting over 1,000 scenes to further enhance diversity and scale.
• Closing the reconstruction gap. The paper shows vision-only methods remain far less accurate than LIV-SLAM; whether recent foundation models can close this gap is left open.

Target Audience

Researchers and engineers working on embodied AI, visual navigation, vision-language navigation, and reinforcement learning for robotics; 3D vision researchers focused on SLAM, 3D reconstruction, Gaussian Splatting, and novel view synthesis; and practitioners building simulation environments for evaluating robots that must operate in open-world indoor-outdoor settings.

Authors’ abstract

Reproducible closed-loop evaluation remains a major bottleneck in Embodied AI such as visual navigation. A promising path forward is high-fidelity simulation that combines photorealistic sensor rendering with geometrically grounded interaction in complex, open-world urban environments. Although recent video-3DGS methods ease open-world scene capturing, they are still unsuitable for benchmarking due to large visual and geometric sim-to-real gaps. To address these challenges, we introduce Wanderland, a real-to-sim framework that features multi-sensor capture, reliable reconstruction, accurate geometry, and robust view synthesis. Using this pipeline, we curate a diverse dataset of indoor-outdoor urban scenes and systematically demonstrate how image-only pipelines scale poorly, how geometry quality impacts novel view synthesis, and how all of these adversely affect navigation policy learning and evaluation reliability. Beyond serving as a trusted testbed for embodied navigation, Wanderland's rich raw sensor data further allows benchmarking of 3D reconstruction and novel view synthesis models. Our work establishes a new foundation for reproducible research in open-world embodied AI. Project website is at https://ai4ce.github.io/wanderland/.