UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos

AI summary

Overview

Research area: computer vision and embodied AI, specifically real-to-sim urban scene generation for robot navigation.

Technical level: Advanced. The paper combines large-scale 3D asset curation, open-vocabulary video parsing, structure-from-motion, and reinforcement learning for robot policies.

Scope: UrbanVerse is a data-driven system that converts crowd-sourced city-tour videos into physics-aware, interactive urban simulation scenes in IsaacSim, and it is evaluated for scene fidelity, policy scaling, and zero-shot sim-to-real transfer on real robots.

What This Paper Is About

Training urban embodied AI agents (delivery robots, quadrupeds) requires diverse, high-fidelity urban environments, but existing options are either hand-crafted (not scalable) or procedurally generated from hard-coded rules (not realistic). The paper asks whether realistic, interactive urban scenes can be built from real-world videos for scalable robot navigation.

UrbanVerse answers this by distilling layout, semantics, ground appearance, and lighting from uncalibrated city-tour videos, then materializing those layouts with matched 3D assets annotated with physics properties, producing simulation scenes that preserve real-world street distributions.

Key Contributions

1. UrbanVerse-100K: a repository of 102,444 metric-scale GLB urban objects across 659 categories, each annotated with 33 semantic, physical, and affordance attributes, plus 288 PBR ground materials (98 road, 190 sidewalk) and 306 HDRI sky maps.

2. UrbanVerse-Gen: an automatic open-vocabulary pipeline that parses uncalibrated RGB videos into a 3D urban scene graph (object, ground, and sky nodes), retrieves digital-cousin assets, and instantiates metric-scale interactive scenes in IsaacSim.

3. A scene library and benchmark: 160 simulation scenes built from 32 YouTube city-tour videos spanning 7 continents, 24 countries, and 27 cities (k_cousin = 5 digital cousin variants per layout), plus a test-only benchmark of 10 AutoBench scenes and 10 artist-designed CraftBench scenes.

4. Empirical validation for policy learning: demonstrations that training on UrbanVerse scenes produces power-law scaling with scene count and strong zero-shot sim-to-real transfer, including a 337 m real-world mission with two human interventions.

Main Findings

• Scene reconstruction fidelity: On 45 KITTI-360 sequences (average length 198.7 m), the default MASt3R plus YoWorldSAM2 setup recovers 93.1% of object categories correctly, with 1.4 m position error, 19.8 degrees orientation error, 0.8 m^3 volume error, and 28.2 mAP25. Asset retrieval accuracy from CraftBench walkthrough videos is 75.1%.

• Human preference for realism: In a study with 32 undergraduates, more than 70% preferred UrbanVerse scenes over procedurally generated UrbanSim scenes on object diversity, layout coherence, and overall realism. On a 1-5 scale, UrbanVerse scenes scored 3.58/5.00 versus 4.08/5.00 for artist-designed scenes.

• Layout and cousin scaling: Increasing unique layouts (N in {1, 8, 16, 32}, corresponding to 5, 40, 80, and 160 scenes) consistently improves success rates, while procedurally generated layouts show very limited improvement. Increasing digital cousins per layout (m in {1, 2, 3, 4, 5}, giving 32-160 scenes) further boosts success, and both follow log-log linear power-law fits with strong Pearson correlations.

• Simulation benchmark results: On CraftBench, PPO-UrbanVerse reaches 41.9% SR, 35.5 CT, and 62.4% RC, outperforming MBRA (35.6% SR), S2E (33.1%), CityWalker (29.2%), and PPO-UrbanSim (9.1%). This is a +6.3% SR gain over the second-best model, MBRA.

• Zero-shot sim-to-real transfer: Across 16 unseen real-world scenes averaging 24.6 m per route, PPO-UrbanVerse achieves 77.1% SR on the Coco wheeled robot and 89.7% SR on the Unitree Go2 quadruped, with 3.6 m and 2.5 m distance-to-goal respectively. The abstract reports +30.1% improvement in zero-shot sim-to-real transfer over prior methods, while the tables report +29.2% SR over S2E on Coco and +31.1% on Go2.

• Long-horizon deployment: The policy completes a 337 m real-world mission in public urban space with only two human interventions and no collisions, following GPS waypoints at 10 m intervals. The abstract states a 300 m mission; the results section states 337 m.

• Overfitting risk: Policies trained directly on test scenes perform poorly on altered routes (26.5% SR), underscoring the need for diverse training scenes.

Methodology in Plain English

The system has three parts.

First, asset curation. The authors start from 800K noisy 3D assets in Objaverse, filter corrupted or low-quality items down to 158K high-quality assets, then classify each into a three-level urban ontology derived from OpenStreetMap tags plus driving and scene-understanding datasets, yielding 659 leaf categories. Classification uses CLIP on thumbnails followed by manual verification; GPT-4.1 annotates semantic, affordance, and physical attributes (for example mass) from a thumbnail plus four rotated snapshots. All assets are standardized to metric scale and a consistent orientation.

Second, scene generation. Given an RGB city-tour video, every third frame is sampled and GPT-4.1 enumerates visible categories to form a candidate vocabulary. MASt3R estimates depth, intrinsics, and SE(3) poses to lift the video to metric 3D. An open-vocabulary parser (YoloWorld plus SAM 2) produces per-frame instance masks, lifted to 3D and fused across frames into persistent object nodes with category, centroid, oriented 3D box, yaw, and crops. A Mask2Former ground parser trained on Cityscapes segments road and sidewalk point clouds, and the upper half of each frame supplies sky data. Each object node is materialized by semantic matching with CLIP, geometry filtering by minimal bounding box distortion (top 1,000 retained), and appearance re-ranking with DINOv2 to keep the top k_cousin matches. Ground materials are matched by pixel-wise MSE, and HDRIs by HSV histogram matching. Finally, the graph is instantiated in UrbanSim on the IsaacSim backend: road and sidewalk planes are fitted from point clouds with sidewalks elevated by 15 cm, an HDRI serves as dome light and spherical background, and objects are placed at their centroids, aligned to heading, and adjusted to avoid penetration. Annotated physical parameters such as mass and friction are assigned and rigid-body dynamics enabled. Following UrbanSim, a GPU-accelerated ORCA-based planner populates scenes with dynamic agents.

Third, policy learning. The authors train mapless position-goal urban navigation policies with PPO, using only RGB observations and relative goal position, no global map. Goals and waypoints are sampled every 5 m from GPS projected into a local metric frame. Training loads 16 different scenes at a time, repeating each scene 4-6 times, and swaps the scene set every 100 RL episodes.

Why This Matters

Impact on research: UrbanVerse offers an alternative to both hand-crafted and procedurally generated urban simulators by grounding scenes in real-world video distributions, and it provides a reusable, physics-annotated asset library and open benchmark for embodied AI.

Real-world applications:

• Training last-mile delivery robots and quadrupeds for cluttered sidewalks, parked cars, and narrow passageways.
• Micromobility and sidewalk navigation research, including wheelchair and scooter agents.
• Closed-loop evaluation of navigation foundation models in artist-designed, safety-critical urban edge cases.
• Sim-to-real policy pretraining that reduces the need for unsafe, labor-intensive real-world data collection.

Industry relevance: The 659-category, 102,444-asset library with physics attributes, support for 20 robot types, and RL/IL training paradigms make the platform directly usable for robotics companies building urban navigation stacks without hand-authoring environments.

Future Directions

• Extending UrbanVerse beyond street-level environments to parks, campuses, and indoor-outdoor transitions, which the authors note would require additional terrain modeling and access structures.
• Making the UrbanVerse-Gen pipeline robust to challenging video conditions such as low light, fast motion, and heavy occlusion, which can introduce depth and pose drift or imperfect object placement.
• Exploring the broader urban embodied tasks the platform supports, including multi-agent interaction, mobile manipulation, and expert data collection for imitation learning.
• Investigating how appearance fidelity could be measured directly, since real-world asset replicas are unavailable and the authors instead used exact-retrieval proportion on CraftBench videos as a proxy.

Target Audience

Robotics and embodied AI researchers working on navigation, sim-to-real transfer, and reinforcement learning; computer vision researchers interested in open-vocabulary 3D scene parsing and real-to-sim generation; and industry practitioners building urban delivery, micromobility, or quadruped robots who need scalable, physically interactive simulation environments. Readers should have some familiarity with simulation platforms, PPO-style RL, and 3D scene representations.

Authors’ abstract

Urban embodied AI agents, ranging from delivery robots to quadrupeds, are increasingly populating our cities, navigating chaotic streets to provide last-mile connectivity. Training such agents requires diverse, high-fidelity urban environments to scale, yet existing human-crafted or procedurally generated simulation scenes either lack scalability or fail to capture real-world complexity. We introduce UrbanVerse, a data-driven real-to-sim system that converts crowd-sourced city-tour videos into physics-aware, interactive simulation scenes. UrbanVerse consists of: (i) UrbanVerse-100K, a repository of 100k+ annotated urban 3D assets with semantic and physical attributes, and (ii) UrbanVerse-Gen, an automatic pipeline that extracts scene layouts from video and instantiates metric-scale 3D simulations using retrieved assets. Running in IsaacSim, UrbanVerse offers 160 high-quality constructed scenes from 24 countries, along with a curated benchmark of 10 artist-designed test scenes. Experiments show that UrbanVerse scenes preserve real-world semantics and layouts, achieving human-evaluated realism comparable to manually crafted scenes. In urban navigation, policies trained in UrbanVerse exhibit scaling power laws and strong generalization, improving success by +6.3% in simulation and +30.1% in zero-shot sim-to-real transfer comparing to prior methods, accomplishing a 300 m real-world mission with only two interventions.