SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration

AI summary

Overview

Research area: Embodied AI and robotics simulation — specifically, photorealistic urban simulators and benchmarks for robot navigation and multi-robot collaboration.

Technical level: Advanced. The paper assumes familiarity with embodied AI simulators, vision-language models (VLMs), vision-and-language navigation, and multi-agent coordination.

Scope: The paper introduces SimWorld-Robotics (SWR), an Unreal Engine 5 simulation platform for procedurally generated photorealistic cities with dynamic pedestrians and traffic, plus two benchmarks (SimWorld-MMNav and SimWorld-MRS) and a 20K-step training dataset (SimWorld-20K).

What This Paper Is About

Most embodied AI simulators and robot foundation-model research target indoor, household scenarios (tabletop tasks, home navigation). Robotics in large outdoor urban environments raises different problems: 3D perception and grounding at city scale, safe navigation around moving people and vehicles, long-range spatial memory, and multi-agent collaboration. The authors build SWR to fill that gap — a simulator that procedurally generates unlimited photorealistic urban scenes populated with pedestrians and traffic — and use it to construct benchmarks that test whether current state-of-the-art VLMs can handle realistic urban robot tasks.

Key Contributions

1. A new embodied AI simulator, SimWorld-Robotics (SWR), built on Unreal Engine 5, that creates and simulates photorealistic, dynamic urban environments with procedurally generated cities and three types of embodied agents (robots, humans, vehicles). SWR supports 26 human actions compared to 2 in CARLA, AirSim, MetaUrban, EmbodiedCity and SocNavBench, 10 in UnrealZoo, 3 in Sean2.0, 8 in Arena3.0, 4 in Habitat 3.0, 25 in VirtualHome 2.0, and 50 in TDW, and it uniquely combines procedural generation, photorealism, scene interaction, and asynchronous multi-agent control among the outdoor simulators compared.
2. Two new benchmarks leveraging the simulator's unique features: SimWorld-MMNav (single-robot multimodal instruction following) and SimWorld-MRS (two-robot cooperative search with verbal communication).
3. A large-scale dataset, SimWorld-20K, for long-horizon multimodal robot navigation, comprising 20K training steps sampled from 200 episodes averaging 500 m in length across 100 procedurally generated cities of average area 2 km².
4. A systematic evaluation of recent baselines, identifying specific capability gaps in state-of-the-art VLMs on urban navigation, safety, and collaboration.

Main Findings

• Zero-shot VLMs fail at the full navigation task. On the SimWorld-MMNav easy set, GPT-4o, Gemini 2.5 Flash, QwenVL 2.5 7B, QwenVL 2.5 72B, Gemma 3 27B, and InternVL 3 78B all score 0 SR%. Reasoning models do better: GPT-o3 reaches 5.0 SR% and GPT-o3-pro reaches 8.3 SR%.
• Fine-tuning produces the only non-zero open-model success. Fine-tuned QwenVL 2.5 7B scores 4.0 SR% (+4.0), 52.45 Subtask SR% (+35.59), and 53.63 Distance Progress% (+45.81). The authors note the absolute success rate remains low, partly because training is grounded on oracle action traces.
• Gemini 2.5 Flash leads zero-shot distance progress. It reaches 31.29 Distance Progress% and 37.06 Subtask SR%, versus GPT-4o's 15.60 and 33.07. GPT-4o's distance progress and subtask completion mismatch because it cannot detect termination conditions and often overshoots the goal.
• Reasoning ability helps. The authors report that reasoning models show improved depth estimation and destination alignment, whereas all zero-shot non-reasoning models score zero SR.
• The hard setting exposes safety failures. GPT-4o records 2.08 SR%, 1.92 static collisions, 10.37 dynamic collisions, and 3.02 red-light violations. Gemini 2.5 Flash records 0 SR%, 3.21 static collisions, 4.29 dynamic collisions, and 7.875 red-light violations — better pedestrian/vehicle avoidance but a higher violation count, attributed to the agent freezing after detecting a red signal, even inside an intersection. Reasoning models were excluded from this setting because of inference latency.
• Dominant failure modes are quantified. For "Moving to Intersection": misestimating distance 53.33%, failing to detect the intersection 28.33%, misidentifying the reference landmark 18.33%. For "Turning": misinterpreting the turning pattern 42.86%, misunderstanding history status summary 42.86%, failing to detect upfront buildings 14.29%. For "Reaching Destination": failing to match a landmark from a different perspective 60.00%, stopping too early 30.00%, failing to align the landmark 10.00%.
• Hybrid and RL baselines fall short. HybridGPT (GPT-4o high-level plus A* low-level) scores 0 SR%, 32.53 Subtask SR%, 27.24 Distance Progress%. The RL baseline VLA-RL scores 0 SR%, 28.37 Subtask SR%, 22.79 Distance Progress% — failing to outperform zero-shot LLMs.
• Ablations point to ReAct and segmentation. Using GPT-4o as the backbone, the explicit ReAct framework and segmentation provide the most significant marginal improvements.
• Multi-robot collaboration results. On SimWorld-MRS, Table 5 reports GPT-4o with Oracle Planner at 65.00 CSR% and 76.90 Task Progress%, Gemini 2.5 Flash with Oracle Planner at 54.55 CSR% and 75.84 Task Progress%, GPT-4o with RoCo at 33.33 CSR% and 22.93 Task Progress%, and QwenVL 2.5 72B with RoCo at 11.11 CSR% and 35.94 Task Progress%. Note: the body text states GPT-4o with the oracle planner achieves "52%" CSR and "68.44%" task progress, which does not match the 65.00 and 76.90 in Table 5.
• Scale advantage over MetaUrban. SWR offers environments 100× larger in area and episodes over 1.2× longer than MetaUrban, which averages 410 meters per episode and 0.02 km² per environment.

Methodology in Plain English

The authors built a city generator on top of Unreal Engine 5. Given a minimal specification, the pipeline runs in four stages: roads, buildings, street elements, and traffic elements. Roads grow through a priority-queue strategy that balances branching and depth, with road-end attachment and intersection validation to keep layouts plausible. Buildings are placed along roads using collision-aware sampling and greedy gap-filling. Street props such as trees, cones, benches, and parked vehicles are then placed in contextual zones. Finally, dynamic actors — pedestrians and vehicles — are added.

Internally, dedicated managers for roads, buildings, and elements index space with lists and quadtrees for efficient queries and collision checks. Traffic runs on a waypoint system: four intersection waypoints per intersection (one per corner) and road waypoints sampled every 17 meters along each segment connecting intersections. Vehicle motion uses a feedback model with a PID controller; pedestrians incrementally adjust orientation toward goals. At intersections, agents route probabilistically, which introduces variability across scenes.

Embodied agents come in three kinds — humans, vehicles, and robots (a scooter and a quadruped) — and are controlled asynchronously rather than in lockstep. Each agent gets observations from a centralized buffer and can submit an action when marked available; the buffer updates at a fixed interval (default 0.01 seconds) and executes valid actions concurrently. Observations include RGB, depth, and semantic segmentation images, plus ground-truth language descriptions and 3D bounding boxes. Actions are continuous for vehicles (acceleration, braking, steering) and include continuous translation plus free-angle rotation for robots; humans have navigation and interaction actions grouped into human–object, human–vehicle, and human–human categories.

For the SimWorld-MMNav benchmark, each task chains 2–4 multimodal instructions, each pairing a natural language instruction with a visual hint showing what the robot should see after reaching the goal. Instructions fall into four types: Orientation Alignment, Move Along the Road, Turn at the Intersection, and Reach Destination. Only one instruction is shown at a time, and the robot must complete the current one before receiving the next. Easy tasks contain no obstacles (no static objects outside buildings, no pedestrians or vehicles); hard tasks add both static obstacles and dynamic pedestrians and vehicles. Evaluation uses Success Rate, Subtask SR, and Distance Progress for easy, plus Static Collision, Dynamic Collision, and Traffic Light Violation for hard. The training set uses 200 oracle trajectories generated by A* with average length greater than 2.5 km, each over 100 steps. Evaluation uses 100 distinct worlds, 200 easy tasks, and 200 hard tasks, with tasks averaging 500 meters over 250 steps and 33% of test-set buildings exclusive to testing.

For SimWorld-MRS, two robots — a main robot with a memorized map and landmark images, and a follower robot new to the city — must meet using physical navigation and natural language messages, each able to send a confirmation signal when it believes it sees the other. Evaluation uses 100 unique environments of 2.5 km² each, 20 distinguishable landmarks per environment distributed across all city blocks, an average initial inter-robot distance of 576 m, and 287 steps for an oracle planner. Metrics are Collaborative Success Rate and Task Progress. Baselines pair VLMs with an oracle planner or follow RoCo's collaboration paradigm, where the follower first describes its location, then the two confirm a plan.

Baselines for SimWorld-MMNav were driven with the ReAct framework using GPT-4o, GPT-o3, GPT-o3-pro, Gemini 2.5 Flash, Qwen-VL 2.5, Gemma 3, and InternVL; a fine-tuned QwenVL2.5-7B; HybridGPT; and VLA-RL, trained with DeBERTa-v3 for language and DINOv2 for vision, following VLN-CE.

Why This Matters

Impact on research. The paper shifts embodied AI benchmarking from indoor, household settings toward city-scale outdoor environments, where grounding, spatial reasoning, safety, and communication all interact. It also provides a large synthetic training set (SimWorld-20K) and shows that fine-tuning a 7B VLM on it can beat much larger proprietary models on several metrics — evidence that simulator-generated data is a viable path to stronger embodied models.

Real-world applications:

• Urban delivery and logistics robots navigating sidewalks, crossings, and intersections.
• Search-and-rescue operations where multiple robots must locate each other and rendezvous in unfamiliar terrain.
• Autonomous micromobility and pedestrian-shared space navigation with social and traffic-rule compliance.
• Assistive or service robots operating alongside pedestrians in cities, requiring grounded natural-language communication.

Industry relevance. Simulator-generated data with fine-grained ground-truth annotations offers a scalable alternative to expensive real-world data collection for robot foundation models. The benchmark results — with most zero-shot models at 0 SR% and the best fine-tuned model at 4.0 SR% — set a clear capability baseline that industry teams can measure against, and the failure-mode statistics identify concrete engineering targets (intersection detection, depth estimation, landmark re-identification across viewpoints, termination detection).

Future Directions

1. Extend beyond outdoor environments. The authors state SWR currently focuses only on outdoor environments and that they plan to incorporate indoor scenes.
2. Expand human action spaces. The authors note the human action set, though more diverse than prior simulators, is still limited, and they intend to scale it using recent human body motion generation models.
3. Improve fine-tuned model robustness. The authors suggest incorporating reinforcement learning or corrective demonstrations, since oracle action traces limit robustness and leave absolute success rates low.
4. Address iterative replanning in multi-robot coordination. The paper notes that without iterative replanning, grounding and control errors under the RoCo-style one-shot communication scheme cannot be corrected, causing path deviations.

Target Audience

Researchers and engineers working on embodied AI, robot navigation, vision-and-language navigation, and multi-agent systems. It is also relevant to simulation and graphics engineers interested in procedural city generation on Unreal Engine 5, and to teams building robot foundation models who need large-scale synthetic training data with ground-truth annotations. Readers should be comfortable with VLM architectures, reinforcement learning baselines, and navigation evaluation metrics.

Authors’ abstract

Recent advances in foundation models have shown promising results in developing generalist robotics that can perform diverse tasks in open-ended scenarios given multimodal inputs. However, current work has been mainly focused on indoor, household scenarios. In this work, we present SimWorld-Robotics~(SWR), a simulation platform for embodied AI in large-scale, photorealistic urban environments. Built on Unreal Engine 5, SWR procedurally generates unlimited photorealistic urban scenes populated with dynamic elements such as pedestrians and traffic systems, surpassing prior urban simulations in realism, complexity, and scalability. It also supports multi-robot control and communication. With these key features, we build two challenging robot benchmarks: (1) a multimodal instruction-following task, where a robot must follow vision-language navigation instructions to reach a destination in the presence of pedestrians and traffic; and (2) a multi-agent search task, where two robots must communicate to cooperatively locate and meet each other. Unlike existing benchmarks, these two new benchmarks comprehensively evaluate a wide range of critical robot capacities in realistic scenarios, including (1) multimodal instructions grounding, (2) 3D spatial reasoning in large environments, (3) safe, long-range navigation with people and traffic, (4) multi-robot collaboration, and (5) grounded communication. Our experimental results demonstrate that state-of-the-art models, including vision-language models (VLMs), struggle with our tasks, lacking robust perception, reasoning, and planning abilities necessary for urban environments.