UrbanNav: Learning Language-Guided Urban Navigation from Web-Scale Human Trajectories

AI summary

Overview

Research area: Robotics and embodied AI, specifically vision-and-language navigation (VLN) in outdoor urban environments.

Technical level: Intermediate. Readers should be comfortable with concepts like imitation learning, visual odometry, vision-language models, and transformer-based policy networks, but the paper's core ideas are accessible without deep specialization.

Scope: The paper introduces UrbanNav, a framework and dataset that trains embodied agents to follow free-form natural language instructions through real urban streets by learning from web-scale first-person walking videos scraped from YouTube.

What This Paper Is About

Autonomous agents such as last-mile delivery robots need to navigate unfamiliar cities using vague human instructions like "go to the café by the old bridge," but existing navigation models are trained mostly in simulators or indoor spaces and often require precise goals such as GPS coordinates or target images. The authors address this gap by building a system that learns navigation policies directly from millions of hours of real-world human walking footage, automatically annotating trajectories with landmarks and natural-language instructions. The result is a policy that can interpret noisy, context-dependent street-level directions and generalize to cities it has never seen.

Key Contributions

1. A scalable framework (UrbanNav) for language-guided urban navigation trained on web-scale human walking videos, targeting real city environments rather than simulations or indoor scenes.

2. An automated data pipeline that segments in-the-wild videos, extracts egocentric trajectories via visual odometry (DPVO), filters out robot-incompatible clips (excessive head pitch, viewpoint-motion misalignment, crowded scenes), and uses a large vision-language model (Qwen2.5-VL-72B) to detect landmarks and generate grounded instructions—producing 3 million instruction-trajectory-landmark triplets from over 1,500 hours of footage.

3. A policy architecture combining frozen CLIP (language) and DINOv2 (vision) encoders, FiLM-based multimodal fusion, a transformer encoder, and a multi-step waypoint prediction head with four training losses (regression, orientation, arrival, and feature hallucination).

4. Empirical validation on a physical robot, showing that web-scale pre-training delivers large gains in real-world success rate (83.3% vs. 62.5% for the strongest baseline) and robustness to noisy instructions, occlusion, and nighttime conditions.

Main Findings

• State-of-the-art offline accuracy: UrbanNav achieves the lowest average orientation error (AOE 8.88°), maximum average orientation error (MAOE 14.62°), average distance error (ADE 0.83 m), and maximum average distance error (MADE 1.57 m) on seen test data, beating NoMaD+CLIP, ViNT+CLIP, and LeLaN. It maintains this lead on entirely unseen environments.

• Strong real-world success rate: On a physical robot in unseen urban settings, UrbanNav succeeds in 83.3% of trials overall (91.7% daytime, 75.0% nighttime), versus 62.5% for the second-best baseline LeLaN and only 33.4% for a variant trained on real-world data without web-scale pre-training.

• Robustness to adversarial conditions: The model reaches 100% success with normal instructions, 87.5% with noisy or misleading instructions, and 62.5% when targets are initially occluded or out of view—each substantially above all baselines in the same conditions.

• Data scaling matters: Error metrics drop steadily as training data grows from 300 to 1,500 hours, with performance beginning to plateau around 1,200 hours.

• FiLM fusion is critical: Ablating the FiLM module that modulates visual features with language embeddings causes the largest single performance drop, indicating that instruction-conditioned attention to visual cues is central to the policy.

• Feature hallucination loss helps here: Unlike some prior work where predicting future visual features hurts zero-shot inference, this auxiliary loss improves results—the authors attribute this to their clean, robot-compatible filtered data.

Methodology in Plain English

The team started with over 2,000 hours of first-person walking videos from YouTube, capturing how real pedestrians move through cities like San Francisco, Tokyo, and European towns. They split videos into 2-minute clips and used a visual odometry model (DPVO) to estimate the camera's position frame by frame, turning each clip into a trajectory of coordinates.

Not all footage is usable. Humans bob their heads, look sideways, and sometimes weave through dense crowds—behaviors a fixed-camera robot cannot replicate. So the team filtered clips aggressively: rejecting any with camera pitch swings over 15°, with a mismatch between walking direction and camera direction over 60°, or with more than five pedestrians in a frame across multiple frames. This cut footage from 3,553 to 1,566 hours of "robot-compatible" data.

Next, they used Qwen2.5-VL-72B to spot stable landmarks (buildings, signs, sculptures—but not moving pedestrians or cars) near each trajectory and to write short natural-language instructions referencing those landmarks, like "walk past the red awning toward the corner bookstore." Averaging 65 landmarks per trajectory and 17-word descriptions, this yielded 3 million instruction-trajectory-landmark triplets.

The policy itself takes a language instruction plus the current and previous eight camera frames as input. CLIP encodes the text; DINOv2 encodes the images; a FiLM layer blends them so the instruction steers which visual features to focus on; and a transformer predicts the next eight waypoints in egocentric coordinates. Four losses supervise the training: waypoint accuracy, heading direction, arrival detection, and predicting high-level features of future frames.

Why This Matters

Impact on research: UrbanNav shows that web video—not just simulation or expensive teleoperation—can serve as a scalable supervisory signal for embodied navigation, and that robot-compatible data filtering is essential for making this work. It reframes real-world navigation as a data-problem as much as an architecture problem.

Real-world applications:

• Last-mile delivery robots that can be told "deliver to the pharmacy next to the green mosque" rather than given coordinates.
• Assistive navigation devices for blind or low-vision users that interpret verbal landmark-based directions.
• Autonomous ride-hailing and shuttles that must interpret passenger instructions referencing street-level landmarks.
• Search-and-rescue or inspection robots dispatched with colloquial human directions in unfamiliar urban terrain.

Industry relevance: The pipeline offers a cost-effective alternative to teleoperated data collection, which is expensive and geographically narrow. Logistics, robotics fleet operators, and mapping companies could fine-tune or continue-training on this style of data to deploy in new cities without retraining from scratch. The released code and dataset lower the barrier for competitors and follow-up research.

Future Directions

• Long-horizon exploration: The policy handles local navigation well but drops to 62.5% success when targets are initially out of view. Combining the model with active exploration or map-building could address this.

• Reducing reliance on VLM annotation: The pipeline depends on a 72-billion-parameter VLM plus manual review; distilling or self-supervised alternatives could make data curation cheaper and faster.

• Closing the sim-to-real and video-to-robot gap further: Human gait and head movement differ from wheeled or legged platforms; better alignment or explicitly modeling embodiment could improve transfer.

• Nighttime and adverse-weather robustness: Success drops from 91.7% to 75.0% at night. Learning from more low-light footage or adding sensor modalities (thermal, LiDAR) may close this gap.

• Interactive instruction following: Testing whether the agent can ask for clarification when instructions are ambiguous, rather than just tolerating noise.

Target Audience

Robotics and embodied AI researchers working on vision-and-language navigation, imitation learning, or outdoor autonomy; graduate students entering the field who want a concrete example of mining web video for robot training data; and industry engineers at delivery, mapping, or autonomous vehicle companies evaluating whether web-scale data can substitute for expensive teleoperation. The paper is most useful to readers with some prior exposure to VLN benchmarks (e.g., R2R, RxR) and modern VLM tooling, though the high-level pipeline is understandable to a broader ML audience.

Authors’ abstract

Navigating complex urban environments using natural language instructions poses significant challenges for embodied agents, including noisy language instructions, ambiguous spatial references, diverse landmarks, and dynamic street scenes. Current visual navigation methods are typically limited to simulated or off-street environments, and often rely on precise goal formats, such as specific coordinates or images. This limits their effectiveness for autonomous agents like last-mile delivery robots navigating unfamiliar cities. To address these limitations, we introduce UrbanNav, a scalable framework that trains embodied agents to follow free-form language instructions in diverse urban settings. Leveraging web-scale city walking videos, we develop an scalable annotation pipeline that aligns human navigation trajectories with language instructions grounded in real-world landmarks. UrbanNav encompasses over 1,500 hours of navigation data and 3 million instruction-trajectory-landmark triplets, capturing a wide range of urban scenarios. Our model learns robust navigation policies to tackle complex urban scenarios, demonstrating superior spatial reasoning, robustness to noisy instructions, and generalization to unseen urban settings. Experimental results show that UrbanNav significantly outperforms existing methods, highlighting the potential of large-scale web video data to enable language-guided, real-world urban navigation for embodied agents.