TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Model

AI summary

Overview

Research area: Robotics — vision-language navigation (VLN), humanoid whole-body control, and vision-language-action (VLA) foundation models. Technical level: Intermediate. Familiarity with navigation benchmarks and transformer-based policies helps, but the core ideas are explained clearly. Scope: This paper introduces TANGO, a unified whole-body VLA framework that lets a humanoid robot follow natural-language instructions through cluttered indoor spaces by directly predicting 29-DoF joint actions, trained entirely in simulation and deployed zero-shot on real hardware.

What This Paper Is About

Most navigation systems treat moving through a building as a 2D path-planning problem: pick a route, then send low-level "go forward" commands. That assumption breaks for humanoids, whose arms, torso, and legs constantly change shape and can collide with narrow passages, low steps, or overhead fixtures. TANGO reframes vision-language navigation as a whole-body problem, asking the robot to jointly decide where to go and how to contort its 29-degree-of-freedom body to get there, using only language instructions and RGB cameras as input.

Key Contributions

1. First whole-body vision-language navigation framework. TANGO directly predicts 29-DoF joint-space actions from egocentric RGB observations and natural-language instructions, eliminating the separate navigation and whole-body control modules used in prior work.
2. A scalable simulation data pipeline (Plan–Edit–Track, or PET). The authors automatically synthesize collision-free, dynamically feasible humanoid traversal behaviors — stepping over obstacles, side-stepping through narrow gaps, and crouching under overhead objects — producing 64,633 training trajectories at a cost of 211 GPU-hours.
3. A triple-system architecture. A Qwen2.5VL-7B vision-language backbone (system-2), a flow-matching diffusion action expert with real-time chunking (system-1), and an off-the-shelf SONIC motion tracker (system-0) are combined to convert language into executable high-frequency control.
4. Zero-shot sim-to-real deployment. TANGO runs on a Unitree G1 humanoid without any real-world navigation training data, navigating a 30-meter cluttered office route and handling obstacle-negotiation scenarios.

Main Findings

• State-of-the-art VLN results despite harder constraints: On the VLNVerse benchmark, TANGO achieves the highest success rate (54.69% seen, 52.89% unseen) and lowest navigation error among all methods — and it is the only method evaluated with actual low-level physical control rather than teleportation.
• Superior collision avoidance in cluttered scenes: On augmented cluttered scenes, TANGO reaches 43.75% success rate with a collision rate of 9.90%, compared to 15.81% for the best modular baseline (InternVLA-N1 + HumanoidPF), even though that baseline additionally uses LiDAR while TANGO uses only RGB.
• Real-world zero-shot transfer works: Across 45 trials in cluttered 3D, long-horizon, and short-horizon settings, TANGO outperforms the baseline in every setting (10/15 vs 6/15 successful trials in cluttered 3D, with fewer than half the average collisions).
• Whole-body actions matter: A planar-only variant of TANGO collapses under physical execution (26.67% SR) compared to the full 29-DoF model (52.89% SR), showing that conventional 2D action spaces fail once real physics and articulated geometry are involved.
• Real-time chunking is critical: Removing RTC causes a dramatic 32.81-percentage-point success rate drop (43.75% to 10.94%) and increases collisions, because actions lose continuity across chunk boundaries.
• Obstacle-aware motion editing aids safety: Removing motion editing from the training pipeline lowers success to 36.25% and roughly doubles the collision rate to 20.60%.
• Controller-agnostic: Swapping the SONIC tracker for ScaleBFM yields comparable results (40.94% SR), suggesting the learned policy is not tied to one specific whole-body controller.

Methodology in Plain English

The researchers sidestep the difficulty of collecting real humanoid navigation data by generating everything in simulation through a three-stage pipeline called Plan–Edit–Track:

1. Plan: Given a start and goal in an augmented indoor scene, A* path planning finds a safe route, with a soft penalty pushing the path away from obstacles. If the route passes through a narrow gap, the system rewrites the robot's heading by 90 degrees so it walks sideways rather than trying to squeeze through face-first.
2. Edit: The planned walking motion is then modified with pseudo-forces applied to key body links, similar to how a puppeteer might push limbs aside. A gait-adaptation module stretches foot placement to step over floor obstacles, and other edits produce arm clearance and crouching.
3. Track: A pre-trained whole-body controller (SONIC) attempts to physically execute each edited motion in simulation. Trajectories that collide or fail to track are discarded. Crucially, the reference motions — not the tracked versions — become training supervision, preserving natural-looking human motion while guaranteeing physical feasibility.

The resulting dataset trains a model that reads stacked front and downward camera images plus a language instruction. A vision-language backbone processes the observation history using budget-aware token sampling, which allocates finer visual detail to recent frames and coarser detail to older ones. A flow-matching diffusion transformer then predicts a chunk of future whole-body actions. Real-time chunking is applied during training so the model learns to keep motions smooth when only part of each chunk is executed. At runtime, actions stream from a server GPU to an onboard Jetson, which runs the tracker at roughly 200 Hz.

Why This Matters

Impact on research: TANGO shows that navigation and whole-body control need not be separate modules. By demonstrating that a single model can output 29-DoF actions directly from language and RGB, it opens a path toward unified loco-manipulation foundation models — robots that use their entire bodies as part of the planning problem, not just as an afterthought.

Real-world applications:

• Domestic service robots navigating cluttered homes with furniture, toys, and low doorways.
• Warehouse or laboratory assistants moving through narrow aisles between shelves and equipment.
• Search-and-rescue humanoids entering damaged buildings with debris and collapsed ceilings.
• Elder-care or hospital robots that must maneuver around beds, wheelchairs, and overhanging medical equipment.

Industry relevance: The pipeline shows that large-scale whole-body navigation data can be synthesized cheaply in simulation, which lowers the barrier for companies without expensive motion-capture studios. The server-edge deployment architecture (heavy VLA on a server, lightweight tracker onboard) is also a practical template for commercial humanoid products where onboard compute is limited.

Future Directions

• Extending beyond the current tracker's ability: The authors note that climbing stairs remains out of reach because the low-level tracker cannot yet handle those terrains — improving the tracker would unlock harder environments.
• Adding depth or LiDAR sensing: TANGO currently relies only on RGB, which limits performance in visually ambiguous, dark, or texture-poor spaces, where depth cameras and LiDAR would likely help.
• Scaling toward general loco-manipulation: The framework could be extended so the robot not only navigates but also actively manipulates objects while moving, using its whole body in coordination.
• Studying language specification effects: The real-world trials used instructions that explicitly named the required behavior (step over, side-step, bend down); whether the model can infer the correct traversal strategy from more implicit language is an open question.

Target Audience

This paper is most valuable for robotics researchers working on vision-language navigation, humanoid control, or embodied AI foundation models. It will also interest engineers building real-world humanoid platforms who need practical deployment architectures, and graduate students looking for a template of how to combine simulation data generation with zero-shot sim-to-real transfer. Readers with a background in imitation learning, diffusion models, or reinforcement learning will find the architecture and training details accessible.

Authors’ abstract

We study the problem of navigating cluttered indoor environments with a humanoid robot. Unlike conventional methods that model navigation as a 2D path planning problem, humanoid traversal in cluttered environments requires continuous geometry-aware whole-body adaptation, including coordinated arm placement, torso adjustment, and gait modulation for collision-free movement through complex 3D spaces. We introduce TANGO, the first whole-body vision-language navigation framework for language-conditioned humanoid traversal in cluttered environments. Given a natural-language instruction and egocentric RGB observations, TANGO directly predicts 29-DoF joint-space actions for downstream whole-body control. We train TANGO entirely in simulation by synthesizing diverse collision-free traversal behaviors via global path planning, kinematic whole-body motion generation, obstacle-aware motion editing, and RL-based tracking. This pipeline provides dynamically feasible action supervision for learning language-conditioned whole-body policies. In extensive simulation experiments, TANGO demonstrates state-of-the-art performance in vision-language navigation, while outperforming strong modular baselines in navigating challenging scenes requiring obstacle negotiation. Lastly, we deploy TANGO zero-shot on a Unitree G1 humanoid robot, and observe robust language-guided traversal in cluttered real-world scenes without training on any real-world navigation data.