HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments

AI summary

Overview

Research area: robotics — specifically Vision-Language Navigation (VLN) for bipedal humanoid robots, combining physics-based simulation, 3D scene reconstruction, and multimodal instruction annotation.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning locomotion policies, PD and MPC control, 3D Gaussian Splatting, and modern vision-language-action models.

Scope: The paper introduces HumanoidVLN, an Isaac Sim-based simulator plus a 933-episode benchmark that evaluates four VLN models across four humanoid robot embodiments under physically executed bipedal locomotion, and reports a 20-episode simulation-to-reality pilot study.

What This Paper Is About

Existing VLN benchmarks move agents by kinematic teleportation, so they ignore the physical limits of walking on two legs — gait stability, step reachability, and center-of-mass dynamics. They also treat humanoids as one interchangeable robot type, use scenes that are not curated for large traversable areas, and generate instructions from idealized videos that lack the camera shake of real bipedal walking. HumanoidVLN addresses all three gaps with a physics-grounded simulator, a navigability-filtered scene suite, and a verified instruction-generation pipeline, then benchmarks four VLN models across four humanoid morphologies.

Key Contributions

1. Embodiment-diverse humanoid simulation platform. An Isaac Sim framework supporting Unitree G1 (12 lower-body DoF, 1.32 m), Unitree H1 (10 DoF, 1.80 m), Internal-A (12 DoF, 1.61 m), and Internal-B (12 DoF, 1.17 m), each driven by a hierarchical stack of a per-embodiment RL locomotion policy beneath an interchangeable PD or MPC path tracker, with a plug-and-play interface for new robots and VLN models.

2. Navigability-curated Real2Sim scene suite. 87 high-fidelity 3D environments with at least 100 m² navigable area, spanning 17 indoor classes across 6 domains, drawn from artist-designed and 3D Gaussian Splatting reconstructed sources. The median navigable area is 266 m² and the mean is 387 m².

3. MAA-augmented instruction dataset. 933 collision-aware episodes, each with one spatially grounded fine-grained instruction plus three coarse-grained stylistic variants (Formal, Natural, Casual), produced by a Dual Generator-Reviewer + Paraphraser Multi-Agent Annotation pipeline with a human-in-the-loop verification pass.

4. Cross-embodiment physics evaluation. A matched zero-shot evaluation of NaVILA, StreamVLN, DualVLN, and JanusVLN across four humanoid configurations under physics-based execution, measuring navigation accuracy, path fidelity, and a newly defined Fall Rate.

Main Findings

• Best overall model: JanusVLN achieves the highest average Success Rate (43.55%) and the highest average nDTW (48.38) across the four embodiments, a result the authors associate with its explicit 3D spatial representation under physical humanoid execution.

• Remaining model ranking: DualVLN leads the other three models in nDTW (43.39), indicating stronger path adherence. NaVILA shows the largest average Oracle Success Rate minus Success Rate gap (17.39 points), consistent with less reliable stopping decisions. StreamVLN records the lowest average SR (23.63%) and nDTW (36.67) of the four.

• Embodiment matters as much as the model: Unitree H1 (10 DoF, 1.80 m) performs worst across all models, with an average SR of 20.84% versus 32–37% for the other three robots. The largest cross-robot SR spreads occur for JanusVLN (18.90 points) and DualVLN (14.98 points). Internal-A achieves the highest average SR (36.66%), while G1 achieves the highest average SPL (25.19), so robot rankings change depending on the metric.

• Fall Rate exposes instability that SR hides: Fall rates stay low on Internal-A, G1, and Internal-B (2.7–10.0%) but rise sharply on Unitree H1, reaching 64.5% for StreamVLN and 71.0% for NaVILA. Averaged across embodiments, DualVLN has the lowest Fall Rate, which the authors suggest may relate to its continuous action interface.

• 3DGS reconstruction ablation: Vanilla gsplat produces noisy, discontinuous surface normals, especially near textureless walls, furniture boundaries, and thin structures, yielding collision meshes unsuitable for reliable physics simulation. Adding depth-normal consistency improves local smoothness, and the full pipeline combining unbiased depth rendering with depth-normal consistency produces the most coherent geometry and the most reliable TSDF-fused collision meshes.

• Simulation tracks reality at the episode level: In a 20-paired-episode pilot with DualVLN on Unitree G1 across a pantry and a studio room, per-episode navigation error correlates strongly between simulation and reality (Pearson r = 0.935; Spearman ρ = 0.911). The mean absolute NE difference is 0.68 m and the mean signed difference is only 0.04 m. Paired trajectory similarity averages 78.2 ± 18.8 nDTW, with scene-level means of 0.803 in the studio and 0.761 in the pantry.

• Dataset statistics: Across 933 episodes, path length has a median of 9.97 m and a mean of 10.74 m; instructions have a median of 50.0 words and a mean of 50.3 words. The path–instruction word-count correlation is modest (r = 0.29), indicating instruction length is not determined solely by trajectory length.

Methodology in Plain English

The team built their simulator on NVIDIA Isaac Sim so that motion is produced by rigid-body dynamics rather than by teleporting the agent along a path. Each of four humanoid robots gets its own learned RL locomotion policy that outputs joint torques for stable walking, and on top of that sits a path-tracking controller that turns a navigation plan into velocity and heading commands. Continuous-action models (DualVLN) are paired with an MPC tracker; discrete-action models (NaVILA, StreamVLN, JanusVLN) use a PD tracker. A VLN model only needs to consume the head-camera RGB stream plus an instruction and emit actions, so new models plug in without modification.

For environments, the team combined artist-designed scenes with scenes reconstructed from real multi-view captures using 3D Gaussian Splatting trained with the open-source gsplat library. They modified the training pipeline to render unbiased depth and enforce depth-normal consistency, then extracted collision meshes by TSDF fusion of rendered depth maps and placed them with the Gaussians in the same USDZ scene. They aligned COLMAP's OpenCV camera convention to Isaac Sim's frame with a fixed axis-switching transform. Only scenes whose verified traversable floor area exceeds 100 m² were kept.

For episode construction, they slice the collision mesh at the robot's body height to build a 2D occupancy map of overhead obstacles, dilate it by the largest supported robot's circumscribed footprint radius, sample random start-goal pairs, and connect them with A* paths. Paths that cannot be completed stably are resampled, so every episode's video necessarily contains real locomotion-induced camera dynamics.

For instructions, two generator models (Gemma-4-31B-it and InternVL3.5-38B) independently infer a structured route graph from egocentric keyframes alone, while Qwen3-VL-30B-A3B supplies the goal landmark and stopping condition from terminal frames. Routes are compared on navigation-critical attributes such as turn direction, landmark identity, side-of-path relations, ordinals, and step order; non-conflicting additions merge and localized contradictions go to a Qwen3-VL-30B-A3B reviewer that checks against trajectory metadata, the A* route over the occupancy map, and a spatial-semantic scene graph restricted to objects visible along the trajectory. Deterministic checks handle geometrically decidable properties. GPT-5.5 then paraphrases the verified instruction into Formal, Natural, and Casual variants at temperature T = 1.0, and each variant is re-parsed and accepted only if its navigation-critical attributes match. Finally, a pool of three trained annotators reviews every episode, with 20% independently re-reviewed by a second annotator and disagreements resolved by a third.

Evaluation is strictly zero-shot on all 933 episodes with no fine-tuning. Metrics are Success Rate (stop within 3.0 m of goal), Oracle Success Rate, Navigation Error in meters, SPL, nDTW, and a new episode-level Fall Rate triggered by three criteria: T1 (dynamic fall, Δh ≥ 0.5·H_e with downward speed above 1.2 m/s), T2 (sustained collapse, Δh ≥ 0.5·H_e for at least 2 s), or T3 (shallow dynamic fall, 0.35·H_e ≤ Δh < 0.5·H_e with downward speed above 1.5 m/s). A detected fall terminates the episode without automatic recovery.

Why This Matters

Impact on research: The paper argues that Fall Rate and embodiment-specific behavior are invisible to benchmarks that move agents kinematically, and demonstrates that the same VLN model can swing from roughly 50% SR to roughly 29% SR depending on which humanoid is under test. It also provides an episode-level sim-to-real correlation (r = 0.935) that supports using reconstructed scenes as a proxy for real evaluation, and it releases a controlled platform where morphology, controller, and model can be varied independently.

Real-world applications:

• Home and service robotics, where a humanoid must follow spoken directions through pantries, living spaces, and studios while staying upright.
• Retail, food, and hospitality floors, which appear in the paper's six application domains (Residential, Food & retail, Culture & leisure, Workplace & education, Healthcare, Fitness).
• Warehouse and workplace logistics, where robots must respect obstacle-sparse traversable corridors and avoid falls around people and equipment.
• Healthcare and assisted-living settings, where stable navigation and reliable stopping behavior matter more than raw speed.

Industry relevance: The platform's plug-and-play interface means a new robot needs only a URDF/USD description and a trained locomotion policy, and a new VLN model needs only the observation-action interface. That lowers the cost of benchmarking commercial humanoid hardware against competing navigation models before committing to physical trials. The pilot study also used commodity-class onboard compute (an NVIDIA Jetson AGX Orin) with inference offloaded to an NVIDIA RTX A6000 Pro GPU, which is representative of deployable robot architectures.

Future Directions

• Integrating 3D foundation models to strengthen instruction grounding beyond the current generator-reviewer-paraphraser chain.
• Developing more efficient 3DGS reconstruction to reduce the cost of building new reconstructed scenes.
• Expanding simulation-to-real validation beyond the current pilot, which covers only two scenes, one VLN checkpoint (DualVLN), and 20 paired episodes and therefore does not establish scene-level generalization.
• Addressing the paper's own stated limitations: limited scene diversity, human verification acting as a scaling bottleneck for annotation, and the computational cost of full physics simulation.

Target Audience

Robotics and embodied-AI researchers working on vision-language navigation, humanoid locomotion, and sim-to-real transfer; engineers at humanoid and mobile-manipulation companies who need a physically grounded benchmark before hardware trials; and simulation or 3D-reconstruction practitioners interested in the 3D Gaussian Splatting to collision-mesh pipeline.

Authors’ abstract

Vision-Language Navigation (VLN) for humanoid robots poses challenges existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled agents, humanoid morphologies vary across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. We present HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments. Built on NVIDIA Isaac Sim, our platform supports an extensible set of humanoid configurations, demonstrated on four robots (Unitree G1, Unitree H1, Internal-A, Internal-B) spanning 10-12 lower-body DoF and heights from 1.17m to 1.80m, via a hierarchical control stack combining a reinforcement learning locomotion policy with interchangeable PD or MPC path trackers. New robots and VLN models integrate with minimal effort; we demonstrate compatibility with NaVILA, DualVLN, StreamVLN, and JanusVLN. Environments are drawn from artist-designed scenes and 3D Gaussian Splatting reconstructions, filtered for navigable areas exceeding 100 square meters. Instructions are generated by a dual generator-reviewer plus paraphraser multi-agent pipeline with human-in-the-loop verification, yielding 933 collision-aware reference episodes, each paired with one fine-grained instruction and three coarse-grained stylistic variants (formal, natural, casual). Across four models and four embodiments, JanusVLN achieves the highest mean success rate of 43.55% and nDTW of 48.38. In a 20-episode sim-to-real pilot with DualVLN and the Unitree G1, navigation errors correlate strongly (r=0.935), with a mean absolute difference of 0.68m and mean trajectory similarity of 0.782 (+/-0.188) nDTW. These results highlight the interaction between VLN models, controllers, and humanoid embodiments under physical execution. Code, benchmark, and data will be released upon acceptance at https://humanoid-vln.github.io/.