HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness

AI summary

Overview

Research area: Embodied AI and robot navigation, specifically zero-shot (training-free) instruction-following navigation and object-goal navigation built on multimodal large language models (MLLMs).

Technical level: Advanced. The paper assumes familiarity with vision-and-language navigation (VLN) benchmarks, ObjectNav, MLLM-based planners, and agent/tool orchestration.

Scope: HarnessVLN is a zero-shot, training-free framework that runs two navigation task families through one shared "Agent Harness" coordinating perception, memory, validation, and execution tools, evaluated on four benchmarks and deployed on a humanoid robot.

What This Paper Is About

Existing MLLM-based navigation systems use the model mainly as a planner inside a fixed task pipeline, where perception feeds observations in and downstream modules handle grounding and control. This leaves no unified mechanism for keeping spatial evidence, validating proposed actions, or recovering from execution failures, so a plausible action proposal can drift away from the agent's actual task state. The paper proposes a shared runtime "harness" that validates every proposed action against evidence, geometric feasibility, subgoal consistency, and failure history before it is executed — and that decides when the agent is genuinely allowed to stop.

Key Contributions

1. A shared Agent Harness for two navigation task families. HarnessVLN is a training-free framework supporting both instruction-following and object-goal navigation through one Agent Harness and a unified tool interface, using task-specific progress representations and completion criteria rather than separate pipelines.

2. A navigation runtime combining two memories with validation and feedback. The Harness pairs a hierarchical event memory (working, progress, reflection) with a persistent Spatiotemporal Graph, plus proposal validation and structured execution feedback, enabling evidence-grounded action, failure recovery, and verified termination.

3. Benchmark evidence across four tasks. Using the same framework, HarnessVLN reports success rates of 60.8% (R2R), 53.9% (RxR), 76.0% (HM3D-v2), and 59.3% (HM3D-OVON), exceeding the best compared training-free results by 5.8, 12.1, 1.6, and 9.1 percentage points respectively.

4. Real-world humanoid deployment. The framework was deployed on a humanoid robot for instruction following, open-vocabulary object search, and a combined route-following plus object-search task, with no task-specific policy training in the deployment environment.

Main Findings

• Instruction-following results (R2R and RxR). HarnessVLN with GPT-5.5 reaches the highest training-free success rate on both. On R2R it beats AgenticNav with the same GPT-5.5 by 5.8 percentage points in SR and 7.7 points in OSR, reducing Navigation Error from 5.19 m to 4.01 m; its 72.7% OSR also exceeds all listed training-based methods. On RxR it improves on HSGM by 12.1 points in SR and 12.9 points in SPL, reduces NE by 1.01 m, and holds comparable nDTW (54.8% versus 54.9%).

• Object-goal results (HM3D-v2 and HM3D-OVON). On HM3D-v2 it reaches 76.0% training-free SR, surpassing MSGNav by 1.6 points and approaching the best listed training-based result of 77.0% (FiLM-Nav). On HM3D-OVON it reports the highest SR and SPL among all listed methods at 59.3% and 36.6%, beating DRIVE-Nav by 9.1 and 4.0 points, and exceeding the strongest listed training-based result (ABot-N0) by 5.3 points.

• Ablation: memory and graph both help. On fixed 100-episode subsets, hierarchical event memory improves SR by 8.0 points (R2R) and 7.0 points (HM3D-OVON); adding graph-based retrieval contributes a further 6.0 and 1.0 points and improves SPL on both; stop validation adds 4.0 and 2.0 points. In full, the Harness improves SR by 18.0 and 10.0 points over the base agent.

• Stop validation has trade-offs. Stop validation cuts the R2R OSR–SR gap from 15.0 to 13.0 points, but slightly widens the HM3D-OVON gap from 18.0 to 19.0 points and lowers SPL from 34.2% to 33.0% — its completion gains do not uniformly improve efficiency or termination behavior.

• Proximity to a target is not the same as task success. In the visualized cases, R2R 839 passes the projected-target check at 0.998 m yet terminates 6.48 m from the goal, and HM3D-OVON 2469 stops after a passing depth check at 1.20 m but is 2.15 m from the benchmark goal and fails.

• A rejected stop can still lead to success. In HM3D-OVON 1297, a search for a "picture" had stopping rejected at step 216 (fresh depth 4.65 m, despite a projected-target distance of 0.40 m); a later depth check passed at step 429 (1.53 m) and the episode succeeded at step 430 with a terminal goal distance of 0.13 m.

• Matched-model comparison favors the Harness. On a fixed 100-episode HM3D-OVON subset with GPT-5.6-luna, MSGNav scores 37.0% SR while HarnessVLN scores 55.0%; HarnessVLN rises to 63.0% with GPT-6-astra and 51.0% with Qwen3.8-flash, indicating sensitivity to the base model.

• Real-world deployment. On an AgiBot A3U humanoid (1.74 m tall, with 3D LiDAR, RGB-D and fisheye cameras, NVIDIA Thor onboard compute), the same Harness protocol handled sequential instruction following, open-vocabulary printer search, and a combined signboard-stop then vending-machine-search task, using Qwen-3.8-27B as the planning model.

Methodology in Plain English

Everything runs as a loop. The agent looks around (RGB-D plus a local geometric map), and the Harness assembles a context from two memories: an event memory recording what subgoals exist and what happened when they were attempted, and a Spatiotemporal Graph that stores places, objects, spatial relations, and timestamps so the agent can tell fresh evidence from stale evidence.

An MLLM planner then proposes an action. Crucially, it cannot execute anything directly. The Harness checks the proposal first: is there supporting evidence, is the target geometrically reachable, and does it match the currently active subgoal given any relevant past failures? Only then is it dispatched to a tool. The tools include directional and panoramic observation, memory retrieval, target grounding, depth querying, navigation via an executor, backtracking, and a stop request. Each tool returns structured feedback — arrival, collision, unreachability, lack of progress — which updates the memories and steers the next decision.

Stopping is treated as a claim to be verified rather than a command. A model's stop proposal is accepted only if semantic identity, geometric validity, and task-progress checks all pass; for object-goal navigation the target must be visually supported and within the stopping radius (the Harness's geometric thresholds are 2.5 m and 1.0 m, distinct from benchmark success distances). Rejected stops are recorded and trigger more observation, target refinement, approach, or backtracking.

Implementation uses GroundingDINO and SAM for open-vocabulary regions and masks, an FMM planner for planar motion, and NavDP when stairs must be traversed. GPT-5.5 is the base model for instruction following and GPT-5.6-luna for object-goal navigation; no task-specific fine-tuning is performed, and individual tools can be swapped without changing the planner or the Harness protocol.

Why This Matters

Impact on research. The paper reframes the role of the MLLM in embodied navigation: rather than treating it as a planner whose outputs are trusted, it treats model proposals as claims requiring evidential and geometric verification. It also shows one runtime protocol can serve two different task families, and it provides a failure-aware memory design (ST Graph plus event memory) that keeps retrieval compact as trajectories grow — offering a concrete alternative to training-heavy navigation policies.

Real-world applications:

• Service and logistics robots on humanoid or mobile platforms that must follow spoken routes and fetch objects without retraining per site.
• Assistive robots in homes or care facilities that respond to open-vocabulary requests ("find the picture") and must know reliably whether they have actually arrived.
• Industrial or facility inspection robots following long multi-stage routes with landmarks and checkpoints.
• Warehouse and retail inventory tasks where target categories change frequently and open-vocabulary grounding matters more than a fixed label set.

Industry relevance. Because the framework is training-free and tool-replaceable, deployment does not require collecting and labeling navigation trajectories for each new environment or task, which lowers the barrier for robotics teams building on general-purpose MLLMs. The dependence on large proprietary models, however, means cost, latency, and API availability shape practical use.

Future Directions

• Replacing predefined orchestration with self-evolution. The conclusion states that the current Harness relies on predefined orchestration and validation, and that future work will explore self-evolving mechanisms refined through interaction in open environments.
• Improving efficiency, not just success. Reported SPL lags SR on several benchmarks (for example 60.8% SR against 43.5% SPL on R2R), and the ablation shows stop validation can lower SPL on HM3D-OVON — better path efficiency and termination behavior remain open.
• Closing the OSR–SR gap. Ablation gaps of 13.0 points (R2R) and 19.0 points (HM3D-OVON) mean the agent often reaches a near-goal state without being credited with success; making stop decisions match benchmark criteria better is an explicit target.
• Broadening base-model and task coverage. The appendix lists planned model-sensitivity evaluations with GPT-5.6-luna, Qwen3.8-flash, and GPT-6-astra, and the failure cases (entering the wrong room after passing a proximity check, stopping outside the success region) point to grounding and termination robustness as areas for further work.

Target Audience

Robotics and embodied-AI researchers working on navigation, VLN, and ObjectNav; engineers building MLLM-driven agents who need tool orchestration, memory design, and validation patterns rather than training pipelines; and graduate students or advanced practitioners interested in how agent-harness ideas from manipulation can be extended to long-horizon spatial tasks. Readers should be comfortable with navigation benchmarks (R2R, RxR, HM3D), success-rate/SPL metrics, and MLLM-based planning terminology.

Authors’ abstract

Embodied navigation requires agents to ground instructions or object goals in spatial observations and translate plans into successful execution. As multimodal large language models (MLLMs) become increasingly capable, they offer stronger support for navigation without task-specific training; however, improved semantic reasoning alone does not ensure that proposed actions remain consistent with spatial evidence, task progress, and execution outcomes. We introduce HarnessVLN, a zero-shot, training-free framework that unifies instruction-following and object-goal navigation through a shared Agent Harness. The Harness coordinates perception, memory, and execution tools through a unified interface, validating planner proposals for evidential support, geometric feasibility, and subgoal consistency before dispatch. It jointly manages hierarchical event memory and a persistent Spatiotemporal Graph to track task progress, preserve spatial evidence, and contextualize failures. Structured execution feedback updates this shared state, guiding subsequent planning, recovery, and termination. Across R2R, RxR, HM3D-v2, and HM3D-OVON, HarnessVLN achieves success rates of 60.8%, 53.9%, 76.0%, and 59.3%, respectively, outperforming prior training-free state-of-the-art methods. Humanoid robot deployment further demonstrates its applicability to both navigation tasks in real-world environments. The project page is available at https://agibot-harnessvln.netlify.app/.