StopWatch: Goal Grounding for Object-goal Navigation

AI summary

Overview

Research area: Embodied AI / vision-language-action (VLA) navigation, specifically open-vocabulary object-goal navigation in continuous 3D indoor environments (HM3D, MP3D, HSSD).

Technical level: Intermediate. The paper is readable without deep mathematics, but assumes familiarity with vision-language models, behavior cloning, and standard embodied navigation benchmarks.

Scope (one sentence): The paper diagnoses "goal grounding"—knowing whether the goal is in view and how far away it is—as the dominant failure mode of fine-tuned VLM navigation policies, and fixes it with a two-model pipeline plus corrected benchmark annotations.

What This Paper Is About

Fine-tuning a pretrained vision-language model on expert demonstrations has become the standard recipe for object-goal navigation, where an agent must find an instance of a named object category in an unfamiliar scene. Despite larger demonstration sets and better history modeling, success rates on unseen goal categories stay low. The authors show the problem is not search but stopping: policies reach the goal and then walk past it, because they cannot tell that the goal is in view or estimate its distance. Their goal is to make that grounding explicit, supervised, and separable from the navigation policy itself.

Key Contributions

1. Diagnosis of goal grounding as the missing capability. The authors show that an end-to-end policy comes within 0.25 m of the goal in 76% of episodes on novel categories yet succeeds in only 52%, and that supplying ground-truth grounding text to the same policy raises success to 89% and drops failures after reaching the goal from 24% to 5%.

2. A two-model pipeline built around explicit, structured grounding text. A perception model (fine-tuned from Qwen3-VL-4B) writes a text describing the object categories in view and a coarse distance to the goal; a policy model (fine-tuned from Qwen3-VL-2B) reads frames interleaved with that text and outputs the next action. The two are trained separately, both supervised by scene annotations.

3. State-of-the-art results on six benchmarks across three scene collections. The system leads every benchmark under every reported protocol, including an 8.8-point margin over the strongest baseline (LightNav-0) on HM3D-OVON val_unseen under corrected annotations.

4. Corrected evaluations and two new benchmarks. The authors merge 2,408 raw scene labels into 540 object classes, manually audit and remove defective episodes (about 23% of HM3D val_unseen), and release MP3D-OV and HSSD-OV as new open-vocabulary benchmarks.

Main Findings

• Reaching is not the bottleneck; stopping is. On 3,000 HM3D-OVON val_unseen episodes, the RGB-only policy reached the goal vicinity in 2,288 episodes, succeeded in 1,569, made 183 wrong stops, and timed out 536 times. With ground-truth grounding, the same policy reached 2,818, succeeded 2,679, made 0 wrong stops, and timed out 139 times.

• Grounding must be predicted by a separate model. A single joint model that writes grounding text and then acts reaches 50.7% success on clean val_unseen, below the 63.6% of the policy with no grounding at all. A separate perception model writing the same text raises it to 69.6%. Success on val_seen is 72.7% with no grounding, 64.4% for the joint model, and 77.6% for the full design.

• Both grounding fields matter, semantics most. Under disjoint grounding, the semantic field alone reaches 68.7% on val_unseen and the distance field alone 60.1%; both together reach 69.6%.

• Remaining headroom is perceptual, not policy-related. Replacing predicted grounding with ground truth adds 22.0 points on clean HM3D-OVON val_unseen. Correcting only the distance bin (when both predicted and ground-truth inventories contain the goal) adds 9.0 points to 78.6%; additionally correcting the category list adds 13.0 points to 91.6%, leaving a 4.5-point gap to the expert.

• Perception errors explain most residual failures. Three error types — false positives (goal reported but not in view), wrong distance, and false negatives — account for 69% of failures. Wrong goal visibility causes 43% and wrong goal distance 26%. Per-category success rises with how often perception reports the goal when it is actually in view; small or thin objects such as hangers, bottles, and books are reported least reliably, while large furniture such as beds and couches is reported reliably.

• Official benchmarks systematically under-credit stronger agents. On HM3D-OVON val_unseen, the corrected protocol raises LightNav-0 by 13 points, the authors' system by 16, and the ground-truth-grounding variant by 19 — because better agents make more valid stops that the official name-matching criterion rejects. The frontier-exploration expert is credited in 96% of episodes under the clean protocol but only 72% under the official one.

• Explicit grounding helps even without navigation training. Under the clean protocol, the zero-shot modular baseline VLFM outperforms the fine-tuned end-to-end Uni-NaVid on every task.

• Stopping accuracy once near the goal. Once within 0.25 m, the system succeeds 90% of the time on clean val_unseen versus 80% for LightNav-0.

• Stronger perception transfers without retraining the policy. With goal distance held at ground truth, GPT-6 Astra reaches 81.4% success versus 78.6% for the authors' perception model and 66.0% for the untrained Qwen3-VL-4B; but Astra is roughly nine times slower per step (9.3–16.6 s versus 1.0 s). Its advantage concentrates on small goals and comes from higher goal recall (95.9 versus 77.9) without a matching rise in false positives.

• Path efficiency. On clean OVON val_unseen the system reports 69.6 SR, 36.5 SPL, and 50.7 SPL_FE, leading all baselines on all three metrics across the three open-vocabulary benchmarks.

Methodology in Plain English

The authors start by asking where episodes actually end, not just whether they succeed. They trace trajectories of an end-to-end policy and find that it explores competently and gets close to the goal, then fails to stop. They then rerun the identical policy with the correct answer handed to it — which objects are visible and how far the goal is — and see success jump from 52% to 89%. That controlled comparison isolates grounding as the missing capability.

To supply grounding at test time, they avoid hand-written reasoning traces (costly and noisy) and instead compile it from the 3D scene annotations that already accompany every training frame. The text has two fields: a list of object categories visible in the image, and the goal distance binned into one of five ranges plus "not seen." Because it comes from annotations, every frame is supervised, not just the single STOP at the end of each episode.

The system is split into two models so each learns one job: a perception model that reads recent RGB frames plus the goal word and writes the grounding text, and a policy model that reads frames interleaved with that text and predicts the next action. The policy is trained on ground-truth grounding text, so swapping in predicted text at test time measures the cost of imperfect perception. Distances are binned rather than regressed to make the prediction problem tractable, and the visible-object inventory is capped at 16 categories ordered by image area.

Separately, the authors repair the evaluation. They merge label synonyms into shared object classes, manually audit episodes by comparing rendered frames against semantic annotations, drop the defective ones, and adopt a criterion requiring a stop within 0.25 m of a viewpoint where the goal fills more than 5% of the image. They also build two new open-vocabulary benchmarks on MP3D and HSSD validation scenes.

Why This Matters

Impact on research. The paper reframes a persistent benchmark failure as a perception problem rather than a policy-learning problem, and shows the two can be trained and scaled independently. It also shows that the official evaluation protocol penalizes stronger agents, which means reported method rankings have been partly measuring label-string matching. The released clean protocol and two new benchmarks give the field a more faithful comparison target.

Real-world applications:

• Household service robots asked to fetch a specific item, such as a towel, in a home the robot has never visited — the scenario the paper uses to motivate the task.
• Assistive robotics for locating misplaced personal items (bottles, books, hangers — notably the categories where perception is weakest).
• Warehouse and retail inventory search, where an agent must find a named product class and confirm arrival rather than merely traverse the aisle.
• Inspection and search tasks in unmapped indoor spaces, where knowing when you have arrived at the target is as important as finding it.

Industry relevance. The result that a stronger off-the-shelf VLM plugged into the same policy raises success without retraining is directly actionable for teams building on VLM backbones: perception quality can be upgraded independently of policy training. The latency comparison (1.0 s per step versus 9.3–16.6 s for GPT-6 Astra) frames the practical trade-off between accuracy and real-time control. The observation that LightNav-0 already runs on real robots without adaptation suggests the gains are not simulation-bound.

Future Directions

1. Grounding without 3D annotations. The current supervision requires semantic and geometric labels from annotated 3D scene collections, which caps the training data. Learning grounding from real videos would unlock far larger sources.

2. Better small-object perception. False negatives on small or thin goals (hangers, bottles, books) drive much of the residual gap; the GPT-6 Astra comparison shows the headroom exists but at high latency cost.

3. Alternative grounding architectures. The authors note that a single model with an explicit grounding objective, or an agentic system that reasons about the goal, could supply grounding other than the two-model split — and that their clean benchmarks let future work compare these routes directly.

4. Real-robot validation. All results are in simulation on static indoor scenes; the paper states this limitation explicitly and expects the gains to transfer, but does not demonstrate it.

Target Audience

Researchers and engineers working on embodied AI, vision-language-action models, and robot navigation, particularly those building object-goal navigation systems or benchmarking them. It is also valuable for practitioners who fine-tune VLMs for control and want evidence about which capability behavior cloning fails to teach, and for anyone maintaining or using HM3D, MP3D, or HSSD evaluation splits, since the annotation audit changes how results should be read. Readers need moderate background in imitation learning and multimodal models; the paper's core argument is accessible without it.

Authors’ abstract

Pretrained VLMs bring rich world knowledge and semantic priors to object-goal navigation, and fine-tuning a VLM into a navigation policy has become a common recipe. Despite this progress, success rates remain low, and it is unclear which capability these fine-tuned policies lack. Our diagnosis identifies goal grounding as the dominant failure. These policies explore well, yet often fail to notice that the goal is already in view or to estimate how far away it is. To address this failure, we ground the goal explicitly. Explicit grounding substantially improves success and reaches state-of-the-art results on six benchmarks across three scene collections. We also correct annotation defects in these benchmarks, so that future methods are evaluated more faithfully. Our analysis on the clean benchmarks shows that, with ground-truth grounding, our policy nearly matches the expert that generated its demonstrations, leaving the headroom in perception rather than policy learning. Our work points to goal grounding as a missing piece of object-goal navigation and offers a simple and effective solution to the task.