FOM-Nav: Frontier-Object Maps for Object Goal Navigation

AI summary

Overview

• Research area: Embodied AI and robotics — specifically Object Goal Navigation (ObjectNav), the task of finding a named object category in an unseen indoor environment from RGB-D sensing.
• Technical level: Advanced. The paper combines online 3D semantic mapping, a frozen vision-language model (LLaVA-v1.6 7B), instance segmentation (Mask2Former), CLIP text encoders, and A*/fast-marching path planning.
• Scope: The paper proposes FOM-Nav, a modular ObjectNav system built around "Frontier-Object Maps," together with an automatically constructed navigation dataset, and reports benchmark results on MP3D, HM3D v1/v2, HM3D-OVON, plus a real-robot deployment.

What This Paper Is About

A robot asked to "find a chair" in a building it has never seen must both explore efficiently and recognize what it is looking at. Prior systems either keep their memory implicitly inside a neural network's hidden state (which loses track of long-term spatial information) or build explicit maps that carry little semantic detail. FOM-Nav addresses both weaknesses by maintaining an explicit, online map that stores both the boundaries of unexplored space and the objects seen so far, and by letting a vision-language model read that map to decide where to go next.

Key Contributions

1. Online frontier-object mapping. A hybrid representation that is built during navigation and jointly encodes frontiers (borders between explored and unexplored regions), obstacle and exploration maps, and 3D object point clouds carrying geometric, visual (DINO-v2) and textual (CLIP) features.
2. A VLM-based high-level goal prediction module. A LLaVA-style architecture whose input encoders turn frontiers, objects and the robot's past trajectory into tokens; the model first classifies the goal type (frontier or object) and then selects the specific goal by similarity.
3. An automatically constructed navigation dataset. Trajectories are generated with the online frontier-object maps in scanned real-world environments (HM3D and MP3D) using the Habitat simulator, with either ground-truth or automatic segmentations.
4. State-of-the-art results. The paper reports strong performance on ObjectNav benchmarks — including an improvement of 11.8 in success rate and 14.9 in SPL over prior methods on HM3D v2 as stated in the abstract — and demonstrates transfer to a real robot.

Main Findings

• HM3D v2 benchmark: FOM-Nav reaches 75.8 SR and 47.9 SPL, compared with the best reproduced VLFM baseline at 64.0 SR / 33.0 SPL. The paper's abstract and contribution list state improvements of 11.8 in SR and 14.9 in SPL; the results section describes an increase of up to 11.8 and 14.3 points in SR and SPL respectively compared to the best VLFM.
• HM3D v1 (sub, no cross-floor episodes): FOM-Nav obtains 73.0 SR and 52.1 SPL, versus VLFM* at 64.7 SR / 37.6 SPL and PIRLNav* at 71.6 SR / 34.7 SPL.
• MP3D (sub): FOM-Nav reaches 44.6 SR and 23.9 SPL — the highest SPL among the compared methods, while RIM* has a higher SR (51.8). The paper notes that MP3D's ground-truth annotations are often incomplete, so episodes where the target was actually reached may be counted as failures.
• Full benchmarks including cross-floor episodes: FOM-Nav gets 35.0 SR / 18.7 SPL on MP3D and 57.3 SR / 40.8 SPL on HM3D v1. Implicit-memory methods that handle stairs score higher on SR, but FOM-Nav reports the highest SPL among all listed methods.
• Frontier visual features matter most when localized. In the ablation table, removing all image features from frontiers and objects drops performance to 63.7 SR / 38.2 SPL. Global frontier image features with object image features give 70.5 / 42.2, while replacing global with localized frontier features lifts the final model to 75.8 / 47.9.
• Text and path tokens add smaller gains. Object text features and the history path tokens each contribute smaller but positive improvements (for example, removing path tokens gives 74.5 SR / 47.4 SPL versus 75.8 / 47.9 with them).
• Path planner comparison: The Habitat ground-truth greedy geodesic planner gives the best SR (77.1) with 47.6 SPL; the proposed A* + FMM planner gives 75.8 SR / 47.9 SPL, outperforming a PointNav policy (75.1 SR / 45.0 SPL). The authors argue their planner is not tied to a discrete action space and transfers more directly to real robots.
• Training data composition: The final model uses 760k steps over 145 environments (490k steps with ground-truth segmentations plus 270k with automatic ones) and scores 75.8 / 47.9. Using only ground-truth data (490k steps) drops to 61.7 / 37.5, while only automatic data (270k steps) gives 73.6 / 46.5 — the automatic data is smaller but more valuable because it narrows the train/inference gap.
• Environment diversity versus trajectory count: 200k steps spread over 145 environments (70.6 SR / 45.2 SPL) and 200k steps over only 70 environments (70.7 SR / 46.2 SPL) perform similarly and both fall short of the full 760k-step model.
• Segmentation quality is a major bottleneck: With ground-truth segmentation masks the method reaches 86.6% SR and 56.1 SPL; switching to automatic instance segmentation costs 11.0% SR and 8.2 SPL.
• Open-vocabulary (OVON): Using Llama 3.2 1B as the base LLM, the HM-all variant achieves 42.5 SR / 27.8 SPL on val-seen, 36.3 / 23.4 on val-synonyms and 38.1 / 25.7 on val-unseen. The auto-labelled variant scores 37.6 / 24.6, 29.4 / 18.5 and 30.9 / 20.6. The authors note their SR trails Uni-NaVid and MTU3D (which train on large-scale vision-language and navigation data) while their SPL remains competitive.
• Real-robot deployment: The model was run on a wheeled TIAGo++ robot in the authors' laboratory with an Orbbec Femto Mega RGB-D camera, all models on an external desktop with an RTX 4060 GPU, localization from ORB-SLAM3 (RGB-D), and an off-the-shelf Mask2Former trained on ADE20K for segmentation.

Methodology in Plain English

The system has three parts that run in a loop.

1. Build and update the map. Every step, the robot takes an RGB-D image. The depth image is turned into a 3D point cloud of obstacles, downsampled to 5 cm voxels, and flattened onto a 2D obstacle and exploration map centered on the robot, with obstacles inflated by the robot's radius. From this map the system extracts frontiers — the boundary lines between explored and unexplored space — and computes a visual feature for each new one. In parallel, Mask2Former segments objects in the RGB image; each object's pixels are back-projected to a 3D point cloud (filtered with DBSCAN, stored at 5 cm), given a DINO-v2 visual feature, and stored along with a probability distribution over categories. Newly seen objects are merged into existing ones when enough points overlap, and the merge repeats until no further matches occur, so accumulated visual features and category distributions are averaged proportional to point counts.

2. Decide where to go. Each frontier and each object becomes a token for a language model. A frontier token sums MLP encodings of its endpoints and center, its localized visual feature, its geodesic distance (from fast marching), and a learnable type vector. An object token sums encodings of its 3D center, bounding-box corners, distance, accumulated visual feature, CLIP text feature of its most probable category name, and its own type vector. Past path locations also become tokens. These are dropped into a prompt template asking where to go to find the target category, and the frozen LLaVA-v1.6 7B model produces output embeddings. A prediction head then does two things: a linear layer classifies the goal as frontier or object, and a learned similarity match picks the best specific frontier or object.

3. Execute the goal. For a frontier goal, the frontier center is the target. For an object goal, the system computes a valid stopping area next to the object if it is in explored space, or heads toward the nearest frontier otherwise. A fast-marching distance map supplies the heuristic for A*, which produces the trajectory. The planner also maintains a collision map of places where the robot got stuck or failed to move. Each episode begins with a 360-degree rotation.

Training data was generated automatically in HM3D and MP3D scenes: the system runs its own frontier-object maps, picks sub-goals with a hand-designed rule (a visible target wins; otherwise the frontier minimizing average distance to both the robot and the closest ground-truth stopping location), and stores maps, frontiers, path history and labels at every step. Each target benchmark gets its own fine-tuned Mask2Former, trained on 2,000 rendered images per training house. The navigation model is trained for 2 epochs on 4 H100 GPUs (about 2 hours for MP3D, 4 hours for HM3D) with a learning rate of 2×10⁻⁴ and a batch size of 32 per GPU, using a combination of binary cross-entropy and cross-entropy losses with λ = 0.5.

Why This Matters

The work argues that explicit, semantically rich memory — rather than hidden states — is the right substrate for long-horizon ObjectNav, and that a large frozen VLM can be attached to that memory without retraining the VLM itself. Its dataset pipeline also shows that navigation supervision can be generated from scanned real environments at scale, including with noisy automatic segmentation, which is cheaper than human demonstrations.

Real-world applications:

• Home and service robots that must fetch or locate everyday objects such as cushions, phones or glasses in unfamiliar apartments.
• Mobile manipulation platforms, where reaching an object is a prerequisite for grasping and interacting with it.
• Assistive robotics for elderly or disabled users, where a robot locates and brings a requested item.
• Laboratory and facility logistics, demonstrated by the authors' own TIAGo++ deployment in their lab.
• Search and rescue or inspection, where a robot must sweep unknown indoor spaces for a specific object category.

Industry relevance: The system runs on commodity hardware in the real-robot test (an RTX 4060 desktop), it uses off-the-shelf components (ORB-SLAM3, Mask2Former, LLaVA), and its planner is not restricted to discrete action spaces, which makes it more directly portable to continuous-control robots than a learned PointNav policy that would need retraining. The code, model and dataset are released at https://github.com/thomaschabal/fom-nav.

Future Directions

• Multi-floor navigation. The authors identify stairs as the main gap: explicit map-based methods treat stairs as obstacles because wheeled robots cannot climb them, which explains why implicit-memory methods score higher on the full benchmarks. Detecting stairs as traversable connections between floors is named as the fix.
• Stronger instance segmentation. The 11.0% SR and 8.2-point SPL gap between ground-truth and automatic segmentation suggests image-based or video-based segmentation models would yield large gains.
• Better open-vocabulary generalization. Because FOM-Nav introduces new input encoders and output heads, it cannot inherit a pretrained VLM's generalization without training those parts; the authors suggest training on larger-scale vision-language data and more environments.
• More complex semantic navigation. Extending beyond object goals to richer semantic tasks is listed as future work.

Target Audience

Robotics and embodied-AI researchers working on ObjectNav, semantic mapping, or vision-language models for navigation; graduate students who want a worked example of the modular "map, predict, plan" pipeline; and industry engineers building indoor service or mobile-manipulation robots, who will find the real-robot setup, the released dataset, and the practical notes on segmentation quality and planner design most useful.

Authors’ abstract

This paper addresses the Object Goal Navigation problem, where a robot must efficiently find a target object in an unknown environment. Existing implicit memory-based methods struggle with long-term memory retention and planning, while explicit map-based approaches lack rich semantic information. To address these challenges, we propose FOM-Nav, a modular framework that enhances exploration efficiency through Frontier-Object Maps and vision-language models. Our Frontier-Object Maps are built online and jointly encode spatial frontiers and fine-grained object information. Using this representation, a vision-language model performs multimodal scene understanding and high-level goal prediction, which is executed by a low-level planner for efficient trajectory generation. To train FOM-Nav, we automatically construct large-scale navigation datasets from real-world scanned environments. Extensive experiments validate the effectiveness of our model design and constructed dataset. FOM-Nav achieves state-of-the-art performance on the MP3D and HM3D benchmarks, particularly in navigation efficiency metric SPL, and yields promising results on a real robot.