LAMP: Implicit Language Map for Robot Navigation

AI summary

Overview

Research area: Robotics — vision-language navigation, map representation, and path planning for mobile robots operating in large-scale environments.

Technical level: Intermediate. The paper assumes familiarity with vision-language models (CLIP), NeRF-style implicit neural fields, topological graphs, and Bayesian uncertainty modeling, but explains its pipeline in a self-contained way.

Scope: The paper proposes LAMP, a navigation framework that stores language features implicitly in a neural field rather than explicitly in a grid or graph, and uses that field, together with a sparse graph, to plan coarse paths and refine them into precise goal poses.

What This Paper Is About

Robots that follow natural language commands ("go to the red oak tree") need a map that connects places to language. Existing language maps store a language vector explicitly at every grid cell or graph node, which either consumes enormous memory or loses the fine detail needed to approach a small object precisely. LAMP instead learns a continuous neural function that maps a camera pose to a language embedding, so embeddings can be generated on demand rather than stored everywhere. The goal is memory-efficient, fine-grained, zero-shot navigation in environments far larger than room scale.

Key Contributions

1. The first implicit language map for fine-grained path generation. LAMP represents language features as a continuous neural field trained on RGB images, and uses that field directly to refine a coarse planned pose into a precise goal pose.
2. A Bayesian formulation of embedding prediction. Each predicted embedding is modeled with a von Mises–Fisher distribution over the unit hypersphere, with a Gamma prior on the concentration parameter, so the network outputs both a mean direction and a confidence value.
3. A score-based graph sampling strategy. Nodes are ranked by a weighted combination of a View Coverage Score, an Uncertainty Score (derived from the Bayesian concentration parameter), and a Semantic Sensitivity Score (the gradient norm of the field with respect to pose), retaining only the most informative nodes.
4. Validation in both simulation and a real building. Experiments run in NVIDIA Isaac Sim on a 1.6 km × 1.8 km scene and on a real multi-floor building, showing memory efficiency and improved goal-reaching accuracy over explicit grid- and node-based baselines.

Main Findings

• Uniform-memory comparison favors LAMP. At roughly equal memory (Grid-based Sparse 0.055 GB, Node-based Sparse 0.057 GB, Ours 0.057 GB), LAMP achieves a success rate of 0.67 on easy objects versus 0.08 for the sparse grid and 0.41 for the sparse node method, and 0.42 on hard objects versus 0.0 for the sparse grid and 0.21 for the sparse node method. Goal-to-distance on easy objects is 6.36 m for LAMP versus 15.50 m and 12.69 m for the baselines.
• Large memory advantages at comparable performance. The paper reports that the grid-based method needs approximately 1,000 times more memory than LAMP to reach comparable performance (dense grid: 56.34 GB at a 0.4 m cell size versus 0.057 GB for LAMP), while the node-based method needs about 70 times more memory (3.962 GB with 2,000,000 nodes).
• The two-stage pipeline drives the gain. The coarse graph stage achieves a success rate of 42–67 percent, after which the fine optimization stage consistently reduces goal distance by about 50 percent. For easy cases the optimization always moves closer to the target; for hard cases it succeeds in 90 percent of cases.
• Inference remains practical. A full navigation query completes in under one second. Reported per-method planning times are 0.8041 s for LAMP, 0.0001 s for the sparse grid, 0.0066 s for the sparse node map, 0.1036 s for the dense grid, and 0.3395 s for the dense node map, measured on an NVIDIA RTX 4090 GPU.
• Qualitative scene analysis explains the failure modes. In the Road scene all methods represent the layout; in the Cube scene the grid-based method fails because a 40 cm grid is too coarse; in the Extinguisher scene the node-based method fails because no node directly observes the goal while LAMP infers it from surrounding context; in the Boxes scene the grid-based method suffers z-axis projection artifacts while the node-based method detects the boxes but cannot plan a precise path.
• Uncertainty and semantic sensitivity improve node sampling. Against Random Node (SR 0.48, SPL 0.44, GDist 4.34), Random Degree Node (0.45, 0.40, 5.31), Random Walk (0.48, 0.43, 4.13), and Forest Fire (0.45, 0.41, 4.79), the full LAMP scoring reaches SR 0.51, SPL 0.47, GDist 4.05. Ablations using only the View Coverage Score give 0.48, 0.43, 4.28; adding the Uncertainty Score gives 0.51, 0.47, 4.36; adding Semantic Sensitivity instead gives 0.48, 0.44, 4.66.
• Real-world goal-reaching precision improves. On the real robot, both the explicit baseline and LAMP reach 90.0 percent success, but LAMP's remaining goal-to-distance is 1.89 m versus 5.19 m for the baseline. Total end-to-end planning time is 3.84 s for LAMP versus 1.82 s for the baseline.
• Goals not directly observed can still be reached. In the real trial, no node in the map directly observes "drinks"; a nearby node is selected and optimization adjusts the robot's pose so the drinks come into view.
• Stated limitations. Effectiveness depends on the vision-language model's accuracy: visually similar objects can cause the wrong node to be selected, and targets with weak or ambiguous appearance can be hard to identify.

Methodology in Plain English

The robot walks through the environment and collects pairs of camera poses and RGB images. A NeRF-style multilayer perceptron with positional encoding and skip connections is trained to map a pose — a 3D position plus a quaternion orientation, concatenated into a 7-dimensional vector — to a 512-dimensional CLIP embedding, so the network learns what language features you would see from any given viewpoint. All CLIP features and network outputs are L2-normalized so they live on the unit hypersphere.

Because CLIP vectors blend the semantics of several objects in a frame, the network is made Bayesian: it outputs both a mean direction and a concentration parameter of a von Mises–Fisher distribution, with a Gamma prior (α = 2, β = 0.5) on the concentration to prevent overconfidence. Training minimizes the negative log-posterior.

The map also contains a topological graph whose nodes store only poses, not embeddings — embeddings are generated on demand. Since searching a huge graph is expensive, nodes are scored with weights w1 = 1 (view coverage), w2 = 1 (uncertainty), and w3 = 0.5 (semantic sensitivity), and only the top-scoring nodes are kept.

At query time, the natural language instruction is encoded into a goal embedding, cosine similarity is computed against the generated embedding of every retained node, and A* finds a coarse path to the best-matching node. From that coarse pose, several candidate poses are sampled nearby, the best is chosen as a guide, and a small correction is optimized with Adam to maximize cosine similarity to the goal embedding minus a distance penalty (λ_dist = 5) that keeps the refined pose near the sampled best and avoids local minima.

Evaluation uses three metrics: success rate (considering only the top 1 percent of predictions, with success defined as ending within 20 m of an object's center), Success weighted by Path Length (SPL), and goal-to-distance for successful trials. Simulation uses the NVIDIA Isaac Sim City Tower Demo 3D Models Pack, spanning about 1.6 km × 1.8 km, with five million pose–image pairs; objects with volume of 1 m³ or more are "easy" and smaller ones are "hard". Real-world experiments use the M2 robot with six cameras across 28 floors of roughly 100 m × 70 m each, collecting 10,000–20,000 pose-image pairs per floor and placing about 20–30 nodes sparsely per floor.

Why This Matters

The paper argues that explicit language maps do not scale: storing a language vector at every grid cell or node becomes prohibitively memory- and compute-heavy as environments grow, while sparsifying them sacrifices the resolution needed for precise goal reaching. LAMP's claim is that decoupling the global structure (a sparse graph of poses) from the semantic detail (a continuous learned field) resolves this tension, and that the same field can be used not just for retrieval but for gradient-based refinement of the final pose. This shifts language mapping from storage toward learned representation, and it demonstrates that a robot can navigate to objects no node ever directly observed.

Real-world applications:

• Service and delivery robots in large indoor spaces such as offices, malls, or hospitals, where a robot must handle ad-hoc language requests without pre-labeled maps.
• Multi-floor building navigation, as demonstrated in the paper's 28-floor real-world trial with a very sparse node set per floor.
• Warehouse and logistics robots that need to locate specific, sometimes small items described in natural language.
• Outdoor or campus-scale autonomy, where the simulated 1.6 km × 1.8 km environment suggests relevance to navigation far beyond room scale.

Industry relevance: memory footprint and inference time are practical deployment constraints on robot hardware, and the paper's claim of a full query in under one second on an RTX 4090 GPU with 0.057 GB of map memory speaks directly to that constraint. The work is published in IEEE Robotics and Automation Letters (DOI 10.1109/LRA.2025.3619820), a venue oriented toward deployable robotics research, and involves authors affiliated with NAVER LABS and Sungkyunkwan University.

Future Directions

• Reducing reliance on the vision-language model's accuracy. The paper states that visually similar objects can cause wrong node selection and that weak or ambiguous target appearance hurts performance, so better VLMs or stronger disambiguation are natural next steps.
• Bridging the time gap between map representations. LAMP's 0.8041 s inference is slower than the baselines' (0.0001 s to 0.3395 s), so speeding up on-demand embedding generation and fine-path optimization is an open practical question.
• Broadening the hard-case results. Hard-object success rate is 0.42 for LAMP (uniform-memory setting), below the 0.47 achieved by the dense node-based method, leaving room to improve on small-object goals.
• Generalizing the uncertainty and sampling machinery. The Uncertainty Score demonstrably improved SR and SPL, but the ablation shows the Semantic Sensitivity Score alone did not improve SR/SPL over view coverage alone (0.48/0.44 versus 0.48/0.43) while it did reduce GDist, so the interplay of these criteria warrants further study.

Target Audience

Robotics researchers and engineers working on vision-language navigation, semantic mapping, and path planning, particularly those interested in implicit neural representations as an alternative to explicit map storage. It also suits practitioners building service, delivery, or warehouse robots that must operate in buildings or campus-scale spaces, and students with intermediate background in CLIP embeddings, NeRF-style fields, and graph search who want a concrete example of combining them into a navigation system.

Authors’ abstract

Recent advances in vision-language models have made zero-shot navigation feasible, enabling robots to follow natural language instructions without requiring labeling. However, existing methods that explicitly store language vectors in grid or node-based maps struggle to scale to large environments due to excessive memory requirements and limited resolution for fine-grained planning. We introduce LAMP (Language Map), a novel neural language field-based navigation framework that learns a continuous, language-driven map and directly leverages it for fine-grained path generation. Unlike prior approaches, our method encodes language features as an implicit neural field rather than storing them explicitly at every location. By combining this implicit representation with a sparse graph, LAMP supports efficient coarse path planning and then performs gradient-based optimization in the learned field to refine poses near the goal. This coarse-to-fine pipeline, language-driven, gradient-guided optimization is the first application of an implicit language map for precise path generation. This refinement is particularly effective at selecting goal regions not directly observed by leveraging semantic similarities in the learned feature space. To further enhance robustness, we adopt a Bayesian framework that models embedding uncertainty via the von Mises-Fisher distribution, thereby improving generalization to unobserved regions. To scale to large environments, LAMP employs a graph sampling strategy that prioritizes spatial coverage and embedding confidence, retaining only the most informative nodes and substantially reducing computational overhead. Our experimental results, both in NVIDIA Isaac Sim and on a real multi-floor building, demonstrate that LAMP outperforms existing explicit methods in both memory efficiency and fine-grained goal-reaching accuracy.