PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memory

AI summary

Overview

Research area: Embodied AI and robot vision — specifically zero-shot object goal navigation (ObjectNav) using multimodal large language models (MLLMs) and large language models (LLMs).

Technical level: Intermediate. The paper is readable for someone familiar with basic navigation metrics and vision-language models, but it assumes comfort with terms like open-vocabulary settings, metric maps, and foundation-model prompting.

Scope: A single paper introducing an RGB-only, map-free, open-vocabulary navigation framework that combines panoramic scene parsing with an LLM-driven decision module backed by a bounded memory queue, evaluated on 200 HM3D validation episodes against 11 baselines.

What This Paper Is About

Household robots asked to "find the sofa" in a building they have never seen must explore efficiently without the depth sensors and prebuilt maps that most existing systems depend on. PanoNav tackles this mapless, RGB-only setting by having a multimodal model look at six panoramic camera views at once and describe the scene, then letting a language model choose where to go next. The core problem it targets is a specific failure mode of prior mapless systems: without any memory of where the robot has already been, the language model keeps sending it back into the same promising-looking room, causing local deadlocks.

Key Contributions

1. PanoNav framework — an RGB-only, mapless, open-vocabulary ObjectNav pipeline that uses multi-view visual input and vision-language reasoning for panoramic scene parsing, requiring no depth sensor and no metric map.
2. Panoramic Scene Parsing — a module that feeds six directional RGB views, each paired with a dot matrix image produced by the Scaffold (SCA) processing method, into an MLLM to produce per-direction local descriptions and a global scene summary.
3. Dynamic Memory-guided Decision-Making — a mechanism built around a Dynamic Bounded Memory Queue that stores recent global summaries so the decision LLM can reason over exploration history and escape local deadlocks.
4. Empirical validation — experiments showing improvements over state-of-the-art baselines in Success Rate (SR) and Success weighted by Path Length (SPL), plus a dedicated Deadlock Avoidance Test and an ablation study. The authors state this is the first RGB-only, mapless open-vocabulary ObjectNav framework to address the local deadlock issue.

Main Findings

• Strongest result among comparable RGB-only mapless methods: PanoNav reaches 43.5% SR and 23.7% SPL on HM3D. Compared with PixNav, which runs under the same navigation settings, this is a 14.76% improvement in SR and a 15.61% improvement in SPL.
• Clear margin over the other RGB-only baseline: ZSON scores 25.5% SR and 12.6% SPL; PixNav scores 37.9% SR and 20.5% SPL.
• Beats several stronger-assumption baselines on SR: PanoNav's 43.5% SR exceeds close-set methods FBE (33.7%), SemExp (37.9%) and Habitat-Web (41.5%), and map-based open-vocabulary methods ESC (39.2%) and VoroNav (42.0%). Baselines with higher SR include OVRL (62.0%), VLFM (52.2%), ImagineNav (53.0%) and L3MVN (50.4%), all of which use richer sensor input or maps.
• Memory is the dominant factor in the Deadlock Avoidance Test: across 10 repeated runs on 5 selected deceptive episodes, memory-guided agents scored 48.0% SR versus 12.0% without memory, 19.2% SPL versus 4.9%, and an Escape Rate of 82.0% versus 32.0%. The paper describes the memory-guided success rate as up to four times greater than the memory-less variant.
• Memory also helps when the agent still fails: mean Distance to Success on failed episodes (DTS(f)) drops from 6.7 with no memory to 4.7 with memory.
• Panorama matters: cutting from six views to a three-view forward-facing setup degrades performance to 19.5% SR and 9.97% SPL.
• Decoupling perception from decision matters: a one-step variant where the MLLM outputs navigation decisions directly, without intermediate textual descriptions, reaches only 35.0% SR and 20.47% SPL.
• Removing memory in the ablation costs performance: 38.5% SR and 22.57% SPL, versus 43.5% and 23.73% for the full framework.
• Qualitative behavior: in trajectory visualizations, the agent reliably reaches targets in straightforward cases and, in deceptive environments, recognizes it is trapped and actively leaves the area to keep searching.

Methodology in Plain English

At every step, the robot takes six RGB photos spaced 60 degrees apart to build a full surround view. Each photo is passed through the Scaffold (SCA) processing method to create a dot matrix version — the plain photo preserves cues about how far away things are, while the dot matrix emphasizes where things sit side by side in the image plane. Both versions go to Qwen-2.5-VL, which returns a written description for each direction plus one overall summary of the surroundings (what objects are around, what kind of room this is). That overall summary is what gets stored in memory.

The memory is a queue of fixed maximum length n, initially empty. New global summaries are appended until the queue fills; after that, the oldest summary drops out each time a new one is added. The decision LLM, DeepSeek-V3, receives the local descriptions and the current global summary, and once the queue is full it also receives the stored history. It then outputs a direction to travel and a flag for whether the target has been found. The actual movement is executed by PixNav, used here purely as the motion controller. The action space is six discrete commands — Stop, MoveAhead, TurnLeft, TurnRight, LookUp, LookDown — with MoveAhead advancing 0.25 meters and turning commands rotating 30 degrees.

Evaluation uses the Habitat simulator on HM3D, which provides high-fidelity reconstructions of 20 complete buildings; the authors randomly selected 200 episodes from its validation set. SR measures the share of episodes where the agent reaches the target within a distance threshold, and SPL measures path efficiency, defined in Equation 5 with per-episode path length relative to the shorter of the agent's path and the ground-truth path, weighted by the binary success indicator. The paper does not report the specific value of the queue length n, the threshold distance for success, compute costs, or inference latency.

Why This Matters

This work pushes object navigation toward cheaper, more deployable hardware assumptions. By proving that an RGB-only, depth-free, map-free pipeline can beat several depth-dependent and map-based systems, it lowers the sensor and localization bar for open-vocabulary navigation. It also isolates a specific, previously under-addressed failure mode — local deadlocks — and shows that a simple bounded memory queue substantially mitigates it, which is a reusable idea beyond this one system.

Real-world applications:

• Home assistance robots that fetch or locate objects on request in unfamiliar houses using only a standard camera.
• Warehouse and retail inventory robots that must find an item in an unmapped or frequently rearranged space.
• Search-and-rescue or inspection robots operating in unknown indoor areas where depth sensors may be unavailable or unreliable.
• Service robots in hotels, hospitals, or offices that navigate with commodity RGB cameras instead of expensive depth and localization stacks.

Industry relevance: The framework depends only on off-the-shelf models (Qwen-2.5-VL and DeepSeek-V3) rather than task-specific training, which keeps deployment and retraining costs low and lets hardware vendors cut sensor bill-of-materials. That combination is attractive for any robotics company trying to ship open-vocabulary navigation without LiDAR or SLAM infrastructure.

Future Directions

• Extend memory beyond text: the authors state that future work will explore leveraging multimodal information to construct memory queues for more robust mapless object navigation.
• Scale and stress-test memory: the queue length n is not reported, so how memory capacity trades off against context length, latency, and deadlock escape performance remains an open question.
• Broaden evaluation: all reported results come from 200 HM3D validation episodes, leaving generalization across other simulators and real robot platforms untested in this paper.
• Address remaining gaps against stronger baselines: methods such as OVRL (62.0% SR), VLFM (52.2%), and ImagineNav (53.0%) still report higher SR than PanoNav's 43.5%, so closing that gap is a natural next target.

Target Audience

Robotics and embodied-AI researchers working on navigation, exploration, or vision-language-action systems; engineers building camera-only service or household robots; and graduate students studying how MLLMs and LLMs can be composed into modular, training-free control pipelines. Those specifically interested in memory design for long-horizon exploration and in benchmark comparisons on HM3D will find the ablation and Deadlock Avoidance Test tables the most directly useful parts.

Authors’ abstract

Zero-shot object navigation (ZSON) in unseen environments remains a challenging problem for household robots, requiring strong perceptual understanding and decision-making capabilities. While recent methods leverage metric maps and Large Language Models (LLMs), they often depend on depth sensors or prebuilt maps, limiting the spatial reasoning ability of Multimodal Large Language Models (MLLMs). Mapless ZSON approaches have emerged to address this, but they typically make short-sighted decisions, leading to local deadlocks due to a lack of historical context. We propose PanoNav, a fully RGB-only, mapless ZSON framework that integrates a Panoramic Scene Parsing module to unlock the spatial parsing potential of MLLMs from panoramic RGB inputs, and a Memory-guided Decision-Making mechanism enhanced by a Dynamic Bounded Memory Queue to incorporate exploration history and avoid local deadlocks. Experiments on the public navigation benchmark show that PanoNav significantly outperforms representative baselines in both SR and SPL metrics.