TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Making

AI summary

Overview

• Research area: Embodied AI / robotics — long-horizon demand-driven visual navigation (a variation of ObjectNav) in simulated home environments.
• Technical level: Advanced. The paper combines large language models, multimodal LLMs, 3D point cloud processing, semantic mapping, and continuous low-level control policies.
• Scope: The paper introduces a new multi-sub-demand navigation benchmark (TP-MDDN) and a modular autonomous decision-making system (AWMSystem) with a Multidimensional Accumulated Semantic Map (MASMap), a Dual-Tempo action generator, and an Adaptive Error Corrector.

What This Paper Is About

Traditional Demand-Driven Navigation (DDN) gives an agent a single need (for example, "I am tired") and asks it to find an object that satisfies that need (for example, a bed or chair). This does not reflect real tasks, where a person has several needs in sequence and a personal preference for how each need should be met. The paper introduces TP-MDDN, a benchmark where a long-horizon instruction contains multiple sub-demands, each paired with an explicit task preference for a specific object category, and it proposes a system to solve that benchmark.

Key Contributions

1. A new benchmark: TP-MDDN (Task-Preferenced Multi-Demand-Driven Navigation). A long-horizon navigation benchmark with multi-sub-demand tasks and explicit task preferences, built with DeepSeekV3 and GPT-4o and manually checked. It contains 200 long-horizon instructions, each with three subtasks, spanning 68 rooms from the ProcTHOR test set.
2. AWMSystem (Autonomous Decision-Making World Model System), composed of three modules — BreakLLM for instruction decomposition, LocateLLM for dynamic goal selection, and StatusMLLM for real-time task monitoring — which performs instruction decomposition, goal selection, and execution monitoring without end-to-end training.
3. MASMap, a lightweight spatial memory design that fuses 3D object detection and 2D semantic mapping (using IoU-based fusion and Ram-Grounded-SAM) with an update-and-prune strategy and a global map structure to reduce storage and redundancy.
4. A Dual-Tempo action generation framework plus an Adaptive Error Corrector. The framework pairs zero-shot planning (A* over affordance and obstacle maps) with a pretrained policy for fine control near targets, and the corrector replans in real time when collisions, out-of-bounds situations, or logic loops occur.

Main Findings

• Success rate advantage: The paper reports that its method (referred to as AWM-Nav in the results tables) achieves a 16% higher success rate than DDN and InstructNav under the TP-MDDN benchmark.
• Main results table: AWM-Nav records STL 20.11, ISR 62.89, SR 32.00, and ISPL 44.19. DDN records 15.50 / 44.67 / 16.00 / 40.66; MO-DDN records 12.11 / 39.78 / 13.33 / 36.25; InstructNav records 9.50 / 42.44 / 16.00 / 39.41.
• Efficiency: Average execution time per long-horizon instruction is 6.82 minutes for AWM-Nav, 1.74 minutes for DDN, 6.79 minutes for MO-DDN, and 88.90 minutes for InstructNav. In the time breakdown, slow-paced actions account for approximately 22 times the duration allocated to fast-paced actions in this method.
• Segmenter ablation: RAM-Grounded-SAM gives the best results (20.11 / 62.89 / 32.00 / 44.19), ahead of YOLO (Ultralytics YOLOv11, 15.56 / 58.00 / 29.33 / 43.69) and GLEE (14.94 / 51.11 / 21.33 / 41.05).
• Reasoning-model ablation: Qwen2-5-VL-72B matches the best configuration (20.11 / 62.89 / 32.00 / 44.19), ahead of GPT-4o (17.51 / 56.44 / 28.67 / 39.95) and Qwen2.5-VL-7B (10.97 / 47.78 / 19.33 / 36.45). The paper notes GPT-4o did not lead to significant improvements and that the number of parameters affects planning performance.
• Adaptive Error Corrector ablation: Removing both re-planning components gives 13.49 / 59.33 / 27.33 / 43.25; keeping only the obstacle-avoidance correction gives 16.86 / 60.44 / 28.00 / 42.20; keeping both gives 20.11 / 62.89 / 32.00 / 44.19.
• StatusMLLM ablation: Without StatusMLLM, results drop to 15.94 / 60.67 / 27.33 / 42.46, versus 20.11 / 62.89 / 32.00 / 44.19 with it.
• Baseline failure modes: InstructNav's planned paths often result in collisions, and its LLM struggles to infer subtask completion from raw action sequences. MO-DDN (whose full implementation is not publicly released, so its policy network was combined with MASMap for coarse exploration) achieves the lowest success rate.
• Task definition detail: A subtask counts as found if the 2D Euclidean distance between agent and object is within a threshold, unlike DDN, which requires the target within the field of vision.

Methodology in Plain English

The system takes a long, multi-part instruction and asks a large language model (BreakLLM) to split it into a list of subtasks, each initially marked as not done. The agent looks around by taking panoramic RGB-D images; an object segmentation model (Ram-Grounded-SAM) labels and segments objects, and depth data is turned into 3D point clouds. Those points are merged over time so that the same physical object seen from different viewpoints becomes one entry, using overlap thresholds of less than 0.25 to create a new object and greater than 0.8 on both overlap metrics to merge, with a Hungarian-algorithm IoU match against the historical 2D map. After merging, the 3D points are discarded and only 2D positions, bounding boxes, and class labels are kept, which saves memory.

When the agent needs a new goal, LocateLLM receives the overall instruction, subtask list, subtask status, target memory, and object memory, and picks the next target object and its 2D location. If the agent has repeatedly failed on the same target (a tolerance of 2 consecutive failures), an auxiliary prompt tells the planner not to choose that object or position again. A different multimodal model, StatusMLLM, checks whether a subtask has actually been completed whenever the low-level policy emits the "Done" action, updating the subtask status list and providing a reason.

Actions are generated at two speeds. The slow tempo plans a route: it computes an affordance map (values are zero where obstacle distance is below 0.25 meters, otherwise a clipped semantic score between 0.1 and 1), runs A* over that map, breaks the path into waypoints, and turns each segment into discrete actions. The fast tempo simply runs a pretrained policy that outputs one of six actions — MoveAhead, RotateRight, RotateLeft, LookUp, LookDown, Done — using forward translations of 0.25 meters and rotations of 30 degrees. An Adaptive Error Corrector re-plans whenever a MoveAhead is likely to collide, using a finer waypoint sampling interval in the initial segment near obstacles. Experiments use AI2-THOR as the simulator and ProcTHOR scenes, with a success distance threshold of 1.5 meters, a maximum of 50 steps, a camera resolution of 300×300, a 90-degree horizontal field of view, and evaluation over 50 tasks per method averaged across three runs on a single NVIDIA H100 80GB GPU.

Why This Matters

The work shifts demand-driven navigation from a single need to a sequence of needs with user preferences, which is closer to how people actually use spaces, and it shows that a modular, training-free combination of foundation models can outperform end-to-end and modular baselines on this setting. It also highlights the cost problem of calling large models at every step and offers a dual-tempo split as a partial answer.

Real-world applications:

• Assistive and service robots in the home that respond to compound requests such as organizing a living space with decorative items or setting up an entertainment corner with seating and media devices.
• Warehouse and retail picking, where an instruction bundles several item requests with preference constraints and the agent must sequence them efficiently across a large space.
• Facility inspection and maintenance, where an agent must complete a multi-stage checklist of objects or locations in unfamiliar layouts.
• Search-and-rescue or caregiving support, where long-horizon goals must be tracked and recovered from when a planned path fails.

Industry relevance: the emphasis on low computational overhead (a single H100 80GB GPU for all experiments), explicit memory management, and real-time error recovery maps directly onto deployment constraints for mobile robots, where repeated LLM inference is prohibitively slow, as the 88.90-minute InstructNav figure illustrates.

Future Directions

• Addressing involuntary mode switching in the Dual-Tempo action generation framework, which the authors list as an outstanding issue.
• Reducing navigation decision errors caused by instruction misjudgment due to over-reliance on pretrained large language models, by training domain-specific language models.
• Optimizing the mode switching of the action generation framework using reinforcement learning.
• Extending the benchmark beyond simulated ProcTHOR scenes to settings that require real continuous control, and clarifying how preference specification should be elicited from users.

Target Audience

Researchers and graduate students in embodied AI, robot learning, and vision-language navigation who want a new long-horizon multi-demand benchmark and a modular, training-free system design. It is also useful for robotics engineers interested in the engineering trade-offs of memory management, large-model call frequency, and real-time error recovery. Readers without background in navigation benchmarks or foundation models will need to consult the cited prior work (DDN, MO-DDN, InstructNav, WMNav) first.

Authors’ abstract

In daily life, people often move through spaces to find objects that meet their needs, posing a key challenge in embodied AI. Traditional Demand-Driven Navigation (DDN) handles one need at a time but does not reflect the complexity of real-world tasks involving multiple needs and personal choices. To bridge this gap, we introduce Task-Preferenced Multi-Demand-Driven Navigation (TP-MDDN), a new benchmark for long-horizon navigation involving multiple sub-demands with explicit task preferences. To solve TP-MDDN, we propose AWMSystem, an autonomous decision-making system composed of three key modules: BreakLLM (instruction decomposition), LocateLLM (goal selection), and StatusMLLM (task monitoring). For spatial memory, we design MASMap, which combines 3D point cloud accumulation with 2D semantic mapping for accurate and efficient environmental understanding. Our Dual-Tempo action generation framework integrates zero-shot planning with policy-based fine control, and is further supported by an Adaptive Error Corrector that handles failure cases in real time. Experiments demonstrate that our approach outperforms state-of-the-art baselines in both perception accuracy and navigation robustness.