Affordance-Guided Coarse-to-Fine Exploration for Base Placement in Open-Vocabulary Mobile Manipulation

AI summary

Overview

• Research area: Robotics — open-vocabulary mobile manipulation (OVMM), specifically where a mobile robot should park its base before manipulating an object. Combines vision-language models (VLMs), affordance reasoning, and occupancy-map geometry.
• Technical level: Intermediate (readers should be comfortable with robotics concepts such as path planning, inverse kinematics, occupancy grids, and VLM prompting).
• Scope: The paper proposes and evaluates a zero-shot framework, Affordance-Guided Coarse-to-Fine Exploration, for choosing base placements in five simulated household manipulation tasks.

What This Paper Is About

In open-vocabulary mobile manipulation, a robot given a natural-language instruction must not only find the right object but also park itself in a spot from which it can actually interact with it. Existing navigation systems typically stop once the robot is merely near the target, and classical planners such as A* and RRT* pick placements using geometry alone, which frequently produces spots that are collision-free but semantically wrong (for example, not facing a drawer, or misaligned with a pot handle). The goal of this paper is a zero-shot method that jointly reasons about task semantics and geometric feasibility, and that can look beyond what a single narrow RGB view shows.

Key Contributions

1. The authors identify base placement as a critical, under-addressed failure source in OVMM, where success requires reasoning about both semantics and geometry under limited perceptual input.
2. They propose a coarse-to-fine strategy that unifies semantic and geometric cues through two cross-modal representations, Affordance RGB and Obstacle Map+, which address limited field of view and overcome the limitations of prior single-focus methods.
3. They build a zero-shot system that requires no task-specific supervision and operates on natural-language instructions, RGB-D images, and an obstacle map, achieving 85% success across five tasks and outperforming classical planners and semantic-only baselines.
4. They provide ablations isolating the contribution of each part of the affordance projection module (12 arrows, the "A" direction arrow, and the projection mechanism itself) and of the semantic-vs-geometric weighting schedule.

Main Findings

• Overall success rate of 85%: Across five OVMM tasks with 20 trials each, the method achieved 17/20, 18/20, 17/20, 16/20, and 17/20 on Throw the Can into Trash, Move Pot Near Red Mug, Put Mug on Shelf, Open Cabinet, and Open Dishwasher respectively.
• Classical planners lag far behind: Object Center + A* reached 47% overall and Object Center + RRT* reached 50%. These methods perform adequately on loosely constrained tasks (e.g., Throw the Can into Trash at 20/20 for A*) but fail where approach angle matters (Open Cabinet at 5/20 for A*, 10/20 for RRT*).
• Affordance keypoints help only modestly: Affordance Point + A* reached 58% and Affordance Point + RRT* reached 61%. Approaching from the side of a handle, or mislocalizing to occluded/non-manipulable regions, still caused failures because these methods lack directional and reachability reasoning.
• VLM-only prompting performs worst: Pivot with RGB only scored 26%; the multimodal variant Pivot (Obstacle Map+, Affordance RGB) scored 23%. Both propose placements at incorrect distances or physically inaccessible locations.
• The coarse-to-fine schedule is the best weighting: Fixed α = 0 (semantics only) gave 43%; α = 0.5 gave 76%; α = 1 (geometry only) gave 79%; the sigmoid schedule used in the paper gave 85%.
• Projection matters more than extra arrows: Removing the 12 auxiliary arrows dropped performance from 85% to 80%; removing the main "A" direction arrow (and its fan-shaped region) dropped it to 62%; disabling projection entirely dropped it to 48%.
• VLMs cannot ground semantics spatially on their own: The authors conclude from the projection ablation that current VLMs have limited ability to convert semantic understanding into spatially grounded reasoning, which is why the explicit projection mechanism is needed.

Methodology in Plain English

The pipeline has three stages. First, navigation: a large language model (GPT-4) parses the instruction into sub-instructions, and a path planner moves the robot to a coarse waypoint typically within 1.5 m of the object, facing it. Second, base placement selection: the robot picks an optimized parking spot. Third, manipulation: a predefined primitive such as pick, place, or open is executed.

The paper's contribution sits in the second stage. The robot maintains a global 2D occupancy grid map (200 × 200 cells at 0.05 m per cell) and derives a local egocentric map at runtime. It builds two linked pictures:

• Affordance RGB — the camera image overlaid with 12 colored directional arrows spaced every 30° around the object, plus one arrow labeled "A" marking the direction the VLM thinks is appropriate.
• Obstacle Map+ — a top-down map showing the segmented object footprint, the robot's current base position, a fan-shaped affordance region spanning ±60° around direction "A", and the same 12 arrows in matching colors.

Because the colors match across both images, the VLM can connect what it sees in the RGB view with the spatial layout on the map, which lets it reason about areas outside the camera's field of view.

For the search itself, the system first finds an affordance keypoint g (e.g., a handle): DINOv2 features are extracted, Grounded SAM segments the target object, k-means clustering (k = 20, cosine similarity, pruning candidates closer than 0.08 m) produces candidate keypoints, and GPT-4o picks the one most semantically aligned with the sub-instruction.

Candidate base placements are then sampled from a truncated Gaussian centered on g, restricted to collision-free locations at least 0.4 m from obstacles. Each candidate gets a score that multiplies a geometric term (preference for a target distance from g) by a semantic term (preference for the current semantic center), weighted by a time-dependent coefficient α_t that follows a sigmoid schedule. Early iterations emphasize semantics, so the search lands in task-relevant regions; later iterations emphasize geometry, so it converges to physically executable spots. After sampling, candidates are projected onto Obstacle Map+ with unique indices and sent, together with Affordance RGB and the sub-instruction, to the VLM for ranking. The VLM's top-k picks update the semantic center. At the final iteration, the top-5 VLM-ranked candidates are taken, the two furthest from their mean are removed, and the remaining three are averaged to give the final placement.

Why This Matters

Impact on research. The paper reframes base placement as a joint semantic-and-geometric inference problem rather than a pure path-planning problem, and provides evidence that VLM semantics alone are insufficient without an explicit mechanism for spatial grounding. It also extends OVMM evaluation beyond pick-and-place to articulated tasks such as opening cabinets and dishwashers.

Real-world applications:

• Home service robots that must open cabinets, dishwashers, or drawers — tasks where the approach angle determines success.
• Kitchen and household assistants that pick up and deliver objects such as mugs, pots, and cans with specific handle orientations.
• Warehouse or logistics robots that need to position themselves relative to shelves and containers before reaching or placing items.
• Assistive robotics for people with limited mobility, where grasp alignment and reachability from a parked base position are safety-relevant.

Industry relevance. The method is zero-shot and needs no task-specific training, using off-the-shelf components (Grounded SAM, GPT-4/GPT-4o, DINOv2) plus a standard occupancy grid. That makes it comparatively easy to bolt onto existing navigation stacks, which typically already produce maps and coarse waypoints. It also highlights a practical limitation for anyone deploying VLMs onboard: raw RGB prompting produced the weakest results in this study.

Future Directions

• Incorporate arm trajectory feasibility into the optimization, so that end-to-end motion plans are guaranteed executable rather than only the base placement being feasible. The authors note the arm may still collide with obstacles in confined or cluttered environments.
• Improve geometric precision of predicted placements, which the authors acknowledge can be limited compared to geometry-based methods in tasks requiring accurate distance estimation.
• Reduce residual VLM reasoning errors, which the authors expect to diminish as VLMs improve.
• Validate beyond simulation. All experiments reported use NVIDIA Isaac Sim with the TIAGo++ platform; real-robot results and runtime/compute costs are not reported in the provided content.

Target Audience

Robotics researchers and graduate students working on mobile manipulation, task-and-motion planning, and VLM-based robot reasoning; engineers integrating foundation models into navigation and manipulation stacks; and practitioners who need to understand why "navigate to the object" is not the same as "navigate to a spot from which the object can actually be manipulated." Readers without background in robotics would find the terminology dense, though the core idea — park where the task, not just the obstacle map, says you should — is accessible.

Authors’ abstract

In open-vocabulary mobile manipulation (OVMM), task success often hinges on the selection of an appropriate base placement for the robot. Existing approaches typically navigate to proximity-based regions without considering affordances, resulting in frequent manipulation failures. We propose Affordance-Guided Coarse-to-Fine Exploration, a zero-shot framework for base placement that integrates semantic understanding from vision-language models (VLMs) with geometric feasibility through an iterative optimization process. Our method constructs cross-modal representations, namely Affordance RGB and Obstacle Map+, to align semantics with spatial context. This enables reasoning that extends beyond the egocentric limitations of RGB perception. To ensure interaction is guided by task-relevant affordances, we leverage coarse semantic priors from VLMs to guide the search toward task-relevant regions and refine placements with geometric constraints, thereby reducing the risk of convergence to local optima. Evaluated on five diverse open-vocabulary mobile manipulation tasks, our system achieves an 85% success rate, significantly outperforming classical geometric planners and VLM-based methods. This demonstrates the promise of affordance-aware and multimodal reasoning for generalizable, instruction-conditioned planning in OVMM.