Using VLM Reasoning to Constrain Task and Motion Planning

AI summary

Overview

Research area: Robotics — specifically task and motion planning (TAMP), combining symbolic AI planning, motion planning, and pretrained vision-language models (VLMs).

Technical level: Advanced. The paper assumes familiarity with PDDL, downward refinement, SMT solvers, and stream-based sampling.

Scope: The paper introduces VIZ-COAST (Visual Insight for Z3-based Constraints and Streams), a pipeline in which a VLM inspects an example scene and produces SMT constraints that prevent symbolic task plans from failing at the motion level, evaluated across three simulated manipulation domains.

What This Paper Is About

Task and motion planning splits a long-horizon robotics problem into a discrete symbolic task plan and a continuous motion plan, but a symbolic plan that looks valid on paper can turn out to be geometrically impossible — a failure known as a downward refinement failure. Prior methods discover these failures only by hitting them, then replanning with added constraints, which wastes search effort. VIZ-COAST instead asks a pretrained VLM to reason about an example scene up front and write constraints that block infeasible task plans before search even starts.

Key Contributions

1. A priori constraint inference from VLM visual reasoning. VIZ-COAST uses a Visual Reasoning Module (VRM) to foresee potential downward refinement failures from an image, geometry, PDDL descriptions, and a grasp specification, producing constraints before any planning occurs.
2. Constraints expressed directly as SMT clauses. Constraints are emitted as a Python program using the Z3 API, feeding a SATPlan/Madagascar encoding. Unlike COAST's approach of editing the PDDL description with auxiliary predicates and entities, this adds minimal search overhead and lets the planner keep the original domain description.
3. A closed-loop constraint refinement mechanism. When initial constraints are insufficient, the VRM consumes feedback from downstream planning in a multi-turn conversation — loosening over-restrictive constraints, debugging erroneous constraints, and revising after a cache of refinement failures — so constraints improve across problem instances.
4. Generalization across problem instances. A single prompting session on one example scene produces constraints that carry over to unseen instances within the same domain, demonstrated on 30 problems per domain.

Main Findings

• Shelf: constraints worked on the first attempt. VIZ-COAST was prompted once on the first 3-block problem, and those constraints generalized to all 30 unique problems spanning 3 to 8 blocks, with 5 problems per block count, encountering zero downward refinement failures. COAST-based baselines encountered as many as 10 failures before finding a successful plan on many-block instances.
• Packing: the feedback loop was necessary. On 30 instances (3, 4, 5, and 6 items with problem counts of 7, 8, 8, and 7, using re-prompt threshold K=5), VIZ-COAST had the lowest planning times on 5- and 6-item problems and, per the reported results, 100% success. COAST+FD slowed and failed completely on 6-item problems, and COAST+Z3 failed on all problems. The no-feedback ablation failed on several 5- and 6-item problems and saw refinement failures increase with problem size, while VIZ-COAST's decreased.
• Cooking: the only method with full success. On 30 instances (2, 3, 4, and 5 ingredients with counts of 8, 8, 7, and 7, using K=1), VIZ-COAST was the only method to reach a 100% success rate. The VLM-only baseline solved most problems but with higher planning times, and all other baselines failed completely beyond 2 ingredients — including the no-feedback ablation.
• VLM-only varies sharply by domain. The baseline that asks the VLM to output the task plan directly reached 100% success in Packing but degraded in Shelf and Cooking as complexity grew. The authors attribute this to Packing becoming straightforward once key constraints are recognized, whereas Shelf and Cooking still require long-horizon, non-ergodic search.
• Robustness across VLM providers and reasoning budgets. Across 20 independent trials of 4 sequential 4–7 block Shelf problems with K=1, all evaluated models reached 100% success except Qwen-3.5-Flash-Med at 73%. Average refinement failures ranged from 0.10 ± 0.31 (Gemini3.5-Flash-Low, the fewest) to 7.05 ± 8.87 (Gemini3.5-Flash-Lite-Low). Average time per prompt ranged from 6.28 ± 2.20 s (Gemini3.5-Flash-Lite-Low) to 241.83 ± 108.52 s (Qwen3.5-Flash-Med).
• Reasoning budget effects differ by provider. Among GPT models, increased reasoning corresponded with fewer refinement failures; for Gemini 3.5 Flash, the paper reports a slight opposite trend.
• SMT encoding outperformed PDDL encoding in hard domains. The paper argues COAST's auxiliary PDDL constructions — timesteps and auxiliary logging and blocking predicates — substantially raise symbolic complexity, and that this difference determined whether the planner found a solution or failed entirely in Packing and Cooking.

Methodology in Plain English

The system has three parts.

Task planner: A general-purpose planner built on the Z3 SMT solver using the Madagascar encoding. This lets the researchers express constraints as arbitrary first-order logic expressions through Z3's Python API, without rewriting the PDDL files.

Visual Reasoning Module: The VLM receives five inputs — the PDDL domain and problem, an image of the initial scene, a text-based geometric representation of the scene, a natural-language grasp specification describing how grasps and approach poses are sampled downstream, and an example of a compliant output. It is first asked to name potential refinement issues and the constraints that would fix them in natural language, then asked to encode those constraints as a Python program using Z3's API.

Feedback loop: If planning still fails, the VLM is prompted again in the same multi-turn conversation. Three failure cases are handled: the task planner finds no plan (make constraints less restrictive), the constraints throw errors (show the error message and ask for debugging), or a task plan fails to ground (write the plan and the failing stream to a feedback cache, add a temporary state-action block, and retry). If no plan succeeds after K such attempts, the temporary blocks are removed and the VLM revises the constraints based on the K cached failures. K was chosen per domain from {1, 2, 5} to minimize total refinement failures.

Motion grounding: Downward refinement uses the COAST framework's stream-based sampling, inherited from PDDLStream. The authors state they modify only the task planning level and adopt COAST's stream approach without modification.

Evaluation setup: Three PyBullet simulation domains. Baselines were COAST+FD (original COAST with Fast Downward), COAST+Z3 (COAST adapted to the SMT planner), Incremental (a naive failure-driven state-action blocker similar to IDTMP), VLM-only (VLM outputs the plan directly), and No feedback (VIZ-COAST without the feedback mechanism). Success rate is the proportion of problems solved; refinement failures count replanning events; planning time is cumulative task planning time, excluding motion grounding and VRM prompting time. An instance counted as a failure if it exceeded 100 refinement failures or a 600-second total timeout. Experiments ran on a VM with 12 vCPUs and 62 GB of RAM on an AMD Ryzen Threadripper PRO 5965WX (3.8 GHz base), Ubuntu 24.04 LTS, CPython 3.10.19. VLM calls used Gemini 3.5 Flash with medium reasoning for Shelf and Packing and Gemini 3.6 Flash with high reasoning for Cooking.

Why This Matters

Research impact. The paper reframes the role of VLMs in TAMP: rather than having models generate plans or subgoals, it has them synthesize constraints that augment the symbolic abstraction, leaving search to a classical planner. It also shows that how constraints are encoded — SMT clauses versus auxiliary PDDL constructs — materially changes whether a planner succeeds in difficult domains, and that a closed feedback loop is not merely error recovery but often necessary for arriving at the right insight.

Potential real-world applications.

• Warehouse and bin packing, where the symbolic domain omits container volume and stacking limits.
• Kitchen or laboratory automation with bimanual robots, where lids, covered containers, and reachability impose unwritten geometric rules.
• Shelf and cabinet retrieval in retail or home settings, where approach corridors are restricted to one direction.
• Any deployment where a robot must reuse a constraint set across many similar task instances rather than solving each from scratch.

Industry relevance. The results indicate that commercially available VLM services are fast enough to be practical — constraints were generated in as little as roughly 20 seconds in some configurations — and that performance is robust across multiple providers and reasoning budgets, which lowers the barrier to adopting the approach without committing to a single vendor.

Future Directions

• Reduce reliance on proprietary cloud services. The authors identify dependence on a cloud-based VLM as a source of variance and propose evaluating VIZ-COAST with local or distilled models.
• Transfer constraints across domains. Constraint generalization is currently limited to problems within the same PDDL domain; the authors suggest transferring or composing constraints learned from simpler domains to solve more complex ones.
• Validate outside simulation. All evaluation was conducted in simulation; real-world testing of geometric and perceptual complexity remains open.
• Combine with complementary VLM strategies. The authors suggest integration with methods like VLM-TAMP, which proposes subgoals, and note possible extensions to richer abstractions, dynamic settings, and constraints grounded from natural language instructions.

Target Audience

Robotics and AI planning researchers working on TAMP, task abstraction, or downward refinement; practitioners applying VLMs to robot decision-making; and roboticists evaluating constraint-based planners such as PDDLStream, IDTMP, and COAST. Readers without background in PDDL or SMT-based planning will find the details dense, but the high-level argument — use a VLM's visual common sense to prune infeasible plans before searching — is accessible to anyone working at the intersection of learning and classical planning.

Authors’ abstract

In task and motion planning, high-level task planning is done over an abstraction of the world to enable efficient search in long-horizon robotics problems. However, the feasibility of these task-level plans relies on the downward refinability of the abstraction into continuous motion. When a domain's refinability is poor, task-level plans that appear valid may ultimately fail during motion planning, requiring replanning and resulting in slower overall performance. Prior works mitigate this by encoding refinement issues as constraints to prune infeasible task plans. However, these approaches only add constraints upon refinement failure, expending significant search effort on infeasible branches. We propose VIZ-COAST, a method of leveraging the common-sense spatial reasoning of large pretrained Vision-Language Models to identify issues with downward refinement a priori, bypassing the need to fix these failures during planning. Experiments on three challenging TAMP domains show that our approach is able to extract plausible constraints from images and domain descriptions, drastically reducing planning times and, in some cases, eliminating downward refinement failures altogether, generalizing to a diverse range of instances from the broader domain.