Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models

AI summary

Overview

Research area: Robotics — failure detection for vision-language robotic manipulation. Technical level: Advanced. Scope: This paper introduces an automated pipeline for synthesizing planning and execution failures across simulation and real-robot data, builds the GuardianFail-36k dataset of over 36K examples, and trains Guardian, a multi-view reasoning vision-language model that detects failures and improves downstream manipulation success when used as a verifier.

What This Paper Is About

Robotic manipulation systems still fail in predictable ways — decomposing a task incorrectly, grabbing the wrong object, or missing a grasp — but training models to catch these failures is blocked by a shortage of failure data, since most robot datasets contain only successful demonstrations. The authors solve this by automatically perturbing successful trajectories to create realistic planning and execution failures in both a simulator and a real-robot dataset, then fine-tuning a vision-language model (VLM) on the resulting data to verify plans and subtask executions with explicit step-by-step reasoning.

Key Contributions

1. An automatic, cross-environment failure synthesis framework that generates diverse planning and execution errors with structured reasoning supervision, producing the large-scale robot failure dataset GuardianFail-36k (over 36K examples).
2. Guardian, a multi-view reasoning VLM fine-tuned on GuardianFail-36k for unified planning and execution failure detection, built on InternVL3-8B with InternViT-300M as the visual encoder and Qwen2.5-7B as the language model.
3. State-of-the-art detection performance on three unseen real-world benchmarks (RoboFail, RoboVQA, and the newly introduced UR5-Fail), plus evidence that scaling structured failure reasoning data improves cross-environment generalization.
4. A plug-and-play verifier integration that raises task success rates in both simulation (RLBench) and real-robot deployment (UR5) when combined with the 3D-LOTUS++ policy.

Main Findings

• Synthetic failures match real ones: Running 3D-LOTUS++ on 92 RLBench tasks and manually annotating failure modes for 3 failure episodes per task showed that the designed synthetic failure modes reflect real failures, with similar overall distributions (Figure 4).
• Best on unseen real-world benchmarks: Guardian-8B reaches 0.86 execution and 0.70 planning accuracy on RoboFail, 0.77 execution and 0.89 planning on UR5-Fail, and 0.85 execution on RoboVQA. GPT-4o reaches 0.80/0.67 on RoboFail, 0.77/0.85 on UR5-Fail, and 0.79 on RoboVQA. AHA-13B reports only 0.64 on RoboFail execution (taken from its original paper), and its planning results are not reported.
• Prompting can hurt: Adding Sentinel-Video-QA self-interrogation prompting to GPT-4o degrades accuracy (RoboFail execution stays at 0.80 but planning drops from 0.67 to 0.63; UR5-Fail 0.76/0.62; RoboVQA 0.66), because it constrains the original model's reasoning.
• Simulation-only training transfers poorly: Simulation-only models such as RoboFAC-7B (0.25/0.05 on RoboFail; 0.54/0.02 on UR5-Fail; 0.52 on RoboVQA) and AHA-13B show limited transfer to real-robot benchmarks.
• Same-data comparison isolates architecture and data: Fine-tuning on GuardianFail-36k consistently improves every method. CLIP+MLP trained on it reaches 0.42/0.43 on RoboFail, 0.51/0.51 on UR5-Fail, and 0.52 on RoboVQA. I-Fail-Sense-3B improves to 0.76/0.52, 0.55/0.6, and 0.58. Cosmos-Reason2-8B reaches 0.82/0.70, 0.65/0.83, and 0.77.
• Cross-environment data matters: With InternVL3-8B (binary accuracy averaged over planning and execution), no fine-tuning gives 0.65 on RLBench-Fail, 0.69 on BridgeDataV2-Fail, 0.65 on RoboFail, 0.73 on UR5-Fail, and 0.75 on RoboVQA. RLBench-Fail-only training gives 0.82, 0.70, 0.69, 0.72, 0.66; BridgeDataV2-Fail-only gives 0.65, 0.86, 0.71, 0.68, 0.77; combining both gives 0.85, 0.88, 0.78, 0.83, 0.85.
• Data scaling helps: Performance on in-domain and unseen real-world datasets improves steadily as GuardianFail-36k data increases (Figure 6).
• Reasoning traces improve accuracy: Vanilla training (answer only) gives 0.80 execution and 0.78 planning at 0.68s per sample. Dropout (train on CoT+answer or answer only, test on answer only) gives 0.81 and 0.83 at 0.68s. Thinking (train and test with CoT) gives 0.83 and 0.84 but costs 4.3s — roughly 6 times slower.
• Separate multi-view images beat concatenation: With multi-view input, separated images reach 0.83 execution and 0.84 planning accuracy versus 0.74 and 0.82 for concatenated grid images. Even in single-view, separating start and end images helps execution slightly (0.72 versus 0.69 for concatenation).
• Downstream success improves in simulation: On 10 unseen RLBench tasks with 100 episodes each (20 per seed across 5 seeds), average success rate rises from 0.45 ± 0.03 with no verifier to 0.49 ± 0.02 with off-the-shelf InternVL3-8B, 0.51 ± 0.04 with a fine-tuned verifier without CoT, and 0.54 ± 0.03 with the full Guardian model.
• Downstream success improves on the real robot: On a 6-DoF UR5 over three unseen tasks with 40 episodes each (20 normal, 20 perturbed), Guardian produces 18/20 and 15/20 on placing food, 14/20 and 12/20 on arranging fruits, and 12/20 and 11/20 on stacking cups, versus 15/20 and 4/20, 10/20 and 3/20, and 9/20 and 2/20 without a verifier. Gains are largest under perturbations.
• Failure analysis: Guardian correctly identifies 91% of successful executions and 76% of failed ones, with a 9% false-positive rate. Detection is strongest for no progress (96%), imprecise grasping/pushing (89%), no gripper close (86%), and wrong object manipulated (85%), and weakest for wrong object state or placement (61%). For planning, it reaches 92% accuracy on valid plans and 85% on invalid ones, with contradictory subtasks (100%), wrong order (88%), and missing subtasks (86%) caught reliably, but wrong object manipulated catching only 63%.
• Reasoning traces are faithful: In a human study on 300 samples (100 each from UR5-Fail, RoboFail, and RoboVQA), the final answer matched the conclusion in the analysis step in 100% of cases on all three benchmarks. Errors were dominated by perception rather than reasoning — 75%, 68%, and 93% of errors on UR5-Fail, RoboFail, and RoboVQA respectively.

Methodology in Plain English

The authors start from successful demonstrations rather than collecting failures, which is slow and unsafe. Tasks are broken into subtasks with matching video segments. For planning failures, they alter the task plan only: an LLM (Mistral-Small-24B) swaps in the wrong object or the wrong target location or state, while rule-based edits add wrong ordering, drop subtasks, or insert contradictory ones. For execution failures in simulation, they directly perturb subtask-level actions inside RLBench using four modes: the gripper does not close, the object ends in the wrong state or placement, the wrong object is manipulated, or the grasp or push is imprecise. Because real data cannot be re-simulated, they instead leave BridgeDataV2 trajectories untouched and perturb the paired subtask text (semantic mismatch, including a rule-based preposition swap) or replace the end image with the start image to signal no progress.

Data comes from RLBench (52 tasks from PerAct and GemBench) and BridgeDataV2 with ECoT annotations, which are cleaned with heuristics and Mistral-Small-3.1-24B. Reversing successful executions and swapping start and end images adds around 20% more successful demonstrations. Reasoning traces are generated automatically: object category, spatial location, robot state, and the failure reason are collected, then InternVL3-38B is prompted to produce step-by-step verification traces averaging 118 tokens.

Guardian itself is InternVL3-8B fine-tuned with LoRA while the visual encoder stays frozen, trained on 4 H100 GPUs for 6 hours. Each image passes through the visual encoder separately rather than being concatenated into one grid, and the model produces reasoning text before its success/failure answer. At deployment, Guardian is inserted at each planning and subtask step of an existing pipeline such as 3D-LOTUS++, and the failure reasoning trace is fed back to trigger replanning or re-execution, with up to three retries. Evaluation uses average classification accuracy and is run with zero temperature and greedy decoding.

Why This Matters

• Research impact: The paper argues that the bottleneck in robot failure detection is data, not model capacity, and shows that synthetic failure generation with reasoning supervision scales. It releases GuardianFail-36k, the UR5-Fail benchmark, code, and models, and demonstrates that cross-environment (simulation plus real) data composition is more valuable than simulation-only data used by prior work such as AHA and RoboFAC.
• Real-world applications:
  • Warehouse and logistics robots that need to notice a missed grasp and retry without human intervention.
  • Household and service robots performing long-horizon tasks like opening drawers, microwaves, and grills, where errors compound over time.
  • Industrial arms performing pick-and-place, where catching a wrong-object or wrong-placement error early avoids downstream damage.
  • Human-robot collaboration settings where a robot should flag uncertainty before continuing a multi-step task.
• Industry relevance: Guardian is a plug-and-play verification layer requiring no architectural modification to the underlying manipulation policy, and can reuse existing successful demonstration datasets rather than requiring expensive failure collection. Its use of an open 8B backbone and LoRA fine-tuning makes it practical to deploy without dependence on closed-source models like GPT-4o.

Future Directions

• Incorporating rich failure feedback directly into policy learning, rather than only using it as a verification and replanning signal, as the authors state.
• Adding more spatial reasoning cases to the training data or co-training on large-scale embodied VQA data, since the model's remaining errors come from spatial judgments (for example, failing to notice an object is upside-down or to compare item heights).
• Improving task-relevant object identification and goal-state reasoning, potentially with bounding boxes or segmentation, to address the weak spots in wrong object manipulated (planning, 63%) and wrong object state or placement (execution, 61%).
• Reducing the cost of reasoning: the Thinking strategy is the most accurate but roughly 6 times slower than the Dropout strategy, leaving open where the accuracy-versus-latency trade-off should land.

Target Audience

Robotics researchers working on manipulation robustness, failure detection, and VLM-based embodied reasoning; engineers integrating verification or recovery modules into deployed robot systems; and machine learning practitioners interested in synthetic data generation, chain-of-thought supervision, and multi-view visual input design. Readers need familiarity with vision-language models and robot learning pipelines, so the paper is most useful at an advanced level, though the failure taxonomy and data generation framework are accessible to intermediate readers.

Authors’ abstract

Robust robotic manipulation requires reliable failure detection and recovery. Although recent Vision-Language Models (VLMs) show promise in robot failure detection, their generalization is severely limited by the scarcity and narrow coverage of failure data. To address this bottleneck, we propose an automatic framework for generating diverse robotic planning and execution failures across both simulated and real-world environments. Our approach perturbs successful manipulation trajectories to synthesize failures that reflect realistic failure distributions, and leverages VLMs to produce structured step-by-step reasoning traces. This yields GuardianFail-36k, a large-scale failure reasoning dataset built upon the RLBench simulator and the BridgeDataV2 real-robot dataset. Using GuardianFail-36k, we train Guardian, a multi-view reasoning VLM for unified planning and execution verification. Guardian achieves state-of-the-art performance on three unseen real-world benchmarks: RoboFail, RoboVQA, and our newly introduced UR5-Fail. When integrated with a state-of-the-art LLM-based manipulation policy, it consistently boosts task success rates in both simulation and real-world deployment. These results demonstrate that scaling high-quality failure reasoning data is critical for improving generalization in robotic failure detection. Code, Data, and Models are available at https://www.di.ens.fr/willow/research/guardian/.