Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols

AI summary

Overview

Research area: Robotics, specifically Vision-Language-Action (VLA) models, Vision-Language Models (VLMs), and robotic manipulation failure diagnosis and recovery.

Technical level: Intermediate. The paper assumes familiarity with imitation learning, VLA/VLM architectures, and fine-tuning, but its core idea (draw symbols on failure videos to teach a model what went wrong) is conceptually accessible.

Scope: The paper introduces ViFailback, a visual-symbol-based framework, dataset (58,128 VQA pairs from 5,202 real-world trajectories), benchmark (ViFailback-Bench Lite and Hard), fine-tuned model (ViFailback-8B), and real-world robot experiments showing recovery from manipulation failures.

What This Paper Is About

VLA models can perform robotic manipulation but cannot reliably identify, explain, or correct their own failures when they encounter out-of-distribution conditions, and existing failure datasets are mostly generated programmatically in simulation, which limits generalization to the real world. The paper's goal is to build an efficient way to annotate real-world failure videos using simple visual symbols (arrows, crosshairs, ON/OFF labels) drawn with a mouse, then use that data to train a model that both diagnoses failures and outputs corrective visual and textual guidance that a robot policy can act on.

Key Contributions

1. ViFailback, a scalable annotation framework that labels real-world robotic failure videos with both textual explanations and visual symbols, covering 7 distinct symbols grouped into Motion, Spatial Relation, and State categories.
2. A dataset of 58,128 high-quality VQA pairs and their corresponding 5,202 real-world manipulation trajectories across 100 tasks, spanning 11 fine-grained question types and 4 failure categories (task planning, gripper 6d-pose, gripper state, human intervention).
3. ViFailback-Bench, a benchmark of 500 trajectories across 22 tasks split into ViFailback-Bench Lite (closed-ended evaluation of failure diagnosis and low-level guidance) and ViFailback-Bench Hard (open-ended evaluation of failure reasoning and CoT-based, high-level guidance).
4. ViFailback-8B, a VLM fine-tuned from Qwen3-VL-8B on the dataset, which outperforms 16 evaluated state-of-the-art models on the benchmark and can generate visual symbols itself for corrective action guidance.

Main Findings

• Baseline models are far from satisfactory at failure reasoning. On ViFailback-Bench, the best closed-ended (Lite) result among baselines was 54.64% (Gemini-2.5-Pro), and the best open-ended (Hard) result was 40.00% (GPT-4o).
• ViFailback-8B substantially outperforms all baselines. It reaches 93.70% average on Lite (a 39.14% gain over the best baseline, Gemini-2.5-Pro) and 72.64% average on Hard (a 39.6% gain over its zero-shot Qwen3-VL-8B counterpart, which scored 33.04%).
• Failure detection is easy; localization and guidance are hard. Baselines score high on failure detection (for example, Gemini-2.5-Pro at 93.13% in Lite) but collapse on failure keyframe localization, failure subtask localization, and low-level guidance; ViFailback-8B scores 98.20% on detection, 92.58% on keyframe localization, 93.48% on subtask localization, 90.79% on failure type identification, 93.15% on low-level avoidance, and 95.93% on low-level correction.
• Open-ended reasoning is the weakest area for baselines. In the Hard benchmark, ViFailback-8B scores 47.95% (low-level avoidance CoT), 65.33% (low-level correction CoT), 83.97% (failure reason), 85.36% (high-level avoidance), and 81.79% (high-level correction). Notably, open-source Qwen3-VL-32B-Instruct surpassed the proprietary models on failure reason with 60.53%.
• Performance scales with data but has not saturated. Training on 1,200, 2,400, 3,600, and the full 4,702 trajectories generally improved accuracy. The visual symbol code generation task reached 38.73% and scaled consistently with data volume. Each additional person-hour of annotation yields an average 0.64% accuracy improvement.
• Visual guidance improves real robot recovery. Integrated with the π0.5 VLA model on an ALOHA dual-arm robot across three unseen tasks (PlaceOne, PlaceTwo, Pull&Place, 21 trials each), the baseline π0.5 with base data reached 50.8% average success and π0.5 with base and symbol data reached 52.4%. Adding ViFailback-8B raised these to 74.6% with the Point-based Motion Control (PMC) method and 73.0% with the Visual Symbols-Following (VSF) method, improvements of 23.8% and 20.6% respectively (the paper reports an average performance improvement of 22.2% in the introduction).
• PMC outperformed VSF on two tasks. The authors attribute this to the π0.5 model lacking sufficient visual symbols-following data to learn low-level guidance action following.
• Visual symbol-following data does not harm base task performance. Comparing the two baselines without ViFailback correction showed no significant difference.

Methodology in Plain English

The team first collected 5,202 real-world ALOHA dual-arm robot manipulation trajectories for 100 tasks, including 657 successful and 4,545 failed trajectories, with 4,995 gathered by human teleoperation and the rest by running the π0.5 VLA model fine-tuned on successful teleoperated samples.

Instead of asking annotators to type long textual explanations, they use a three-stage annotation pipeline. In stage 1, annotators fill in basic semantics such as failure detection, keyframe localization, and subtask localization through simple UI controls, with task descriptions decomposed into subtasks by Qwen2.5-Max. In stage 2, annotators pick corrective action options from predefined categories and drag the mouse to draw visual symbols on keyframes, with each symbol's category, start points, and end points recorded. In stage 3, Qwen3-VL-235B is prompted with all annotations and symbols to generate high-level failure reasoning and guidance, which humans then verify and refine.

The seven symbols encode motion and intent: a colored straight arrow (red for forward-backward, green for left-right, blue for up-down), a semi-circular arrow for end-effector rotation, dual crosshairs linked by a dashed line for aligning two objects, a crosshair for highlighting an object or area, ON/OFF labels for gripper state, a prohibition icon for halting, and a rewind icon for returning to a previous state.

From this annotated data they generated 58,128 VQA pairs, of which 52,418 pairs from 4,702 trajectories across 95 tasks form the training split. They fine-tuned Qwen3-VL-8B with LoRA (rank 32, scaling factor 64, 1 epoch, batch size 1, gradient accumulation 4, learning rate 1e-5, on 4 NVIDIA Hopper GPUs) to produce ViFailback-8B. Evaluation uses accuracy for multiple-choice questions and a GPT-4o-based evaluator scoring semantic similarity, content completeness, and functional equivalence for open-ended questions, with all models set to temperature 0 and a maximum generation length of 2,048 tokens.

For the real-world tests, ViFailback-8B acts as an external supervisor that reads the head camera stream at intervals of 6 action chunks, analyzes the past 5 seconds, and generates CoT-based diagnosis plus visual symbol code when a failure is detected. Two mechanisms deliver the correction to π0.5: the VSF method fine-tunes π0.5 on a visual symbols-following dataset (masking irrelevant head-camera regions using a region of interest expanded by a 50-pixel margin with a 50-pixel minimum, and fully masking the idle arm's wrist camera), while the PMC method uses a low-level motion controller plus GraspNet to estimate grasp poses.

Why This Matters

Impact on research: The paper argues that correcting VLA failures through visual symbols is more direct and effective than language instructions, particularly given the fragile instruction-following abilities of current VLA models, and that visual guidance provides an actionable supervisory signal bridging real-time correction with policy learning. It also positions its benchmark as complementary to existing robotics VLM benchmarks such as Robo2VLM and ManipBench, which assess what to do and how to do it rather than what went wrong and why. Using real rather than simulated failure data directly addresses the sim-to-real gap that constrains prior failure-dataset approaches.

Real-world applications:

• Industrial and warehouse robots that need to detect and recover from failed picks or placements without human intervention.
• Household service robots performing multi-step tasks such as placing objects in drawers or bowls, where a supervisor model can catch mistakes mid-execution.
• Teleoperation and data-collection pipelines, where abundant incidental failure data can be annotated cheaply using visual symbols instead of manual text.
• General robotic fleet monitoring, where an external VLM supervisor watches video streams and intervenes when a policy deviates.

Industry relevance: The framework offers a low-cost annotation path for the large volumes of failure data that already accumulate during policy rollouts, and it demonstrates measurable success-rate gains (up to 74.6% average on three tasks versus roughly 51% without correction) using an 8B-parameter model, which is relevant for teams deploying VLA policies that need reliability guarantees in open-world settings.

Future Directions

• Exploiting the action distribution of failure trajectories. The discussion states that while the framework extracts a learning signal from videos, the actions in failure trajectories contain valuable information that is not yet used.
• Scaling the dataset further. The authors note that performance trends had not reached saturation, indicating that additional data could yield further improvements, particularly for the Hard benchmark tasks that are rare in base models' pre-training data.
• Improving visual symbol following by VLA policies. The gap between PMC and VSF performance suggests better ways to teach policies to follow visual symbols directly.
• Extending to additional failure categories, manipulation skills, and robot platforms beyond the 4 failure types and the ALOHA dual-arm setup used here.

Target Audience

Robotics and embodied AI researchers working on VLA models, imitation learning, and policy robustness; practitioners building failure detection and recovery systems for real robots; and dataset and benchmark designers interested in efficient annotation methods for manipulation data. Readers with a background in robot learning will get the most from the benchmark tables and real-world experiment design, while those newer to the area can follow the visual-symbol annotation concept without deep technical prerequisites.

Authors’ abstract

Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance. Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the dataset, we establish ViFailback-Bench, a benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of Vision-Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation. To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance. Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures. Project Website: https://x1nyuzhou.github.io/vifailback.github.io/