FLARE: A Failure-Aware Framework for Autonomous Correction and Recovery in Visual-Language Robotic Manipulation

AI summary

Overview

Research area: Robotics and embodied AI — specifically Vision-Language-Action (VLA) models for contact-rich robotic manipulation, and failure recovery / self-correction for learned policies.

Technical level: Advanced. The paper assumes familiarity with imitation learning, action-chunking policies, distribution shift (ID vs. OOD), LoRA adapters, and multimodal LLM prompting.

Scope: The paper proposes FLARE, a "Retry/Reset" framework that augments demonstration data and orchestrates a library of VLA adapters so a robot can recover from both pose-level and environment-level execution failures, evaluated on 9 RoboMimic simulation tasks plus two real-world tasks.

What This Paper Is About

Current VLA models are trained on success-biased, "trajectory-monotonic" human demonstrations, so they learn a spurious correlation between the robot's own pose and task progress rather than truly tracking the environment state. As a result, a small perturbation, a missed grasp, or a dropped object can cause apparently irreversible failure, because the policy has never seen recovery behavior. FLARE's goal is to make VLAs resilient by giving them two learned capabilities: an innate "Retry" ability for in-distribution (ID) errors, and a set of object-centric "Reset" skills for out-of-distribution (OOD) errors, arbitrated at inference by an online multimodal LLM monitor.

Key Contributions

1. A unified Retry/Reset paradigm grounded in an ID/OOD error taxonomy. The paper formalizes two failure types: ID errors, where the environment state is still valid but the robot pose is novel, versus OOD errors, where the environment state itself can no longer be salvaged by the task policy. It reframes robotic autonomy as resilience rather than perfection.

2. A perturbation-bridging augmentation strategy. Random perturbation segments are injected between task subtasks, followed by bridging segments that return the robot to a valid pose for the next subtask. Training on bridging-to-task subsequences explicitly decouples robot pose from environment state and installs a built-in "retry" capability.

3. An MLLM-driven dual-loop system. Gemini-2.5-Pro serves both as an offline failure analyst — parsing execution videos into structured JSON with error_type, reset_target, error_group, and failure_timestamp — and as an online monitor that arbitrates between task execution and reset skills during deployment.

4. A modular LoRA-based expert policy library. Rather than one monolithic model, a main task adapter and one reset adapter per object-centric reset skill are trained on a shared pre-trained VLA backbone, avoiding conflicting gradients and allowing new recovery skills to be added without disturbing existing ones.

Main Findings

• State-of-the-art simulation results: FLARE achieves a mean success rate of 84.0% across 9 RoboMimic manipulation tasks, compared with 57.8% for Phoenix, 78.9% for Phoenix-Human, 72.2% for the π0.5 backbone, and 38.0% for OpenVLA.

• Gains over the backbone: The method improves on π0.5 alone by 11.8% on average, and outperforms Phoenix-Human, which is supplied with manual human correction and serves as an upper bound for self-reflection methods.

• Best on 8 of 9 tasks: FLARE leads on 8 of the 9 tasks. On the remaining task, Threading_D0 (72%), it does not obtain the best result but still improves substantially over π0.5 (42%) and outperforms Phoenix (68%), while Phoenix-Human reaches 100% on that task.

• Larger gains on harder randomization: The 'D' suffix denotes the range of object randomization (e.g., Coffee_D0 vs. Coffee_D1). Improvements are generally more pronounced on D1 versions than D0 versions, attributed to the decoupling effect of perturbation-and-bridging.

• Reset not applicable for Stack: For the Stack task (stacking two cubes of different colors), the reset setting is not applicable because there are almost no non-retryable errors during execution; FLARE still achieves comparable performance where Phoenix-Human and π0.5 reach 100%.

• Ablation on reset skills: Removing reset skills ("Ours w/o Reset") reduces the average success rate by 3.5% across Coffee and ThreePieceAssembly. The human-instruction variant ("Ours-Oracle") achieves an additional 7% improvement, suggesting stronger multimodal LLMs could further help monitoring.

• Reset skill success depends on the object: Reset success is 84% (coffee machine lid) and 24% (coffee pod) for Coffee, and 88% (T-shaped block) and 20% (U-shaped block) for ThreePieceAssembly. Demonstration-generation efficiency follows the same trend: 83.7% and 11.6% for Coffee; 48.6% and 5.9% for ThreePieceAssembly.

• MLLM failure identification: On 50 manually labeled videos, Gemini-2.5-Pro reaches 88% (Coffee) and 96% (ThreePieceAssembly) for reset/retry classification, 88% and 78% for reset-object identification, and 78% and 66% for timestamp identification.

• Hyperparameter sensitivity: On Coffee_D1, best performance is achieved with rotation r = 30° and translation t = 0.7. Larger rotations and translations produce higher-variance demonstrations and higher task success but lower generation efficiency; excessively large perturbations eventually degrade performance.

• Real-world validation: On a Piper arm with RealSense D435i top/wrist views over 40 trials, FLARE reaches 75.0% vs. π0.5's 62.5% on Stack Three Blocks and 55.0% vs. 45.0% on Insert U-shaped Block, using only visual sensors (object poses from Any6D) rather than privileged simulator state.

Methodology in Plain English

The authors start from an observation about the data, not the architecture. Human demos are expensive, so datasets are small and monotonous: the demonstrator always grasps the cup the same way, always approaches from the same angle. A policy trained on this learns "my arm is in this pose, therefore the task is at this stage" instead of "the cup is in the gripper, therefore the task is at this stage." That misfires the moment the robot's pose changes.

To fix the pose-level problem, they build new trajectories by stitching together demonstrated subtask segments, then deliberately insert a random perturbation — moving the arm by up to 45 degrees of rotation and 0.5 meters of translation — and a bridging motion that brings it back to a valid pose for the next subtask. The perturbation actions themselves are not used as training targets; only the bridging-plus-task sequences are. This forces the policy to practice "from wherever I am now, do the right next thing," which is exactly the retry skill.

For catastrophic failures the environment itself is broken (a toppled cup), so no amount of retrying helps. Here the authors let Gemini-2.5-Pro watch failure videos offline and label each one, identifying the error type, which object needs resetting, and when in the video the failure happened. That labeling tells them which object-centric reset skills to collect — a small number (20) of human demonstrations per skill, starting from the mined failure state — which are then expanded to 500 demonstrations using the same perturbation-and-bridging trick. They deliberately scope out truly unrecoverable cases, such as a toppled coffee machine that is too large for the gripper or an object that falls out of reach.

Rather than train one model on everything, they train the main task policy and each reset skill as separate LoRA adapters on the same pre-trained π0.5 backbone. At deployment the online MLLM monitor watches execution: ID errors are left to the VLA's innate retry robustness, while OOD errors trigger a prompt switch to the relevant reset adapter; once the environment is restored, the main task adapter is reloaded and the task resumes.

Why This Matters

Research impact. The paper reframes VLA brittleness as a data-regime problem rather than a purely architectural one, offering a concrete taxonomy (ID vs. OOD) that makes "recovery" a learnable capability instead of an emergent hope. It also demonstrates a useful division of labor: a large multimodal LLM handles high-level semantic failure diagnosis while small, specialized VLA adapters handle low-level control — a pattern that may generalize well beyond manipulation.

Real-world applications:

• Household and service robots that must keep working after dropping a plate or knocking over a container, rather than stalling until a human intervenes.
• Warehouse and logistics pick-and-place, where a missed grasp or a toppled item on a shelf currently halts the line; a reset skill could recover autonomously.
• Laboratory and manufacturing automation involving contact-rich assembly (the paper's Insert U-shaped Block task is representative) where misalignment is common and re-grasping is routine.
• Long-horizon multi-step tasks such as coffee preparation, where a stuck pod or a closed lid requires a targeted corrective action before the recipe can continue.

Industry relevance. The framework's use of LoRA adapters means new recovery skills can be added incrementally without retraining or disturbing existing policies, which maps well onto how deployed robot fleets are maintained. The real-world experiments showing that approximate failure reconstruction and vision-only pose estimation (Any6D) are sufficient also reduce the dependence on privileged simulator state, an important practical constraint.

Future Directions

• Stronger monitoring models. The Ours-Oracle variant, which substitutes human instructions for MLLM-generated ones, gains an additional 7%, indicating headroom in the online failure monitor and the offline analyst.

• Dexterous recovery maneuvers. The large gap between resetting "easy" objects (84%, 88%) and "hard" ones (24%, 20%) points to a need for in-hand pose refinement and better grasping capability; the authors explicitly name this as a hardware limitation.

• Scaling the reset skill library. The paper scopes out unrecoverable situations such as oversized objects or objects outside the workspace; extending coverage to these cases and to larger libraries of reset skills is left open.

• Generalization beyond the studied task suite. The evaluation covers 9 RoboMimic tasks and two real-world tasks; how the Retry/Reset paradigm transfers to more varied embodiments, environments, and open-world settings is not established by the reported results.

Target Audience

Robotics and embodied-AI researchers working on VLA models, imitation learning, and manipulation policy robustness will get the most from this paper. It is also relevant to practitioners deploying learned manipulation policies in industrial or service settings, and to researchers interested in MLLM-based agentic supervision and modular adapter-based policy composition. Readers without a background in imitation learning, action-chunking policies, or distribution shift will find the methodology sections dense, though the ID/OOD framing itself is accessible.

Authors’ abstract

Vision-Language-Action Models~(VLAs) have demonstrated significant promise in generalizing to complex, long-horizon robotic manipulation tasks. However, their performance remains brittle, as they are typically trained on trajectory-monotonic, failure-free demonstrations. This reliance on ``perfect" data leaves them unable to recover from common execution errors, such as a missed grasp, a dropped object, or an unexpected collision. In this paper, we propose FLARE, a novel framework that endows VLAs with robust error recovery capabilities through a ``Retry" and ``Reset" paradigm. First, we introduce a ``Retry" mechanism by injecting perturbation and bridging segments that decouple robot pose from environment state into demonstrations, enabling the policy to autonomously handle execution deviations. Second, to address critical, state-breaking (OOD) failures, we introduce a ``Reset" pipeline. We leverage an MLLM for offline failure analysis to automatically identify OOD states from execution videos. This analysis enables the efficient, targeted collection of a small library of object-centric ``Reset" skills, which are trained to restore the environment to a task-valid state. Our full framework integrates these learned policies. At inference, an online MLLM monitor arbitrates between task execution and ``Reset" skills. Experiments on challenging, contact-rich manipulation tasks show our approach significantly improves task success and robustness.