Failure Prediction at Runtime for Generative Robot Policies

AI summary

Overview

Research area: Robotics / imitation learning — specifically runtime safety monitoring for generative (diffusion- and flow-matching-based) robot policies.

Technical level: Intermediate to Advanced. The paper assumes familiarity with imitation learning, diffusion and flow matching, out-of-distribution (OOD) detection, and conformal prediction.

Scope: The paper introduces FIPER, a framework that predicts task failures of generative imitation-learning policies at runtime without any failure data, by combining an observation-based OOD score with an action-uncertainty score, and evaluates it in five simulation and real-world environments.

What This Paper Is About

Generative imitation-learning policies, such as diffusion and flow matching models, can perform complex long-horizon robot tasks, but they still fail unpredictably when environments shift from their training distribution or when small action errors compound over time. The goal of this paper is to raise an accurate and early alarm when a rollout is heading toward failure, using only a small set of successful rollouts for training and calibration, since deliberately generating failure data would endanger the robot and its surroundings. The authors argue that failure signals appear in two places at once: repeated out-of-distribution observations, and persistent uncertainty in the actions the policy generates.

Key Contributions

1. FIPER, a failure-prediction framework for generative IL policies that needs no failure data. It is calibrated only on a small set of successful in-distribution rollouts and is designed to be task-agnostic.

2. Two complementary failure indicators. (i) RND-OE, which applies random network distillation in the policy's own observation embedding space to measure how far observations deviate from successful patterns, and (ii) ACE (action-chunk entropy), a novel entropy-based score over sampled action chunks that captures uncertainty at the level of behavior modes rather than raw action variance.

3. A conjunction rule with statistical calibration. Both scores are aggregated over sliding time windows, and thresholds are made time-varying and calibrated with conformal prediction for functional data. The combined predictor flags failure only when both scores exceed their thresholds. Proposition 1 gives an upper bound of δ on the probability that FIPER falsely flags a new successful in-distribution rollout.

4. A new evaluation metric, timestep-wise accuracy (TWA), which assigns a true positive the value 1 − DT (detection time) instead of 1, so that early correct predictions are rewarded more than late ones — addressing the fact that accuracy and detection time alone can be gamed trivially.

Main Findings

• FIPER gives the best overall trade-off. Averaged across all five environments, FIPER reaches the highest TWA of 0.65 ± 0.01 and the highest balanced accuracy of 0.78 ± 0.00, with a mean detection time (DT) of 0.30 ± 0.02, which the authors describe as lower than the baselines. An overall TPR of 0.92 is reported.

• Our scores separate benign OOD from real failure. Rollouts are grouped into Success-ID, Success-OOD, Fail-ID and Fail-OOD. The authors expect the ordering Success ID ≤ Success OOD < Fail ID ≤ Fail OOD, and report that RND-OE and ACE show a clear separation between Success OOD and Fail ID, with RND-OE giving a larger Success-OOD to Fail-ID gap than PCA-kmeans and logpZO, and ACE doing the same relative to RND-A and STAC.

• Action-based scores have a smaller Fail-versus-Success gap than observation-based scores. The paper states that failures are harder to detect from policy outputs than from policy inputs.

• Combining the two signals beats using either alone. FIPER has the highest average TWA and accuracy, whereas its individual submodules RND-OE and ACE can predict failures faster on their own (DT of 0.18 ± 0.03 and 0.25 ± 0.00 respectively) but with lower TWA and accuracy (RND-OE 0.59 / 0.67, ACE 0.63 / 0.74).

• ACE outperforms STAC in highly multimodal tasks. In Sorting, Stacking and Pretzel, which involve a high degree of action multimodality, ACE predicts failures much more accurately than STAC.

• Per-environment results for FIPER. Sorting: TWA 0.54, Acc 0.66, DT 0.32. Stacking: TWA 0.62, Acc 0.73, DT 0.28. PushT: TWA 0.55, Acc 0.71, DT 0.32. Pretzel: TWA 0.68 ± 0.03, Acc 0.85 ± 0.00, DT 0.33 ± 0.07. PushChair: TWA 0.83 ± 0.02, Acc 0.96 ± 0.02, DT 0.27 ± 0.00. On PushChair, logpZO reaches TWA 0.78 and Acc 0.92, and on Pretzel ACE reaches TWA 0.75 and Acc 0.82.

• Baseline comparison. Averaged over tasks, PCA-kmeans scores TWA 0.57, Acc 0.61 and DT (0.09); logpZO scores 0.60, 0.69 and 0.35; RND-A scores 0.56, 0.62 and 0.34; STAC scores 0.57, 0.68 and 0.42. Detection times in brackets correspond to TPR or TNR below 0.4 — PCA-kmeans, for instance, achieves a TNR of only 0.24, showing it can largely not distinguish OOD from failure despite a large average score gap.

• Sliding-window aggregation predicts earlier than cumulative aggregation. Figure 5 compares sliding windows (w = 5, 15, 25) against accumulating scores over all past timesteps (w → ∞, as in STAC); the cumulative variant detects failure rollouts very late, mostly because of their greater length.

Methodology in Plain English

The researchers start from the observation that actual failures are usually accompanied by two things happening at once: the robot keeps seeing situations that look unlike anything in its successful experience, and the policy stays unusually unsure about what to do. FIPER builds one detector for each.

The first detector, RND-OE, reuses the policy's own image encoder — a ResNet-18 — and feeds its embeddings to two small networks: a frozen, randomly initialized target network, and a trainable predictor network. The predictor is trained only on successful rollouts to imitate the target; on familiar inputs the two agree, while on novel inputs they diverge, and the size of that divergence becomes the OOD score. Because the encoder is pre-trained and frozen, the RND networks can be trained from a small dataset.

The second detector, ACE, samples a batch of action chunks from the policy at the current observation and measures the entropy of the sampled actions at each prediction timestep, using dimension-wise binning for computational efficiency. This is computed in the Cartesian space of predicted end-effector positions. The authors chose entropy rather than variance because imitation data is often multimodal — a robot may pick object A, B or C, approach from the side or from above — so widely spread actions can still be a sign of confident, well-defined intent. STAC, by contrast, tends to flag the timesteps at which the policy switches behavior mode. Handling each timestep of the chunk separately keeps the required batch size tractable, since the joint chunk space grows exponentially with chunk length.

Both scores are summed over a sliding window of recent policy timesteps. Thresholds are time-varying, because the scores are typically smallest at the start of a rollout, and are computed from a calibration set of successful rollouts using conformal prediction for functional data: the calibration data is split into two disjoint parts to separately estimate a time-varying mean and a band width, giving an upper threshold. A larger δ makes the system more sensitive at the cost of more false alarms. The overall predictor is a logical AND of the two threshold checks, so both must fire.

Evaluation covers three simulation benchmarks (Sorting, Stacking, PushT) and two real-world tasks (Pretzel, PushChair), spanning a Franka arm and a mobile manipulator, with diffusion policies using a temporal U-Net backbone and flow-matching policies using an ACT transformer backbone. OOD scenarios are induced by changing block dimensions and box positions (Sorting), block sizes and target location (Stacking), T-object shape and dimensions (PushT), rope initial configuration and axial rotation (Pretzel), and initial chair pose (PushChair). Baselines are PCA-kmeans, logpZO, STAC and RND-A. For training the learning-based predictors and calibrating thresholds, the authors use M = 50 successful rollouts for the simulation environments and M = 10 for the real-world tasks, average results over 1 − δ ∈ {0.9, 0.91, …, 0.99} and across five seeds, and select the window size w ∈ {1, …, 50} and threshold type by highest TWA across all environments.

Why This Matters

Failure prediction matters because generative IL policies are increasingly deployed in human-centered and safety-critical settings, where even a few seconds of warning can enable a timely intervention, a safe fallback, or a request for a human expert to demonstrate the task. Prior approaches generally either trigger on any novelty (producing false alarms in situations the policy could actually handle) or rely on vision-language models that only raise alarms after errors have already manifested, providing no foresight. FIPER's contribution is showing that combining input-side and output-side evidence, calibrated statistically, gives both fewer false alarms and earlier warnings.

Real-world applications:

• Industrial assembly and sorting cells, where a robot placing or stacking objects could be paused before it makes an irreversible mistake (the Sorting and Stacking benchmarks).
• Deformable-object manipulation such as rope or fabric folding, where failure is hard to detect visually and OOD conditions change the material's bending behavior (the Pretzel task).
• Mobile manipulation and logistics, where a robot pushing furniture or carts must stop safely if its plan is going wrong (the PushChair task).
• Human-robot collaboration and assistive robotics, where early warning triggers a safe fallback or a handoff to a human operator rather than erratic motion near people.

Industry relevance: The method requires no failure data, only a small set of successful rollouts, which is far cheaper and safer to collect than deliberately crashing a robot. It applies on top of existing diffusion and flow-matching policies without retraining them, which makes it attractive as a lightweight runtime safety layer for commercial robot fleets, and it comes with a stated statistical bound (δ) on false alarms that engineers can tune as a design parameter.

Future Directions

• Providing guarantees about failures, not just successes. Proposition 1 bounds the probability of falsely flagging a successful rollout; the authors state that an analogous result about failure detection would require assuming failure data is available, which they deliberately avoid. Closing that gap is an open problem.
• Better and more principled threshold calibration. The authors note there is generally no "best" quantile value for calibration, which is why they average over 1 − δ ∈ {0.9, …, 0.99}, and they compare a one-sided CP band, a CP constant threshold and a simpler time-varying threshold (Proposition 1 covers the first two but not the third).
• Extending beyond current policy and task classes. The paper positions FIPER as task-agnostic and applicable to vision-language-action models, but evaluation covers five specific environments; broader generalization across embodiments, more severe failure modes and even longer-horizon tasks remains to be tested.
• Closing the loop from prediction to recovery. The authors motivate early prediction by the ability to intervene or invoke safe fallbacks, but the paper evaluates prediction, not the downstream intervention or recovery policy.

Target Audience

Robotics and machine learning researchers working on imitation learning, diffusion and flow-matching policies, and vision-language-action models will benefit most, along with safety and reliability engineers who need runtime monitoring for learned controllers. The paper is also relevant to researchers interested in OOD detection, uncertainty quantification and conformal prediction who want to see those tools applied to closed-loop robot control. Readers without a background in generative modeling or conformal prediction should expect a moderate learning curve, particularly for the ACE score and the calibration argument in Proposition 1.

Authors’ abstract

Imitation learning (IL) with generative models, such as diffusion and flow matching, has enabled robots to perform complex, long-horizon tasks. However, distribution shifts from unseen environments or compounding action errors can still cause unpredictable and unsafe behavior, leading to task failure. Early failure prediction during runtime is therefore essential for deploying robots in human-centered and safety-critical environments. We propose FIPER, a general framework for Failure Prediction at Runtime for generative IL policies that does not require failure data. FIPER identifies two key indicators of impending failure: (i) out-of-distribution (OOD) observations detected via random network distillation in the policy's embedding space, and (ii) high uncertainty in generated actions measured by a novel action-chunk entropy score. Both failure prediction scores are calibrated using a small set of successful rollouts via conformal prediction. A failure alarm is triggered when both indicators, aggregated over short time windows, exceed their thresholds. We evaluate FIPER across five simulation and real-world environments involving diverse failure modes. Our results demonstrate that FIPER better distinguishes actual failures from benign OOD situations and predicts failures more accurately and earlier than existing methods. We thus consider this work an important step towards more interpretable and safer generative robot policies. Code, data and videos are available at https://tum-lsy.github.io/fiper_website.