PROVIA: Procedure State Tracking for Online Mistake Detection in Egocentric Videos

AI summary

Overview

Research area: Computer vision, specifically online (streaming) procedural mistake detection in egocentric video, combining causal video segmentation, Bayesian filtering, and sequential change detection.

Technical level: Advanced. The paper assumes familiarity with point processes, Bayesian filtering over latent discrete state spaces, probabilistic automata, and sequential hypothesis testing; the evaluation critique also requires comfort with benchmark construction.

Scope: The paper argues that the dominant evaluation protocol for online mistake detection is solvable without looking at any video, then proposes a two-track online model (PROVIA) that separates what was actually performed from the progress a procedure has correctly accepted, and validates it on four egocentric benchmarks under a stated false-alarm budget.

What This Paper Is About

An assistant watching someone cook, assemble, or disassemble something from a head-mounted camera should notice a mistake as the video streams in, before the next step starts, and keep functioning when the person repairs the error. The paper shows that the standard benchmark protocol for this task — cut every recording at its first mistake and label the method's last predicted segment as the mistake — can be scored perfectly by a rule that ignores the video entirely, because a fixed-time pair of events wins by construction. PROVIA instead evaluates on complete trials, where mistakes and recoveries occur naturally, and models the procedure as a latent state that must be inferred online, distinguishing the observed record of what each actor did from the accepted progress that the procedure has correctly advanced.

Key Contributions

1. A diagnosis of the first-mistake protocol. The authors demonstrate with four measurements that the protocol used by prior online methods (PREGO, DTGL) admits a perception-free solution with average F1 of 1.000, that a position prior alone reaches the level of published results, that 73 percent of annotated mistakes are never tested, and that the splits are not participant-disjoint. They replace it with alarm-level evaluation on complete trials under a validation false-alarm budget, with timing-only controls.

2. A two-track online model. PROVIA keeps a factual state — a learned running summary of the steps each actor was observed to perform, mistakes included — apart from the accepted progress, an exact posterior over the state of a procedure automaton and over each actor's execution status (correct, mistake, correction, or insertion). Procedure-state transitions occur only in the correct branch, so a correction is read against what was actually done rather than against progress the mistake did not make.

3. Automaton induction without labels or validation data. Correct demonstrations are decoded into actor-tagged step sequences and merged into a probabilistic finite automaton by Bayesian state merging, using an integrated-evidence criterion rather than a similarity threshold or edit cost. Per-step visual effects are modelled as a Gaussian random-effects law for the change in each actor's descriptor.

4. State-of-the-art controlled results and ablations. One filter and one optimization rule produce the highest step-level AP and AUROC among controlled online baselines across four benchmarks, and ablations show that discarding the factual state at inference costs accuracy on all three single-actor benchmarks.

Main Findings

• Best step-level results on all four benchmarks. PROVIA reaches AP/AUROC of .364/.554 on CaptainCook4D, .153/.625 on IndustReal, .131/.670 on HoloAssist, and .147/.567 on IMPACT-ego, against the strongest controlled baseline per benchmark (for example .346/.527 on CaptainCook4D and .111/.633 on HoloAssist). The largest margin is on HoloAssist, the benchmark with the most steps per execution.

• Alarm-level gains under a false-alarm budget. At 0.1 false alarms per minute, PROVIA recalls .154 of mistakes on CaptainCook4D against .128 for the best baseline, and .034 against .015 on HoloAssist, where it leads at every tested budget. Thresholds are frozen on validation, and the timing-only controls establish the floor a method must beat to be credited with detection.

• The factual state matters. Resetting the running record before every segment costs about a seventh of AP on HoloAssist and a tenth on IndustReal and CaptainCook4D. Retraining a matched architecture without it leaves the model below the full version on HoloAssist and CaptainCook4D, so the gain is not explained by parameter count alone.

• The status transition is the second dependence on the past. Removing the learned prior over how one execution status follows the last costs AP everywhere, and it is the dominant component on CaptainCook4D, where only 3.9 error-free executions per recipe exist and the induced automaton is a shallow prefix tree.

• The procedure-state belief contributes little at the step level. After merging, states average only 1.01 to 1.18 admissible next steps, and merging compresses the demonstration prefix tree by just 1 to 14 percent. A belief over a near-linear automaton carries little beyond position; the completion gate, which ties the reported probability to evidence the step has ended, is the more useful component and is worth most on IndustReal.

• Bimanual coverage limits IMPACT-ego. Steps there last one to two seconds and the two hands interleave. Decoding segments only at events leaves 57 percent of annotated steps holding a decision; keeping an open step on every hand raises coverage to 86 percent and AP from .129 to .147.

• Real-time operation. The full pipeline runs at 58 to 70 frames per second on one A100, with 0.71 to 0.78 million trainable parameters and no per-dataset head, task embedding, or language model.

Methodology in Plain English

The system assumes that a correct demonstration of the task already exists, and it learns the shape of the procedure rather than the mistakes. Everything starts with a segmentation model that watches the stream causally, one actor at a time, and predicts when the current step finishes. It is a marked point process: an event type fires when a step completes, and a "mark" names the step. Because it is causal, it can only look backwards. It emits both a hazard — the probability the step ends now — and soft distributions over the verb, noun, and tool of the step.

From the correct demonstrations, the authors build a map of how the procedure can unfold: they decode all the correct videos into sequences of steps tagged by actor, lay them out as a tree of histories, and then merge histories that predict the same continuation, using a Bayesian evidence comparison instead of a hand-tuned similarity threshold. The result is a compact probabilistic automaton, plus, for each step, a learned picture of how the actor's visual descriptor should change while performing it.

At query time the model maintains two separate beliefs. The factual state is a recurrent summary updated after every segment, regardless of whether that segment was right or wrong, so it records what the person actually did. The accepted progress is an exact Bayesian filter over the pair (procedure state, execution status of each actor). The key design choice is that only the correct-status branch moves the procedure forward; mistake and correction branches keep the source procedure state. This means a later step, and especially a repair, is judged against the mess that was actually made rather than against the plan.

Finally, per-frame mistake probabilities are converted into alarms. The system computes the ratio of posterior odds to prior odds for each completed segment and accumulates it with a Shiryaev–Roberts statistic, which sums the evidence over all plausible onset points. An alarm fires when that statistic crosses a threshold, and only the statistic resets afterwards. The threshold is chosen on validation to hit a stated false-alarm rate per minute of correct operation, which is what an assistant interrupting a human actually costs.

Why This Matters

Research impact. The paper exposes a protocol-level flaw that let perception-free baselines score near-perfectly and made a large fraction of published online mistake-detection results uninterpretable. By formalizing an alarm-level protocol with false-alarm budgets and timing-only controls, it gives the field a benchmark that cannot be gamed by knowing when mistakes tend to occur, and it reframes the problem as online inference over procedure state rather than frame-by-frame classification. The two-track separation of "what was done" from "what was accepted" is a reusable idea for any streaming task where a failure does not erase history.

Real-world applications:

• Assistive cooking or recipe guidance that warns about a wrong ingredient or order before the dish is spoiled, while tolerating the cook's own corrections.
• Industrial and manufacturing assembly support, where an error in seating a part or choosing a fastener is costly and the worker may silently fix it.
• Augmented-reality or robotic co-workers that hold back the next part or highlight the right component, using PROVIA's alarms to decide when to intervene.
• Procedural training and skill assessment, where complete trials with natural mistakes and recoveries need to be scored, not truncated at the first error.

Industry relevance. An assistant that raises a false alarm interrupts a worker, so a method that reports accuracy at a stated false-alarm rate maps directly onto deployable product requirements. PROVIA's low parameter count (under a million trainable parameters), modest compute (58 to 70 FPS on a single A100), and single training recipe across four domains make it far easier to integrate into embedded or on-premise systems than methods that query a language model at every step.

Future Directions

• Richer procedure structure. The induced automata are nearly linear, which is why the procedure-state belief adds little at the step level. Scaling to genuinely branching or order-flexible procedures — perhaps with larger or more varied demonstration sets — is the obvious way to make that component pay off.

• Harder bimanual and fine-grained settings. IMPACT-ego shows that step identity, not mistake evidence, is the bottleneck when steps last a second or two and two hands interleave. Better per-hand segmentation, contact detection, and cross-hand reasoning are natural next steps.

• Fewer assumptions about demonstrations. The method needs correct reference executions of each task, falling back on a deterministic medoid when none exist. Reducing that dependence, or learning the automaton from partially incorrect data, would broaden applicability.

• Tighter coupling with intervention. The paper detects mistakes but does not decide what to say or do. Combining the false-alarm-calibrated alarms with language-model based guidance — while preserving the stated error rate — is the bridge from detection to a usable assistant.

Target Audience

Researchers and graduate students in egocentric vision, video understanding, and procedural activity recognition will find the protocol critique and the new benchmark setup directly actionable. Practitioners building AR guidance, robotics assistance, or smart-manufacturing monitoring will benefit from the deployable framing around false-alarm budgets and real-time throughput. Benchmark designers and evaluation researchers should read the protocol analysis section regardless of their application area, since the failure mode it documents — a metric that a perception-free rule can win — generalizes to other streaming detection tasks.

Authors’ abstract

An assistant watching egocentric video should notice a mistake from past frames alone, before the next step begins, and keep working once the person recovers. A mistake changes the state of the work, so every later step has to be read against what was done rather than against the plan. The first-mistake protocol that current online methods report on cuts each recording at its first mistake, so a fixed-time rule that never looks at the video is right on every case. We evaluate on complete trials, where mistakes and recoveries arise naturally, under a validation false-alarm budget and against controls that use timing alone. PROVIA keeps two records apart: a factual state, a learned summary of the steps each actor performed, mistakes included, and the accepted progress, an exact posterior over the state of an automaton induced from correct demonstrations by Bayesian state merging and over the execution status of each actor. Procedure-state transitions occur only in the correct-status branch; the mistake and correction branches retain the source state. A sequential test turns the per-frame mistake probability into alarms. With one filter and one optimization rule, PROVIA ranks mistakes best among the evaluated controlled baselines on CaptainCook4D, IndustReal, HoloAssist and IMPACT-ego. At a validation budget of 0.1 false alarms per minute it recalls .154 against .128 on CaptainCook4D and .034 against .015 on HoloAssist, where it leads at every budget. The pipeline runs at 58-70 frames per second. The source code is available at https://github.com/Kratos-Wen/PROVIA.