Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos

AI summary

Overview

Research area: Computer vision and video-language understanding, specifically egocentric (first-person) video analysis, instructional AI, and human mistake understanding.

Technical level: Intermediate to Advanced. The paper assumes familiarity with transformer architectures, cross-attention, semantic role labeling, and standard video benchmarks, though the core idea (what/when/where a mistake occurs) is described in accessible terms.

Scope: The paper defines a new task called Mistake Attribution (MATT), builds an automatic data engine (MisEngine) that produces two large mistake datasets from existing egocentric corpora, and proposes a unified transformer model (MisFormer) that predicts which instruction component was violated, when the mistake became irreversible, and where it appears in the frame.

What This Paper Is About

Existing work on mistakes in egocentric video mostly asks a binary question: did a mistake happen, or not? This paper argues that a useful AI assistant needs a much finer answer — specifically what part of the user's instruction text was violated (for example, the wrong object was picked up), when during the action the mistake became irreversible (the Point-of-No-Return, or PNR), and where in that frame the mistake is visible. The authors' goal is to define this three-part "mistake attribution" problem, create enough labeled data to train and benchmark models on it, and show that a single unified model can handle all three parts at once.

Key Contributions

1. The MATT task definition. Mistake Attribution is formalized as a mapping from an instruction text and an attempt video to a triplet: per-semantic-role correctness labels, a Point-of-No-Return (PNR) timestamp, and a mistake grounding box in the PNR frame.

2. MisEngine, an automatic data engine. MisEngine uses Semantic Role Labeling to parse action descriptions, cross-matches instructions against other action descriptions by semantic role (predicate, object) to create misaligned instruction-video pairs, and inherits temporal and spatial annotations from the source datasets — producing fully labeled data for free.

3. Two large-scale MATT datasets. Applying MisEngine to Ego4D and EPIC-KITCHENS yields Ego4D-M (257,584 samples from 16,099 instruction texts) and EPIC-KITCHENS-M (221,094 samples from 12,283 instruction texts), described as at least two orders of magnitude larger than prior mistake datasets. A human study is reported to confirm the ecological validity of these samples.

4. MisFormer, a unified attribution model. A transformer-based architecture with a dual-branch video/text feature extractor and three heads for semantic, temporal, and spatial attribution, reported to outperform task-specific state-of-the-art methods by at least 6.66%, 21.81%, 18.7%, and 3.00% in video-language understanding, temporal localization, hand-object interaction, and mistake detection, respectively.

Main Findings

• Semantic attribution: MisFormer reaches 83.89% F1 on EPIC-KITCHENS-M and 56.24% F1 on Ego4D-M, surpassing the best baseline by 6.66% and 5.30% respectively. ChatGPT reaches 77.23% and 50.95% F1; fine-tuned open-source video-language models (LLaVA-Vid, Vid-Chat) stay below 75% F1 on EPIC-KITCHENS-M and below 45% F1 on Ego4D-M. Role-wise, MisFormer improves Predicate by 10.60% and 5.02% F1, and Object by 1.99% and 5.55% F1, on the two datasets.

• Object versus Predicate: Across methods, Object consistently outperforms Predicate, which the authors attribute to the difficulty of modeling fine-grained motion (verbs) in egocentric video.

• Temporal attribution: On Ego4D-M, MisFormer reduces mean absolute error by 21.81% relative to the best existing method — a decrease of 5.34 frames and 0.178 seconds. Its absolute MAE is 19.14 frames (0.638 seconds), versus 24.48 frames (0.816 seconds) for EgoT2 and 48.96 frames (1.632 seconds) for EgoMotion-COMPASS. The authors note the average clip is about 8 seconds (240 frames at 30 fps), so this equals a 2.23% reduction relative to video duration.

• Spatial attribution: MisFormer improves mIoU by +9.33 over hand-detection baselines (MediaPipe-U: 49.88 mIoU, 13.47 CD, 20.98 BSE; MisFormer: 59.21 mIoU, 10.36 CD, 16.27 BSE), and reduces Center Distance and Box Size Error by 3.11 and 4.71. A state-of-the-art hand-object interaction detector (SSDA) is more accurate on localization (64.54 mIoU, 7.21 CD, 12.34 BSE), but MisFormer runs its spatial module at 68.9 FPS versus SSDA's 28.44 FPS on the same NVIDIA H100 GPU. The table caption describes the hand-detection improvement as 18.70% mIoU.

• Mistake detection on the new benchmarks: MisFormer reaches 78.05 F1@0.5 and 80.90 accuracy on EPIC-KITCHENS-M, and 57.55 F1@0.5 and 59.89 accuracy on Ego4D-M, versus AMNAR (16.34/17.45 on EK; 12.57/14.24 on Ego4D-M) and EgoPED (21.18/22.23 on EK; 15.62/17.34 on Ego4D-M).

• Mistake detection on the prior EgoPER benchmark: Training MisFormer from scratch with only semantic supervision underperforms specialized detectors (26.78 F1@0.5, 27.89 accuracy, versus AMNAR's 28.54/29.65 and EgoPED's 32.18/33.23). However, pretraining on EPIC-KITCHENS-M and then fine-tuning on EgoPER yields 35.18 F1@0.5 and 36.23 accuracy, competitive with the specialized detectors while preserving attribution capability.

• Ablation results (Ego4D-M): The default model scores 56.24 semantic F1@0.5, 0.438 s temporal MAE, 59.21 spatial mIoU, and 57.55 detection F1@0.5. Replacing the backbone with LaViLa drops all four to 49.16 / 0.561 / 51.37 / 46.05. Removing the projection block gives 51.34 / 0.457 / 55.43 / 52.75. Removing temporal supervision gives 51.29 / 0.623 / 57.78 / 57.46. Using GradCAM heatmaps gives 55.52 / 0.482 / 55.03 / 57.51.

• Dataset composition: With predicate and object roles, there are C = 4 misalignment categories (Predicate Mistake, Object Mistake, Both Mistakes, No Mistake). EPIC-KITCHENS-M carries only semantic annotation because the original dataset lacks PNR frame-number annotation; Ego4D-M carries semantic, temporal, and spatial annotation.

• Splits: Each dataset is split 8:1:1, yielding 206K/25K/25K samples for Ego4D-M and 176K/22K/22K for EPIC-KITCHENS-M.

Methodology in Plain English

The researchers start from the observation that mistake data is hard to collect: as people practice a task they make fewer mistakes, and staging mistakes introduces unnatural visual bias. Their alternative is to manufacture mistakes automatically from ordinary action-recognition datasets.

They take action descriptions (instruction texts) and use Semantic Role Labeling to break each one into components such as predicate ("pick up") and object ("the sieve"). They then cross-match components between different instructions: a video that originally shows "pick up the pan" can be paired with the instruction "pick up the sieve," which makes the object role a mistake. Because the pairing is controlled per role, the correct/mistake label for each role falls out for free, and because the source videos already have annotations, the PNR timestamp and grounding box can be inherited rather than newly labeled. They apply this to Ego4D and EPIC-KITCHENS; EPIC-KITCHENS-M ends up with semantic labels only because the source lacks PNR timestamps.

For the model, they extract text features per semantic role and video features per frame using InternVideo2, then pass them through a projection block where roles attend to each other and to the video. Three heads follow: a feed-forward classifier producing correct/mistake per role; a temporal head that downsamples video features to frame level, cross-attends to the instruction, and outputs a probability distribution over frames whose argmax is the PNR; and a spatial head that reuses cross-attention weights as a saliency map, combines it with the RGB PNR frame, and regresses a bounding box. At inference, temporal and spatial heads only run if at least one role is predicted as a mistake. Semantic attribution also doubles as mistake detection — a clip is a mistake if any role is flagged.

Why This Matters

Impact on research: The paper reframes mistake understanding from binary detection and free-form explanation to a structured, instruction-grounded, three-axis attribution problem. It also shows that large, attribution-rich supervision can be generated automatically rather than collected by hand, which is a reusable idea for other structured video-understanding tasks. The results indicate that existing specialized models — video-language, PNR localization, hand-object interaction, and mistake detection — all struggle with the combined task.

Real-world applications:

• Instructional AI assistants that can tell a user not just that they erred but which part of the instruction they violated and at what moment the action became irreversible.
• Self-learning and skill-acquisition tools where a learner reviews their own attempt videos annotated with what/when/where they went wrong.
• Robotics and industrial procedural support, where a wrong tool or wrong component can be caught at the point of no return rather than after the fact.
• Automated quality or safety review of recorded procedures, using the grounding box to point a reviewer at the exact region of the mistake.

Industry relevance: The paper reports spatial attribution at 68.9 FPS versus 28.44 FPS for SSDA on an NVIDIA H100, and shows that a single unified model replaces several task-specific pipelines. That combination of speed and unification matters for deployment in assistants that must respond while a task is still in progress. The authors are affiliated with the University of Michigan, with one author also affiliated with Voxel51.

Future Directions

1. Longer, compositional instructions. The authors state that real applications will involve longer instructions with richer role inventories, and that studying MATT under those conditions — where different roles pose distinct challenges — is an important next step.

2. Task-driven representations for mistake understanding. Rather than relying on features from existing video-language pretraining, the authors propose developing a representation that jointly embeds video and instruction specifically for mistake understanding, potentially as a general foundation for the problem.

3. Scaling the data engine's inherited signals. MisEngine can in principle inherit additional signals already present in action datasets, such as eye gaze, head pose, and audio, which the authors mention as a path toward further large-scale mistake understanding.

4. Closing the spatial gap to specialized detectors. MisFormer trails SSDA on spatial localization accuracy (59.21 versus 64.54 mIoU) while being faster and unified, leaving open the question of whether a unified model can match specialized ones on grounding quality.

Target Audience

This paper is most useful to computer vision researchers working on egocentric video, video-language models, and instructional or assistive AI; to researchers studying mistake, anomaly, or error detection in procedural video; and to practitioners building video-based guidance systems who need structured outputs rather than binary flags. Readers with a background in transformers and video understanding will get the most out of the methodology sections, while the MATT formulation and dataset statistics are accessible to a broader audience.

Authors’ abstract

We introduce Mistake Attribution (MATT), a new task for fine-grained understanding of human mistakes in egocentric videos. While prior work detects whether a mistake occurs, MATT attributes the mistake to what part of the instruction is violated (semantic role), when in the video the deviation becomes irreversible (the Point-of-No-Return, PNR), and where the mistake appears in the PNR frame. We develop MisEngine, a data engine that automatically constructs mistake samples from existing datasets with attribution-rich annotations. Applied to large egocentric corpora, MisEngine yields EPIC-KITCHENS-M and Ego4D-M -- two datasets up to two orders of magnitude larger than prior mistake datasets. We then present MisFormer, a unified attention-based model for mistake attribution across semantic, temporal, and spatial dimensions, trained with MisEngine supervision. A human study demonstrates the ecological validity of our MisEngine-constructed mistake samples, confirming that EPIC-KITCHENS-M and Ego4D-M can serve as reliable benchmarks for mistake understanding. Experiments on both our datasets and prior benchmarks show that MisFormer, as a single unified model, outperforms task-specific SOTA methods by at least 6.66%, 21.81%, 18.7%, and 3.00% in video-language understanding, temporal localization, hand-object interaction, and mistake detection, respectively. Project page: https://yayuanli.github.io/MATT/