IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants

AI summary

Overview

Research area: Egocentric computer vision, multimodal dataset construction, and AI assistants for industrial settings, with a secondary focus on computer vision benchmarks (mistake detection, video question answering, collaborative task understanding).

Technical level: Intermediate. Readers will benefit from familiarity with egocentric vision (e.g., EPIC-KITCHENS, Ego4D) and video-language models, though the paper's framing and contribution are accessible without deep technical background.

Scope: IndEgo is a large multimodal dataset of real industrial work, captured from wearable (egocentric) and fixed (exocentric) cameras, designed to support AI research on assisting workers in cognitively and physically demanding collaborative tasks.

What This Paper Is About

Most egocentric vision datasets focus on everyday activities like cooking or shopping, while existing industry-adjacent datasets (e.g., Assembly101, HoloAssist) largely involve seated participants at a workstation doing short procedural tasks. Real industrial work, by contrast, involves moving through cluttered spaces, long arduous tasks, tool handling, and collaboration between workers—conditions that current datasets rarely capture.

The authors introduce IndEgo, a multimodal dataset collected using Meta's Project Aria smart glasses plus external cameras, covering five categories of industrial work (assembly/disassembly, inspection/repair, logistics/organisation, woodworking, and miscellaneous) with rich annotations and benchmarks for mistake detection, reasoning-based question answering, and collaborative task understanding.

Key Contributions

1. A large multimodal industrial dataset: 3,460 egocentric recordings (~197.1 hours) and 1,092 exocentric recordings (~96.8 hours), with synchronised eye gaze, hand pose, motion/trajectory, audio, narration, and semi-dense point cloud data.

2. Explicit coverage of collaborative work: Many recordings feature two workers jointly performing a task in roles such as partners, teacher–student, or leader–assistant, with both wearing Aria devices and annotations from each person's perspective.

3. Rich annotation layer: Approximately 34k fine-grained action annotations (verbs, nouns, adjectives), keystep labels for procedural tasks, task graphs, instruction guides, audio transcripts, mistake labels with a context-dependent severity taxonomy, and summaries from multiple viewpoints.

4. Challenging benchmarks with baselines: Benchmarks for Mistake Detection (MD), reasoning-based Video Question Answering (QA), collaborative task understanding, and summarisation, evaluated with state-of-the-art video-language models (VideoLLaMA3, InternVL-2.5, Qwen2.5-VL, Gemini 2.0 Flash Thinking).

Main Findings

• State-of-the-art models perform poorly on Mistake Detection: Zero-shot F1 scores range from ~23% to ~41%, with Gemini 2.0 Flash Thinking strongest. Fine-tuned classifiers (MLP or Transformer heads on VLM embeddings) improve results to ~34–40% F1, still far from reliable.

• Early Mistake Detection is harder: When only the first 50% of frames from a segment are available, F1 typically drops to ~30–33%, showing that models struggle to anticipate mistakes before they fully unfold.

• Industry-specific mistakes are modelled in context: The authors categorise 1166 MD sequences with mistakes into Severe (2.3%), Process Failure (18.7%), Impact Future Steps (7%), and Harm risk (5%), going beyond binary right/wrong labels.

• Joint ego + exo views help modestly: Combining both perspectives consistently improves MD F1 slightly across models (e.g., 0.44 vs. 0.43 for Gemini zero-shot), but gains are small because the exocentric view mainly helps when the ego view is occluded.

• Modality utility is context-dependent: Adding audio or gaze helps in some cases (e.g., collaborative tasks where audio conveys intent), but audio can hurt in noisy shop-floor environments, and gaze overlay can hurt when the mistake occurs outside the worker's gaze region.

• Video QA on industrial singular actions is challenging: Best model accuracy is 64.1% (Gemini), compared to 90.0% for human evaluators. Temporal (Tm), situated reasoning (Si), recognition (Re), and abductive (A) questions all expose weaknesses—models often confuse similar actions (attaching vs. detaching) or misjudge spatial situations.

• Collaborative understanding is early-stage: Zero-shot accuracy for predicting whether an action belongs to the wearer or the coworker is 35.2%, and predicting what the wearer will do given the coworker's action reaches 42.1% with a fine-tuned transformer.

• Annotation reliability varies by granularity: Keystep annotations show excellent agreement (Krippendorff's α = 0.97), while strict fine-grained action agreement is lower (α = 0.25) and rises to 0.54 when similar verbs/nouns are grouped—reflecting genuine ambiguity in how actions can be described.

Methodology in Plain English

The researchers equipped 20 participants (15 male, 5 female, varying skill levels from beginner to expert) with Meta Project Aria smart glasses in an industrial research facility at Fraunhofer IPK and TU Berlin. Aria captures RGB video, eye tracking, SLAM, IMU, and audio. A second external camera (Sony A6400 or similar) recorded an exocentric third-person view for many sessions.

Participants performed real industrial tasks in five categories: assembly, disassembly, inspection and repair, logistics and organisation, woodworking, and miscellaneous. For each session, they were given either a bare goal (unguided, narrate your inner monologue), step-by-step instructions (guided), or a collaborative setup where two workers performed the task together—sometimes in teacher–student roles with matched tool setups. Sessions were repeated with deliberate planned and unplanned mistakes for the MD benchmark.

After recording, participants annotated their own actions in logical segments (e.g., "connect metal bar," "attach drill bit"), which a second person reviewed. For collaborative recordings, each worker annotated their own actions separately. Raw sensor data was processed to extract eye gaze, hand pose, semi-dense point clouds, and trajectories. Audio was transcribed. For procedural tasks, task graphs and instruction guides were also collected.

The authors then benchmarked state-of-the-art video-language models in zero-shot settings and with fine-tuned classifier heads on the extracted embeddings. They also ran ablations on modalities (RGB only, +audio, +gaze, +both) and on ego vs. exo vs. joint views.

Why This Matters

Impact on research: IndEgo addresses a clear gap: existing egocentric datasets underrepresent true industrial conditions—long-horizon tasks, physical movement, tool handling, and human collaboration. The benchmarks show that current multimodal models, including strong proprietary ones, fail substantially on industrial mistake detection, situated reasoning, and collaborative understanding, providing a concrete target for future model development.

Real-world applications:

• Wearable industrial assistants that guide workers through complex assembly or repair procedures, answer questions in situ, and flag errors in real time.
• Training and skill transfer systems that capture expert demonstrations (including teacher–student dynamics) and help onboard new workers.
• Safety monitoring that detects hazardous mistakes such as mishandling fragile or dangerous objects, misusing tools, or failing to follow PPE protocols.
• Collaborative robotics and embodied agents that need to understand what a human coworker is doing and what they are likely to do next.

Industry relevance: The dataset is drawn from a genuine industrial facility with domain-specific tools, mechanical assemblies, PC cabinets, and woodworking equipment. The mistake taxonomy (Severe, Process Failure, Impact Future, Harm) reflects the reality that in industry, some errors are merely inefficient while others are hazardous or cascading—an important distinction for any deployed AI assistant.

Future Directions

• Improving temporal and situated reasoning in VLMs: Current models confuse visually similar actions and misjudge spatial context; targeted architectures or training strategies are needed.
• Better use of multimodal signals: Since audio and gaze help only contextually, future work could develop adaptive fusion methods that weight modalities based on scene and task characteristics.
• Scaling collaborative understanding: With accuracy near 35–42%, predicting and differentiating worker vs. coworker actions—and inferring intent in collaborative tasks—remains largely unsolved.
• Long-horizon video understanding and summarisation: Videos up to 68 minutes raise open questions about summarisation fidelity, memory, and reasoning over extended sequences.
• Cross-view and cross-modal alignment: The joint ego + exo setup opens questions about how to transfer knowledge between views and use one view to compensate for the other's limitations.

Target Audience

• Computer vision and multimodal ML researchers working on egocentric video, action recognition, video-language models, or embodied AI.
• Human-robot collaboration and industrial AI practitioners interested in assistive systems, mistake detection, or worker support tools.
• Dataset and benchmark designers seeking a reference for multimodal data collection protocols, annotation schemes, and inter-annotator agreement reporting.
• Industry R&D teams exploring wearable smart glasses, augmented reality guidance, or quality assurance in manufacturing and skilled trades.

Authors’ abstract

We introduce IndEgo, a multimodal egocentric and exocentric dataset addressing common industrial tasks, including assembly/disassembly, logistics and organisation, inspection and repair, woodworking, and others. The dataset contains 3,460 egocentric recordings (approximately 197 hours), along with 1,092 exocentric recordings (approximately 97 hours). A key focus of the dataset is collaborative work, where two workers jointly perform cognitively and physically intensive tasks. The egocentric recordings include rich multimodal data and added context via eye gaze, narration, sound, motion, and others. We provide detailed annotations (actions, summaries, mistake annotations, narrations), metadata, processed outputs (eye gaze, hand pose, semi-dense point cloud), and benchmarks on procedural and non-procedural task understanding, Mistake Detection, and reasoning-based Question Answering. Baseline evaluations for Mistake Detection, Question Answering and collaborative task understanding show that the dataset presents a challenge for the state-of-the-art multimodal models. Our dataset is available at: https://huggingface.co/datasets/FraunhoferIPK/IndEgo