Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchen

AI summary

Overview

Research area: Neuromorphic (event-based) computer vision, dataset construction, and egocentric multimodal perception.

Technical level: Intermediate — readers benefit from familiarity with event cameras and standard computer-vision benchmark tasks (action recognition, object detection, stereo depth estimation), though the paper explains its pipeline in accessible terms.

Scope: This paper introduces EventKitchen, a large-scale, egocentric, stereo event camera dataset of unscripted human cooking activities, together with baseline results for three event-based perception tasks.

What This Paper Is About

Event cameras offer high temporal resolution, high dynamic range, and low power consumption, but existing neuromorphic datasets concentrate on automotive and drone settings, and the few event-based human activity datasets use scripted actions or fixed viewpoints. The authors address this gap by recording natural, unscripted cooking activities with a wearable multi-sensor helmet — including stereo event cameras — in 13 real kitchens. The goal is to provide a challenging, multi-task benchmark for event-based perception in human-centric daily-life environments.

Key Contributions

1. The first large-scale stereo event camera benchmark dataset collected in a real-world, daily-life, egocentric setting for multiple tasks. The dataset comprises 5.5 hours of stereo event recordings captured from 10 participants across 13 diverse kitchens (8 private and 5 public).
2. Human annotations of action labels, action segments, object labels, and object bounding boxes, totaling 10,762 action segments (268 action classes, 32 verb classes) and 13,482 bounding boxes across 12 object classes.
3. Ground-truth depth maps and calibration matrices among sensors, including 297,547 raw ground-truth depth maps and rectification procedures that map depth onto the left and right event camera domains for stereo depth estimation.
4. Seven baseline models evaluated across three event-based tasks, establishing benchmark numbers for action recognition, object detection, and stereo depth estimation.

Main Findings

• Dataset scale and composition: EventKitchen contains 14 recording sessions and 110 sequences across 13 kitchens, totaling 19,750 seconds (5.5 hours), with an average sequence duration of approximately three minutes. It provides 5.5 hours of stereo event recordings, 5.5 hours of stereo RGB video from CMOS cameras at 30 fps, 5.5 hours of D-RGB video at 15 fps, 5.5 hours of 16-bit depth video at 15 fps, and 5.5 hours of 6-axis IMU data at 200 fps.
• Sensor resolution: The stereo event cameras and RGB streams provide 1280 × 720 resolution, matching or exceeding the resolution of the compared real-world event-based human activity datasets listed in Table 1.
• Verb distribution resembles unscripted data: The frequency ranking of common verbs in EventKitchen closely matches that of EPIC-KITCHENS — Put (1st), Take (2nd), Open (3rd), Close (4th), Move (5th), Cut (6th), Rinse/Wash (7th) — which the authors interpret as evidence that the actions are unscripted.
• Action recognition is difficult: On 69 action classes, TSM reaches 19.23% top-1 and 42.26% top-5 accuracy, while Swin reaches 24.69% top-1 and 56.48% top-5. On 18 verb classes, TSM reaches 38.17% top-1 and 87.20% top-5, while Swin reaches 46.08% top-1 and 90.67% top-5. Verb classification is easier than action classification, and Swin consistently outperforms TSM.
• Object detection degrades sharply relative to in-domain benchmarks: YOLOv10 reaches 16.2% AP (29.9% AP50, 38.1% AP05) on EventKitchen versus 54.4% AP on COCO. RVT reaches 7.6% AP (16.1% AP50, 22.7% AP05) versus 47.4% AP on 1Mpx. EvRT-DETR reaches 8.5% AP (16.7% AP50, 22.7% AP05) versus 50.1% AP on 1Mpx.
• Small, occluded objects are hardest: Detection rates for "fork" and "spoon" are especially low — YOLOv10 achieves 1.1% and 1.0% AP respectively; RVT achieves 0 and 0.1%; EvRT-DETR achieves 0 and 0.2%. The authors attribute this to their small size, occlusion by hands and food, and fast actions such as stirring.
• Stereo depth estimation benefits more from more events than from higher sampling rate: SE-CFF achieves 88.19 mm RMSE / 59.32 mm MAE with a 5M event stack at 3 Hz, 88.21 mm RMSE / 57.38 mm MAE with a 5M stack at 1 Hz, and 84.91 mm RMSE / 54.84 mm MAE with a 15M stack at 1 Hz. Ground-truth STD is 106.26 mm and MAD is 82.00 mm.
• A frame-based foundation model underperforms zero-shot: FoundationStereo yields 155.06 mm RMSE and 123.85 mm MAE, higher than SE-CFF, which the authors attribute to the need for adaptation strategies when converting events to frame-based inputs.
• Depth statistics span the kitchen workspace: Across 10,800 sampled raw depth maps, depths range from 200 mm to 1500 mm, with objects typically about 700 mm away.
• Kitchen-level split protocol: Nine kitchens (10 sessions, 82 sequences, 15,683 s, 11,065 bounding boxes, 8,598 action segments, 235,271 depth maps) are used for training and four kitchens (4 sessions, 28 sequences, 4,067 s, 2,417 bounding boxes, 2,164 action segments, 62,276 depth maps) for testing, a roughly 0.8:0.2 split.
• Natural behavior varies across participants: For the "fry egg" activity, 2 participants use chopsticks, 5 use a spatula, and 1 uses no tool at all.

Methodology in Plain English

The authors built a wearable data collection device on a bicycle helmet carrying two HD Prophesee Gen4 event cameras forming a stereo pair, two CMOS RGB cameras positioned next to each event camera as RGB-EVENT units, an Intel RealSense depth camera (D435i or D455) providing depth maps, an additional RGB stream, and an IMU. All sensors connect to a laptop over USB-3 and are managed and synchronized through the Robot Operating System (ROS).

Ten participants wore the helmet and a backpack containing the recording laptop while naturally performing cooking activities. Participants each chose from 13 possible activities — cut bread, cut cake, fry bacon, fry egg, fry pepper, make cereal bowl, make coffee, make fruit salad, make lemon water, make a sandwich, make tea, make vegetable salad, and wash dish — with no instructions on how to perform them. Identifiable items were removed beforehand for privacy, and calibration was performed before and after collection.

Because event cameras are asynchronous, standard corner-detection calibration does not apply directly. The authors used E2Calib to reconstruct event streams into grayscale frames, then applied standard OpenCV calibration, achieving reprojection errors below two pixels for all camera pairs, with the left event camera as the reference for event-based stereo depth estimation.

Annotation was performed on the D-RGB video rather than directly on events, since identifying objects and actions in raw events is too difficult. Action segments were labeled as {verb, noun} pairs referencing the EPIC-KITCHENS verb list, using three randomly assigned annotators working in sequence (define labels, annotate segments, verify). For bounding boxes, one annotator labeled the D-RGB frames and a second reviewed them; each 2D box was turned into a set of 3D points using the aligned depth map, then projected onto the left and right event cameras, with edge smoothing and removal of boxes outside the event cameras' field of view. Bounding boxes were annotated at a rate of 1 frame per 4 seconds. Depth maps were similarly projected into the left event camera field of view and rectified with the stereo event pair.

Baseline experiments summed event representations from the left and right event cameras as the combined input, used a kitchen-level split so that test environments were entirely unseen during training, and evaluated action recognition (TSM with a ResNet-50 backbone and Swin with a Swin-Base backbone, both pretrained on Kinetics 400), object detection (YOLOv10-x pretrained on MS-COCO, RVT-Base pretrained on 1Mpx, and EvRT-DETR-B initialized from a trained RT-DETR-B model), and stereo depth estimation (SE-CFF trained from scratch in an event-only setting, plus zero-shot FoundationStereo applied to E2VID-reconstructed grayscale images). Depth training and evaluation were restricted to 200 mm–1500 mm to filter invalid values.

Why This Matters

Impact on research: Event-based vision has been dominated by automotive and drone benchmarks such as 1Mpx, GEN1, DSEC, and MVSEC. EventKitchen shifts the focus to human-centric daily-life activity with real (not simulated) stereo event recordings, unscripted actions, egocentric viewpoints, and multiple simultaneously supported tasks. The low baseline scores — well below the same models' results on COCO and 1Mpx — show that the dataset poses challenges that current approaches do not solve. The authors also note that methods developed prior to 2021 already exceeded 90% object recognition accuracy on several older event datasets, motivating newer and harder benchmarks.

Real-world applications:

• Wearable and assistive AI that recognizes cooking or daily-living activities through a head-mounted sensor.
• Augmented Reality, Virtual Reality, and Human-Computer Interaction systems that require low-power, high-temporal-resolution sensing in indoor environments.
• Robotics and manipulation systems operating alongside people in kitchens and other cluttered indoor spaces, where fast motion and small objects matter.
• Indoor near-field 3D perception, using the stereo event and rectified depth data for depth estimation at close range.

Industry relevance: The dataset's low-power, high-dynamic-range sensor setup and its multi-modal stream layout are directly relevant to companies building event sensors, wearable devices, and neuromorphic perception pipelines. The paper's device includes hardware from Prophesee and Intel, and an author is affiliated with STMicroelectronics.

Future Directions

• Addressing class imbalance and long tails: The authors observe long-tail distributions in both object classes and action classes and identify class imbalance as a significant remaining challenge for baselines, suggesting few-shot learning and generalization as relevant directions.
• Scaling annotation volume: The paper notes that the number of human annotations is limited compared to 1Mpx datasets with 25 million non-human annotations, and calls expanding annotation volume an important direction.
• Improving event-to-frame adaptation for foundation models: The high error of the FoundationStereo model highlights the need for better strategies to convert event data into frame-based inputs for foundation models.
• Enabling large-scale pre-training in human-centric domains: The authors frame EventKitchen as a step toward large-scale pre-training and systematic benchmarking for neuromorphic vision beyond autonomous driving.

Target Audience

Researchers and engineers working on event-based or neuromorphic vision, egocentric and wearable perception, human activity recognition, event-based object detection, and event-based stereo depth estimation. The dataset is also relevant to those studying multimodal sensor fusion (events, RGB, depth, IMU), benchmark dataset design, and applications in AR/VR, HCI, robotics, and wearable AI.

Authors’ abstract

Event cameras, also known as neuromorphic cameras, have gained significant attention in recent years due to their high temporal resolution, high dynamic range, and low power consumption. While many studies and datasets in neuromorphic vision have focused on automotive and drone applications, human-centric daily-life scenarios remain largely underrepresented, despite their importance for developing and benchmarking event-based perception systems. Moreover, the few existing event-based human activity datasets are typically recorded with scripted human actions, limiting their ability to capture natural human behaviors. In this paper, we introduce EventKitchen, a large-scale stereo event camera benchmark dataset of human cooking activities in the kitchen. EventKitchen is egocentrically collected from 10 participants in 13 diverse kitchens, where the participants wear a helmet with multiple sensors and naturally perform cooking activities, without any scripted actions. EventKitchen comprises 5.5 hours of stereo event recordings with synchronized RGB, depth, and IMU data. We provide human annotations for 10,762 action segments and 13,482 bounding boxes. We train baseline models on EventKitchen to perform multiple event-based tasks, including action recognition, object detection, and stereo depth estimation. By capturing natural, real-world human activities, EventKitchen establishes a challenging benchmark for neuromorphic vision beyond autonomous driving.