E-RGB-D: Real-Time Event-Based Perception with Structured Light

AI summary

Overview

• Research area: Event-based computer vision, active structured light, 3D sensing and RGB-D perception.
• Technical level: Advanced (assumes familiarity with event cameras, epipolar geometry, projector-camera calibration, and stereo triangulation).
• Scope: A framework that pairs a high-speed DLP projector with a monochrome event camera to achieve per-event, asynchronous color and depth sensing without frame accumulation.

What This Paper Is About

Event cameras are fast, low-power, and high-dynamic-range, but standard monochrome models cannot see static or slowly moving objects and carry no color information. This paper solves both problems by adding an active structured light projector to a monochrome event camera, so that every event is tagged with a depth and a color value as it arrives. The goal is frameless RGB-D sensing that runs at kilohertz rates instead of waiting for complete pattern sequences or reconstructed frames.

Key Contributions

1. Per-event RGB-D tagging: A pipeline that stamps each individual event with color and depth immediately, with no intermediate 2D frame or spatio-temporal map reconstruction.
2. Direct disparity lookup for DMD projectors: Since a Digital Micromirror Device projector illuminates all pixels simultaneously rather than raster-scanning row by row, the authors replace temporal maps with a calibration-based lookup table and an epipolar intersection formula, eliminating disparity search entirely.
3. Adaptive structured light pattern design: Four projection modes (color only, depth only, depth-then-color, depth-and-color) combined with swappable dot, line, and pseudo-random patterns, letting the system trade fill rate for speed or detail on demand.
4. Working ROS implementation and validation: A full hardware setup (TI LightCrafter 4500 plus Prophesee Gen3 VGA camera) with public code, calibrated and benchmarked against a RealSense D455 and the ESL baseline.

Main Findings

• Color from a monochrome sensor: Projecting patterns in red, green, and blue wavelengths in sequence lets the event camera reconstruct full color images, validated against a Macbeth ColorChecker chart and compared with a RealSense D455 RGB camera.
• Kilometer-rate acquisition: Color detection reaches the equivalent of 1400 fps, and per-pixel depth detection reaches 4 kHz, enabled by the projector's 235 µs exposure and 4.225 kHz pattern-switching rate.
• Speed advantage over baselines: In the Duck scene, the proposed method produced results in 12.57 ms and 17.23 ms, versus 33 ms for the D455, 1.55–2.36 s for ESL, and 10 s for the accumulated ground truth.
• No disparity search needed: Depth comes from intersecting the projected line with the event's epipolar line using precomputed lookup tables, removing the row-by-row search that makes ESL computationally heavy and prevents it from running in real time.
• Speed-versus-detail is tunable: Patterns with 23 lines complete faster and achieve higher fill rate, while 45-line patterns give more detail; this deliberate trade-off is not available to coded-pattern or raster-based methods, which must capture every pattern to report depth.
• Robust to texture and lighting: Because depth comes from active projection rather than passive stereo, reconstruction is independent of surface texture and color, and the event camera's >120 dB dynamic range keeps it working in low-light and high-contrast scenes.
• Handles dynamic scenes: A volleyball thrown in front of the camera was reconstructed at 120 fps from roughly 1.5 m, demonstrating operation without a fixed scanning speed.

Methodology in Plain English

The system replaces one camera of a stereo pair with a projector that throws known patterns onto the scene, a classic structured light setup. What changes is the camera: an event camera only reports pixels where brightness changes, so when a bright line sweeps across an object, it fires events exactly along that line at microsecond precision.

To get color, the projector shows the same pattern three times, once in red, once in green, and once in blue. Events captured during each pass are labeled with that channel, and the three sets are combined into a full-color image, even though the sensor itself is monochrome.

To get depth, the authors calibrate the camera and projector together so that every camera pixel and every projector pixel can be mapped onto a rectified, epipolar-aligned plane. Because the DMD projector lights the whole frame at once, the system knows which pattern column is currently active. When an event arrives, the code looks up that column's position on the projector plane, intersects it with the event's horizontal epipolar line, and computes the horizontal offset (disparity) between the two. A single division converts disparity into depth.

Because each event is processed individually, there is no frame buffer, no accumulation window, and no waiting for a full pattern sequence. The team also solved a hardware quirk: the DMD's diamond pixel grid distorts small projected dots, so the pattern is pre-rotated so the intended shape lands correctly on the object. Calibration uses a board with flickering LED dots alongside projected dots, which the authors found more stable than checkerboards.

Why This Matters

• Research impact: The work shifts structured light from a frame-and-sequence paradigm to a fully asynchronous, event-by-event paradigm. It shows that a DMD projector's simultaneous illumination makes temporal maps unnecessary, contradicting an assumption baked into prior event-based structured light work.
• Real-world applications:
  • Robot navigation and manipulation in fast-moving or low-light environments, where frame-based depth sensors blur or fail.
  • High-speed 3D scanning and industrial inspection, producing colored point clouds in milliseconds.
  • Augmented and spatial augmented reality, where low latency is required to avoid visible lag between physical and virtual content.
  • Autonomous vehicles and drones, where high dynamic range and low power matter for perception under harsh lighting.
• Industry relevance: The method runs on commercial off-the-shelf parts (TI LightCrafter 4500 and a Prophesee evaluation kit) and is released as a ROS package, which lowers the barrier for robotics teams already using ROS. It offers a potential alternative to Time-of-Flight and stereo depth sensors that struggle with color accuracy, textureless surfaces, or sunlight.

Future Directions

• Hardware refinement: The paper notes visible noise patterns in reconstructed color images caused by optical interactions between the camera and projector lenses, which the authors defer to future lens and hardware improvements.
• Higher-end patterns and projectors: Dense coded or multi-frequency patterns currently require custom projection hardware; extending the framework to support these is framed as an optional path to higher resolution.
• Mobile robot deployment: The authors claim their system adapts to both static and dynamic scenes and could tune patterns based on battery level or robot speed, but they have not yet demonstrated it on a moving platform.
• Fair benchmarking against X-maps and SEG: The paper explains why direct comparison with X-maps (raster laser projector, frame-discretized X-map) and SEG (unavailable code) was not feasible; establishing a common benchmark for event-based structured light remains open.

Target Audience

Researchers and graduate students in event-based vision, computational imaging, and 3D reconstruction, plus robotics and embedded perception engineers who need kilohertz-rate depth and color on power-constrained platforms. Practitioners already working with event cameras or structured light systems will get the most value, since the paper assumes comfort with calibration, epipolar geometry, and stereo triangulation.

Authors’ abstract

Event-based cameras (ECs) have emerged as bio-inspired sensors that report pixel brightness changes asynchronously, offering unmatched speed and efficiency in vision sensing. Despite their high dynamic range, temporal resolution, low power consumption, and computational simplicity, traditional monochrome ECs face limitations in detecting static or slowly moving objects and lack color information essential for certain applications. To address these challenges, we present a novel approach that integrates a Digital Light Processing (DLP) projector, forming Active Structured Light (ASL) for RGB-D sensing. By combining the benefits of ECs and projection-based techniques, our method enables the detection of color and the depth of each pixel separately. Dynamic projection adjustments optimize bandwidth, ensuring selective color data acquisition and yielding colorful point clouds without sacrificing spatial resolution. This integration, facilitated by a commercial TI LightCrafter 4500 projector and a monocular monochrome EC, not only enables frameless RGB-D sensing applications but also achieves remarkable performance milestones. With our approach, we achieved a color detection speed equivalent to 1400 fps and 4 kHz of pixel depth detection, significantly advancing the realm of computer vision across diverse fields from robotics to 3D reconstruction methods. Our code is publicly available: https://github.com/MISTLab/event_based_rgbd_ros