Hybrid Event Frame Sensors: Modeling, Calibration, and Simulation

AI summary

Overview

Research area: Computational imaging and computer vision, specifically hybrid event–frame (EVS + APS) sensor modeling, noise calibration, and sensor simulation.

Technical level: Advanced. The paper assumes familiarity with sensor noise theory, event-camera triggering mechanisms, the Q-function from statistics, and the standard camera pipeline (RAW, ISP, Bayer/Quad-Bayer color filter arrays).

Scope (one sentence): The paper proposes the first unified statistical noise model that jointly describes APS and EVS pixels in a single hybrid chip, derives a calibration pipeline to estimate its parameters from real data, and builds H-ESIM, a simulator that generates jointly calibrated RAW frames and event streams.

What This Paper Is About

Hybrid event–frame sensors place an Event Vision Sensor (EVS) and an Active Pixel Sensor (APS) on one chip, giving compact, spatiotemporally aligned events and intensity frames. However, the tighter integration and mixed pixel layout introduce noise patterns that prior work never quantified or modeled for the EVS side. The paper's goal is to explain, measure, and reproduce that noise — and to show that data simulated with calibrated noise transfers to real hybrid-sensor data.

Key Contributions

1. A unified statistical noise model for hybrid event sensors. The formulation explicitly incorporates photon shot noise, dark current noise, fixed-pattern noise, and quantization noise, and links EVS noise to illumination level and dark current through a Q-function treatment of the threshold-comparator trigger.
2. A quantitative calibration pipeline and dataset. APS parameters are estimated through multi-exposure compositing and a second-order polynomial variance fit, while EVS parameters are recovered via an inverse Q-function regression, using dark (lens-covered) and static multi-brightness measurements (checkerboards, color checker, resolution charts).
3. H-ESIM, a hybrid sensor simulator. Given a 3200 fps high-speed video (1296 × 1024), it generates RAW frames and events using jointly calibrated parameters, including an open ISP library implemented in NumPy (black-level correction, demosaicing, color correction, gamma and tone mapping).
4. Validation on two real hybrid sensors. Both sensors use a Quad-Bayer APS; GEN2 embeds one white event pixel per Quad-Bayer block, while Eiger integrates four color-filtered event pixels. APS resolution is 3246 × 2448, and the reported EVS resolutions are 1632 × 1224 and 816 × 612.

Main Findings

• Stronger APS–EVS integration amplifies fixed-pattern noise. The Eiger exhibits strong row noise, consistent with the influence of its embedded EVS pixels (one quarter per Quad-Bayer) on the readout path, whereas GEN2 shows no visible banding. The authors conclude that fabrication and circuit design also contribute, but that tighter integration increases fixed-pattern noise.
• Color filters and pixel position jointly shape per-pixel noise. Pixels under different color filters follow distinct noise trends across brightness levels, which supports the paper's color-separated, per-position modeling (16 positions within a Quad-Bayer block).
• Simulated APS noise statistically matches real noise. In GEN2, pixel-wise noise variance rises with image brightness, and the predicted noise from the clean intensity frame closely matches the measured distribution.
• Event probability increases with brightness. Per-pixel event probability rises with illumination, confirming the model's predicted link between illumination and noise events.
• Polarity asymmetry is brightness-dependent. In darkness, ON events slightly exceed OFF events; under illumination, the ON/OFF balance varies with brightness, indicating nonlinear threshold–brightness coupling.
• EVS parameter estimation is numerically stable. Across 200 independent runs with random initialization, all coefficients of variation are below 0.066.
• Calibrated simulation reduces the domain gap. On the Eiger sensor, video frame interpolation improves when models are fine-tuned on H-ESIM: HR-INR goes from 32.4129 to 35.2425 PSNR, 0.7821 to 0.8862 SSIM, and 0.1979 to 0.0787 LPIPS; TL-XL goes from 32.3046 to 33.8743 PSNR, 0.7755 to 0.8629 SSIM, and 0.2143 to 0.1677 LPIPS. On GEN2, TL-XL rises from 34.1787 to 34.5801 PSNR and HR-INR from 34.2631 to 35.5198 PSNR.
• Deblurring quality improves with H-ESIM fine-tuning. On Eiger, eSL improves from 0.3070 to 0.4346 CLIP-IQA, 18.74 to 22.97 MUSIQ, and 4.091 to 5.899 NRQM when fine-tuned on H-ESIM in addition to the public dataset.
• The truncated table means some results are not reported here. The Table 2 row for MAER (w/o) is cut off in the provided content, so its scores cannot be stated.

Methodology in Plain English

Both pixel types share the same lens, aperture, and photoreceptor, so the paper starts from a single shared ideal electrical signal produced by integrating scene radiance through a color filter and photodiode circuit. The difference is how each pixel reads that signal: the APS integrates it into a RAW intensity frame, while the EVS takes a logarithmic, thresholded difference between two consecutive samples and emits an ON or OFF event only when the change exceeds a threshold.

For the APS, the authors model the output as the clean signal plus illumination-dependent shot noise, exposure-time-dependent noise (a dark-current drift term plus dark-current shot noise), and fixed terms (row noise, per-pixel black-level offsets, read noise, and uniform quantization noise). For the EVS, they expand the logarithmic voltage difference with a first-order Taylor approximation into a signal term and a noise term, showing the noise follows a Gaussian whose mean and variance depend on illumination, dark current, fixed offsets, and the correlation between shot and dark-current noise. The Q-function then converts a threshold-to-noise ratio into the probability of an ON or OFF event.

Calibration follows a control-variable strategy. APS pixels are observed across many exposure times under both covered-lens (dark) and illuminated static scenes; the dark mean's linear dependence on exposure yields the dark-current and fixed-pattern terms, and the illuminated variance is fit with a second-order polynomial in intensity and exposure time for each of the 16 positions in a Quad-Bayer block. For the EVS, the affinely mapped APS intensity bridges the digital and voltage domains, and an inverse Q-function regression — solved by gradient descent because it is nonlinear — recovers the shot, dark-current, correlation, and threshold-scaling parameters, along with a bad-pixel mask for pixels with non-zero offsets.

H-ESIM then reverses the standard camera pipeline (inverse gamma, inverse color matrix and white balance, resampling to a Bayer pattern), injects the calibrated fixed terms and sampled random noise, and quantizes to the camera bit depth to produce RAW frames. The same per-pixel intensity is mapped into the EVS voltage domain, the signal and noise terms are formed, and events are sampled according to the calibrated probabilities. Both simulators are implemented in NumPy and PyTorch with a shared parameter interface.

Why This Matters

Impact on research. Prior event simulators such as ESIM and v2e use random contrast thresholds or hand-tuned hyperparameters, and existing noise studies do not provide a unified framework for joint APS–EVS modeling. This work supplies an interpretable, physically grounded alternative whose parameters are estimated from real hardware, and it makes the pipeline open and reproducible rather than a black box.

Real-world applications:

• Mobile phones and drones, where hybrid sensors are described as easily embeddable and where the compact single-chip design removes the beam-splitter alignment problems of earlier fusion setups.
• High-speed video imaging, including video frame interpolation under aggressive frame skipping.
• Video deblurring and motion restoration using jointly captured RAW frames and events.
• Optical flow estimation and other event–image fusion tasks that need aligned, temporally precise inputs.

Industry relevance. Sensor makers and camera-pipeline engineers get a calibration recipe and simulator that can be applied to new hybrid designs without new hardware-specific heuristics, and downstream product teams can pre-train networks on simulated data that better matches the target device.

Future Directions

• Separate photo response non-uniformity (PRNU) from the per-position coefficients. The paper states PRNU is not separately identified in the current calibration and could be absorbed by those coefficients.
• Handle defective and outlier pixels more systematically. The current pipeline estimates the fixed offset under dark conditions and produces a bad-pixel mask, but separate modeling or removal is left as the handling strategy.
• Extend calibration to other hybrid layouts and color filter arrangements. The authors evaluate only two sensors, one with a single white event pixel per Quad-Bayer block and one with four color-filtered event pixels.
• Broaden downstream evaluation beyond interpolation and deblurring. The paper demonstrates transfer on those two tasks on real data; whether the calibrated simulation helps other event–frame fusion problems is left open.

Target Audience

Researchers and engineers working on event-based vision, computational photography, and sensor simulation who need to reproduce hybrid-sensor noise faithfully for training or benchmarking. It is also relevant to hardware and ISP engineers at sensor and mobile-platform companies designing or validating hybrid event–frame chips, and to graduate students with a background in imaging physics or statistical signal modeling who want a worked example of deriving a noise model and calibrating it end to end.

Authors’ abstract

Hybrid event-frame sensors integrate an Event Vision Sensor (EVS) and an Active Pixel Sensor (APS) within a single chip, combining the high dynamic range and low latency of the EVS with the rich spatial intensity information from the APS. While this tight integration offers compact and temporally precise imaging, the complex circuit architecture introduces nontrivial noise patterns that remain poorly understood and unmodeled. In this work, we present the first unified statistics-based imaging noise model that jointly describes the noise behavior of APS and EVS pixels. Our formulation explicitly incorporates photon shot noise, dark current noise, fixed-pattern noise, and quantization noise, and links EVS noise to illumination level and dark current. Based on this formulation, we further develop a calibration pipeline to estimate noise parameters from real data and provide a detailed analysis of both APS and EVS noise behaviors. Finally, we propose H-ESIM, a statistically grounded simulator that generates RAW frames and events under realistic jointly calibrated noise statistics. Experiments on two hybrid sensors validate our model across multiple imaging tasks, including video frame interpolation and deblurring, demonstrating strong transfer from simulation to real data.