PIE-PS: Photometric Stereo from Physical Irradiance Event Streams

AI summary

Overview

• Research area: Computer vision and computational photography, specifically event-camera-based photometric stereo for dense surface normal reconstruction.
• Technical level: Advanced. The paper derives event-trigger physics, pairwise photometric constraints, and a graph neural network with attention, so it assumes familiarity with photometric stereo, event cameras, and graph learning.
• One-sentence scope: PIE-PS turns pairs of adjacent events from a moving-light capture into "Physical Irradiance Events" and feeds them to a graph network with reliability attention to predict dense surface normals without needing the event camera's unknown contrast threshold.

Published as a SIGGRAPH Asia 2026 Conference Paper (December 01–04, 2026, Kuala Lumpur, Malaysia), arXiv:2610.08188v1 [cs.CV], DOI 10.1145/3829340.3842249. Authors are affiliated with Jilin University, Guangdong Institute of Intelligence Science and Technology, Beijing Institute of Technology, and The Hong Kong University of Science and Technology. Project page: https://pie-ps.github.io

What This Paper Is About

Photometric stereo recovers surface shape by watching how shading changes as light moves, but conventional frame-based cameras are limited by frame rate and dynamic range under fast or high-contrast lighting. Event cameras record asynchronous log-image-irradiance changes with microsecond latency and high dynamic range, which makes them attractive for this task, yet the raw events are sparse and their interpretation depends on an unknown contrast threshold. This paper's goal is to reconstruct dense, accurate surface normals directly from raw event streams under known, moving illumination by replacing the threshold-dependent per-pixel analytical formulation with a learned pairwise representation.

Key Contributions

1. Pairwise event photometry with a direct solver baseline. The authors derive a photometric relation between two adjacent events at the same pixel and the light directions at their timestamps, yielding a physics-only numerical solver (Eq. 7) that they keep as an explicit baseline to expose the limits of a purely analytical route.

2. The PIE-PS framework. A learning-based method for event-based photometric stereo that takes raw event streams and known lighting as input and produces dense surface normals, using Physical Irradiance Events (PIEs) — structured observations built from two adjacent same-pixel events plus their corresponding light directions.

3. PIEF, PIE-GNN and RGA. Three named components: the Physical Irradiance Event Feature, a threshold-free signed event rate (event polarity divided by inter-event time); PIE-GNN, which treats each PIE as a graph node and shares spatial and temporal context through a spline graph convolution using light-pair geometry; and Reliability-Grading Attention, which predicts a soft reliability weight per PIE to down-weight unreliable observations before pixel aggregation.

4. A prototype capture system and evaluation on synthetic and real data. A CeleX-V event camera rig with a programmable ring light, plus the new PIE-Sim synthetic dataset (77 training scenes, 30 test scenes, 600 rendered frames per sequence) and DiLiGenT-Sim (9 test scenes), used to show gains over prior event-based photometric stereo methods and the direct solver.

Main Findings

• DiLiGenT-Sim results: PIE-PS achieves an average MAE of 8.70°, which the paper reports as 5.44° lower than the best published baseline, EventUPS (14.14°), and 6.90° lower than the direct solver (15.60°). Per-object PIE-PS errors are Ball 3.93, Buddha 8.67, Cat 5.15, Cow 12.90, Goblet 7.81, Harvest 12.91, Pot1 6.33, Pot2 8.34, Reading 12.25. EventPS-OP averages 19.73, EventPS-FCN 16.26, and EventPS-CNN 16.45.

• PIE-Sim results by object family and lighting trajectory: PIE-PS gives the lowest error in every group, with an average MAE of 10.66° versus 21.88° for the direct solver (11.22° lower) and 25.84° for EventPS-FCN (15.18° lower). EventPS-CNN averages 39.15°. PIE-PS's best groups are Blobby/Hypotrochoid at 6.41° and Blobby/DiLiGenT at 4.81°; its worst is Blobby/Circle at 10.68°, and Sculpture/Circle is its highest at 17.17°.

• Real-world results: Over six real scenes (three EventPS real scenes and three captured objects), PIE-PS is best on all six, averaging 10.38° — 10.53° lower than EventPS-FCN (20.91°) and 13.52° lower than the direct solver (23.90°). EventPS-CNN averages 32.12°. Individual PIE-PS numbers are Real-1 6.51, Real-2 6.09, Real-3 5.92, Bunny 12.69, Rectangle 11.52, Flower 19.57.

• Ablation on PIEF: Removing the signed event-rate input (keeping light-pair geometry and the rest of the network) raises average MAE from 10.66° to 14.52°, indicating that light-pair geometry alone does not carry the photometric signal caused by the light change.

• Ablation on RGA: Removing learned reliability weighting and pooling PIE observations directly raises average MAE to 11.99°, supporting the value of learned reliability grading.

• RGA behaves as a reliability module: The low-RGA regions overlap more with GT-identified unreliable PIEs than with reliable ones, and the RGA weight histogram shows unreliable PIEs receive lower weights on average than reliable PIEs.

• Qualitative behavior: EventPS-FCN and EventPS-CNN often recover coarse object shape but show large local errors, especially on high-curvature regions and real captures; the direct solver produces noisy normals because it solves each pixel independently and cannot reject unreliable event pairs; PIE-PS gives smoother and more accurate normals across the four settings.

• Not compared: PS-EIP is omitted from quantitative comparison because its code and benchmark data are unavailable, and EventUPS has no public code. The paper also states that PIE-PS was additionally evaluated under three CeleX SDK sensitivity settings, with statistics reported in the supplementary material (not in the truncated main text).

Methodology in Plain English

The researchers begin with the physics of how an event camera fires. Under a distant directional light, the image irradiance at a pixel is proportional to the dot product of the surface normal and the light direction. Event cameras report an event when the accumulated change in the log of that irradiance reaches a contrast threshold. Taking the time derivative removes all the constant unknowns (gain, light intensity, albedo) and leaves a clean ratio: the normal dotted with the light's velocity, divided by the normal dotted with the light direction itself.

Matching this theoretical derivative against what the event camera actually measures gives a per-pixel objective that a plain numerical optimizer can solve — this is the authors' "direct solver." But it needs to know the contrast threshold, needs enough events per pixel, and solves each pixel in isolation. The paper's core move is to avoid the threshold entirely.

Instead of using one event as a measurement, they pair two adjacent events at the same pixel and record the pair together with the light directions at the two timestamps. The feature attached to this pair is simply the signed event rate: the second event's polarity divided by the time between the events. This quantity retains timing and polarity, and because the threshold appears only as an unknown scale factor, the network can learn around it.

Each of these pairs becomes a node in a graph. Nodes are connected if they are nearby in image space and close in time. Each node's base feature combines the signed rate with the two light directions, and edges carry relative position-and-time information. A spline graph convolution lets each pair borrow context from neighboring pairs, so information is shared across space and time rather than solved pixel by pixel.

Because some pairs are unreliable — due to local appearance, illumination geometry, or sensor noise — a Reliability-Grading Attention module reads each pixel's time-ordered sequence of pair embeddings with a small transformer-like scorer and outputs a sigmoid confidence weight per observation. The paper motivates this with a discrete pairwise constraint (derived by exponentiating the log-irradiance relation) whose violation indicates an unreliable pair; ground-truth normals are used only to label unreliable pairs for diagnostics, never as network inputs.

Weighted embeddings are pooled into a per-pixel feature (a weighted mean plus a weighted element-wise max), a lightweight pixel aggregator refines features over neighboring valid pixels, and a normal head outputs a unit-length normal. Training uses direct supervision — the mean angular error between predicted and ground-truth normals.

Implementation specifics: PyTorch for the network, NumPy for the direct solver (exact SVD initialization, five IRLS iterations, two tangent-plane Gauss-Newton steps, and pixels with fewer than five PIE observations are not evaluated). The graph uses a 7-D node feature, 64-D PIE embedding, and 128-D hidden graph feature, with custom CUDA kernels for neighborhood search, a spatial radius of 0.05, a full normalized temporal window (Δtmax = 1), time scale γ = 0.3, and a two-layer transformer with four attention heads and 64-D hidden feature for RGA. Training used Adam for 200 epochs on a single NVIDIA H100 PCIe GPU, taking about 3.4 hours.

Data: PIE-Sim follows the EventPS simulation pipeline using Blobby and Sculpture objects with random poses and appearances, ESIM-simulated event streams under circle, hypotrochoid, and interpolated DiLiGenT light trajectories, with independently sampled contrast thresholds and noise per scene. Real data comes from three published EventPS scenes plus three 3D-printed objects (Bunny, Rectangle, Flower) captured with a prototype rig: a CeleX-V event camera at 1280 × 800, a Zhonglian Kechuang TM5028MP12 macro lens (50 mm focal length, 1.1-inch format, C-mount), a programmable ring light with 60 WS2812 5050 RGB LED beads driven by an Arduino Uno R3, and cross-modal alignment of CAD ground truth via MeshLab's mutual information registration filter.

Why This Matters

Research impact. The paper reframes event-based photometric stereo away from per-pixel analytical constraints toward a pairwise, threshold-free representation. It shows that the unknown contrast threshold — a long-standing nuisance parameter in event-based reconstruction — can be sidestepped by pairing adjacent events and learning around the unknown scale, rather than calibrating or estimating it. The negative result is also informative: physics alone, expressed as the direct solver, is not enough, and learning is needed to handle uneven event density, unreliable observations, and spatial context. The released PIE-Sim dataset, benchmarked against EventPS, DiLiGenT-Sim, and real captures, provides a comparison point for future work.

Real-world applications.

• Content creation and 3D asset scanning, where fine surface detail matters and lighting can be controlled via programmable sources like the ring light used here.
• Industrial inspection of manufactured or 3D-printed parts, where the captured Bunny, Rectangle, and Flower objects are representative targets and high dynamic range helps with shiny or high-contrast surfaces.
• Augmented reality, where dense, accurate normals are needed to integrate virtual objects with real shading.
• High-speed capture scenarios, such as fast-moving or rapidly lit subjects, where a frame-based camera would be limited by frame rate, motion blur, and dynamic range.

Industry relevance. Event cameras are commercially available sensors, and the paper reports practical hardware (a CeleX-V camera, a commodity C-mount macro lens, and a low-cost WS2812 LED ring driven by an Arduino). The method requires only event pairs and known light directions at test time, which keeps the runtime pipeline simple. The stated requirement of calibrated light directions is a real deployment constraint, but it is compatible with fixed rigs and calibrated light stages of the kind used for industrial scanning. The work was supported by the National Natural Science Foundation of China (Grant Nos. 62572212 and 62506084), the Science and Technology Development Plan of Jilin Province (Grant No. 20260203049SF), the Fundamental Research Funds for the Central Universities, and the High-Level Talents Innovation Team Project of the Guangdong-Macao In-Depth Cooperation Zone in Hengqin (Grant No. 2630004018939).

Future Directions

• Uncalibrated lighting. The authors state explicitly that extending the model to uncalibrated lighting is planned future work, removing the current requirement for known light directions. (EventUPS is cited as prior work that already addresses the uncalibrated setting, so this is a competitive direction.)
• More general reflectance. The paper notes that strong specular highlights and complex inter-reflections reduce PIE reliability and degrade normal estimation; extending to more general reflectance is named as future work.
• Better handling of sparse event regions. The direct solver is not evaluated on pixels with fewer than five PIE observations, and per-object errors such as Sculpture/Circle at 17.17° suggest that event density and local geometry still matter. How PIE-GNN performs under very low event counts is an open question.
• Threshold and sensor-setting sensitivity. The paper reports that PIE-PS was evaluated under three CeleX SDK sensitivity settings with details deferred to the supplementary material. How stable the learned model is across sensor sensitivity settings, and whether contrast-threshold variability across pixels can be exploited rather than avoided, are open questions the paper raises but does not resolve in the main text.

Target Audience

Researchers and graduate students working on event-based vision, photometric stereo, or computational photography who already know the basics of surface normal estimation and are comfortable with graph neural networks and attention. It is also relevant to practitioners building calibrated 3D scanning or inspection rigs with event cameras, and to anyone looking for a benchmark (PIE-Sim, DiLiGenT-Sim, and the captured real scenes) for comparing event-based normal reconstruction methods. Readers seeking an introductory treatment of photometric stereo or event cameras will find the derivations dense; the paper is advanced rather than beginner-friendly.

Authors’ abstract

Event cameras record asynchronous log-image-irradiance changes with microsecond latency and high dynamic range. These properties are useful for photometric stereo under moving illumination, but raw events are sparse and depend on an unknown contrast threshold. We start from the event trigger model and derive a physical relation between adjacent events, light motion, and surface normals. This relation gives a direct physics-only solver, but the solver needs the threshold, enough events at each pixel, and independent per-pixel optimization. To address these limits, we introduce PIE-PS, a learning-based framework for dense surface normal reconstruction from raw event streams and known lighting. We form Physical Irradiance Events (PIEs) by pairing two adjacent events at the same pixel with their corresponding light directions. Each PIE provides a Physical Irradiance Event Feature (PIEF), defined as the signed event rate. PIEF does not require the unknown contrast threshold. To share spatial and temporal context across nearby PIEs, we introduce PIE-GNN, which treats each PIE as a graph node and encodes it with its light-pair geometry. Since the reliability of PIE observations can vary with local appearance, illumination geometry, and sensor noise, Reliability-Grading Attention (RGA) predicts reliability weights to down-weight unreliable PIEs. Pixel aggregation then produces dense normals. Experiments on synthetic and real data show that PIE-PS outperforms prior event-based photometric stereo methods and the direct solver baseline.