4D Reconstruction with Events & RGB

AI summary

Overview

Research area: Computer vision — feed-forward 4D (dynamic 3D-over-time) scene reconstruction, and multi-modal sensing that combines standard RGB frames with event-camera streams.

Technical level: Intermediate. The method is a deliberately minimal modification of an existing model (MonST3R), so the core idea is easy to grasp, but the evaluation metrics (ATE, RPE, Abs Rel, δ<1.25) assume familiarity with 3D vision benchmarking.

Scope: A preliminary workshop study (accepted at the NeVi 2026 workshop at ECCV 2026) testing whether a lightweight event-voxel input branch added to an RGB-only 4D reconstruction pipeline improves camera pose and video depth accuracy, particularly under fast camera and object motion.

What This Paper Is About

Reconstructing a dynamic 3D scene over time from a single moving camera is important for robotics and AR/VR, but state-of-the-art feed-forward methods such as MonST3R consume only RGB frames and therefore inherit conventional camera failures — motion blur and temporal aliasing when camera or objects move fast or non-uniformly. Event cameras report per-pixel brightness changes asynchronously at microsecond resolution and are largely immune to these artifacts. The paper asks a narrow, practical question: can event data be added to an existing RGB-based 4D reconstruction pipeline without a full architectural redesign, and does it help?

Key Contributions

1. A minimal event branch on MonST3R. The authors add an event-voxel input to MonST3R and leave the rest of the architecture unchanged: the 3-channel RGB image and a 5-channel event voxel grid are concatenated into an 8-channel tensor, and a single convolution layer projects it back to 3 channels so the pretrained ViT encoder can accept it.
2. A four-way input comparison on camera pose estimation (Sintel) and video depth estimation (Sintel, Bonn, KITTI): original RGB, RGB reconstructed from events via the HyperE2VID event-to-RGB model, event voxels alone, and concatenated RGB + event voxels.
3. A motion-sensitivity study on a subset of the PointOdyssey test set, splitting sequences into three camera-motion levels (Low, Medium, High, from ground-truth poses) and two object-motion levels (Low, High, from ground-truth depth variation), to show where fusion gains appear.
4. A repositioning of the event-fusion motivation relative to EAG3R, the first event-augmented 4D reconstruction method, which targets low-light conditions; this paper instead holds illumination aside and varies the motion regime.

Main Findings

• Best pose accuracy comes from fusion. On Sintel, RGB + Ev. Vox. achieves the lowest error on all three metrics: ATE mean 0.106, RPE trans mean 0.038, RPE rot mean 0.476, versus 0.108 / 0.042 / 0.732 for the RGB-only MonST3R baseline.
• Translating events to RGB hurts. The RGB (HyperE2VID) variant is worse than using the original RGB frames on every pose metric (0.132 / 0.067 / 0.767) and on depth across all three datasets, which the authors attribute to errors introduced during event-to-RGB conversion.
• Event-only is not enough. Event voxels alone give higher ATE (0.154) and RPE trans (0.076) than fusion, although its RPE rot (0.532) is lower than plain RGB's 0.732. In depth, the event-only variant is the worst of all four inputs on every dataset.
• Fusion gives the best depth on all three datasets. RGB + Ev. Vox. records Abs Rel / δ<1.25 of 0.331 / 58.1 on Sintel, 0.064 / 96.4 on Bonn, and 0.101 / 90.6 on KITTI, versus 0.345 / 56.2, 0.065 / 96.3, and 0.106 / 89.3 for RGB-only. The abstract summarizes this as a 3.4% average Abs Rel reduction across 3 benchmark datasets.
• Gains grow with motion. Fusion achieves lower Log RMSE than RGB alone at all three camera-motion levels, and the gap widens under high object motion, where it achieves a 0.327 lower Log RMSE than RGB alone.
• Non-uniform camera motion is the hard case. Predicted trajectories on the Sintel sequences shaman_3 and ambush_4 show that smooth, uniform camera motion is estimated more accurately, while non-uniform motion accumulates error over time.

Methodology in Plain English

The authors start from MonST3R, a feed-forward model that predicts per-frame depth and camera pose from video without per-scene optimization, and add one extra input channel group. Events are converted into a "voxel grid" — a stack of B = 5 temporal bins that records, per pixel and per time bin, whether brightness increased or decreased. Each view therefore carries 3 RGB channels plus 5 event channels; one convolution layer compresses those 8 channels to 3 so the pretrained Siamese ViT encoder can process them exactly as before, and the original decoder and prediction heads are untouched. Frames are processed in a sliding window to produce per-window pointmaps, which are aligned into a single dynamic point cloud using camera intrinsics and extrinsics estimated from overlapping windows.

Because the benchmark datasets contain only RGB video, event streams are synthesized with the V2E simulator: it interpolates intermediate frames to raise temporal resolution, generates events from brightness changes between them, and accumulates them into voxel grids. The variants are retrained on PointOdyssey, TartanAir, and Spring augmented with these synthetic events, with all other training settings identical to the original MonST3R implementation. Evaluation uses Sintel, Bonn, KITTI, and PointOdyssey with the standard 4D reconstruction metrics used by MonST3R.

Why This Matters

Impact on research. The paper isolates a cheap, non-invasive change and shows measurable benefit, which argues that motion robustness — not just illumination — is a productive axis for event-camera fusion in 4D reconstruction. It also provides a clean negative result: event-to-RGB translation is worse than direct feature-level fusion, and event-only input is worse still, supporting the claim that RGB supplies spatial structure while events supply motion cues.

Real-world applications:

• AR/VR content creation, where fast head or object motion degrades capture quality.
• Robotics, where a single moving camera must track scene geometry during rapid or uneven motion.
• Autonomous driving, where other vehicles and the ego-camera move quickly and non-uniformly.
• Digital twins, which require accurate time-varying geometry of a scene.

Industry relevance. The approach requires no architectural redesign of an existing model, so it suggests a retrofit path for systems already built on RGB-based reconstruction. That matters for products where event sensors could be added alongside a conventional camera rather than replacing the software stack.

Future Directions

• Event-specific encoders. Events are currently encoded as dense voxel grids and pushed through an RGB backbone; Spiking Neural Networks or event transformers could extract richer motion cues from the sparse, asynchronous signal.
• Adaptive multi-modal fusion. Concatenation treats both modalities uniformly regardless of local motion; cross-attention or gating could weight RGB and events according to local motion, since events are motion-adaptive while RGB gives uniform spatial coverage.
• Newer 4D base models. MonST3R is no longer the most accurate feed-forward 4D reconstruction model; repeating the extension on VGGT or PAGE4D would test whether the benefit generalizes beyond one backbone.
• Real event data. All event streams here are generated by the V2E simulator because Sintel, Bonn, KITTI, and PointOdyssey do not natively contain events; validating on real event-camera captures is stated as necessary before drawing deployment conclusions. The paper also does not report a direct comparison against EAG3R or any illumination-varying evaluation.

Target Audience

Researchers and graduate students working on 3D/4D reconstruction, event-based vision, or multi-modal sensor fusion, particularly those interested in robustness to fast motion. It is also useful for robotics and AR/VR practitioners evaluating whether event cameras are worth adding to an existing RGB pipeline, and for readers who want a concise example of a minimal, controlled ablation study rather than a full new architecture.

Authors’ abstract

Reconstructing dynamic 3D scenes over time (4D reconstruction) from a single moving camera is central to robotics and AR/VR applications, yet state-of the-art feed-forward methods rely solely on RGB frames and therefore inherit the failure modes of conventional cameras, including motion blur and temporal aliasing under fast or non-uniform camera and object motion. Event cameras report per-pixel brightness changes asynchronously at microsecond resolution, offering a complementary signal that is largely immune to these artifacts, but whether this modality can be incorporated into existing RGB based 4D reconstruction pipelines without a full architectural redesign remains an open question. In this preliminary study, we investigate a minimal extension: we augment Monst3R, an existing RGB-only 4D reconstruction model, with an additional event-camera input branch, leaving the rest of the architecture unchanged. Across 3 benchmark datasets, this simple RGB-event fusion reduces Abs Rel by 3.4% on average relative to the RGB-only Monst3R baseline, and a sensitivity analysis shows that the improvement grows with the magnitude of camera and object motion, with the largest gains observed under high object motion conditions where RGB-only reconstruction degrades most. These early results suggest that even a lightweight, non-invasive fusion of event data can meaningfully improve 4D reconstruction robustness, motivating a more thorough architectural investigation in future work.