MESSENGER: Memory-Enhanced Sequential Scene Flow Estimation via Autoregressive Next-Frame Forecasting

AI summary

Overview

Research area: Computer vision, specifically 3D scene flow estimation and forecasting from LiDAR point cloud sequences.

Technical level: Advanced. The paper assumes familiarity with point cloud learning, diffusion decoders, attention mechanisms, and autoregressive sequence modeling.

Scope: The paper introduces MESSENGER, a memory-enhanced autoregressive framework that jointly estimates scene flow between observed LiDAR frames and extrapolates future scene flow beyond them.

What This Paper Is About

Scene flow describes the dense 3D displacement of every point between consecutive LiDAR scans. Earlier methods only compare two frames at a time, so they miss long-term motion patterns and cannot predict motion beyond what they have seen, while newer multi-frame methods grow expensive as more frames are added and degrade when asked to forecast far into the future. MESSENGER addresses both problems by storing a short history of past flow estimates and hidden motion states in a memory buffer, then retrieving them to forecast the next frame's flow, with an uncertainty measure that downweights unreliable historical memories before they compound into long-horizon errors.

Key Contributions

1. A memory-enhanced autoregressive pipeline for joint sequential scene flow estimation and prediction, built on a next-frame forecasting mechanism rather than sequence-to-sequence processing.
2. A memory buffer that stores multi-frame history flows and flow-related hidden states at only the coarsest stage, which are correlated and retrieved via cross-attention to forecast the next-frame flow, strengthening long-term temporal consistency and future extrapolation.
3. An uncertainty-aware measurement module that quantifies per-point reliability from the discrepancy between retrieved historical motion and currently refined dynamics, and reweights memory fusion to filter outliers and mitigate compounding errors.
4. State-of-the-art results in both estimation and prediction, reducing EPE3D by 71.6% on nuScenes and 67.7% on Argoverse 2 relative to GenFlow3D in long-horizon future extrapolation.

Main Findings

• Estimation accuracy (3 input frames). On nuScenes, MESSENGER reaches EPE3D 0.0154, Acc3DS 0.9629, Out3D 0.3283 for the first flow and 0.0133 / 0.9767 / 0.2973 for the second, versus GenFlow3D at 0.0161 / 0.9580 / 0.3399 and 0.0138 / 0.9756 / 0.3071. On Argoverse 2, MESSENGER reports 0.0167 / 0.9691 / 0.1760 and 0.0156 / 0.9743 / 0.1712, versus GenFlow3D at 0.0175 / 0.9660 / 0.1823 and 0.0164 / 0.9711 / 0.1771.
• Small margins in short-horizon estimation. The paper states the discrepancy between GenFlow3D and MESSENGER is small with 3-frame input because temporal information is limited; the gap widens when input frames increase.
• Future prediction with H = 6 observed frames. On nuScenes, predicting the first future flow gives EPE3D 0.015 and Acc3DS 0.969 for MESSENGER versus 0.029 and 0.948 for GenFlow3D; at the fourth future flow it is 0.025 and 0.883 versus 0.088 and 0.754. On Argoverse 2 the fourth future flow gives 0.062 and 0.555 versus 0.192 and 0.130.
• Degradation with horizon length. GenFlow3D's Acc3DS on Argoverse 2 falls from 0.651 to 0.130 across the four forecast steps, while MESSENGER falls from 0.915 to 0.555.
• Latency. One forward pass takes 55.9 ms for MESSENGER, compared with 88.7 ms for GenFlow3D and 216.7 ms for DifFlow3D.
• GPU memory. A figure compares training GPU memory and shows MESSENGER's consumption is constant with respect to input frame length, unlike sequence-to-sequence stacking.
• Ablation, memory removal. Removing the memory buffer increases EPE3D to 0.088 with Acc3DS 0.754 in the 4th future prediction setting (the "w/o Memory" row, which matches GenFlow3D's 0.088 / 0.754).
• Ablation, uncertainty removal. Removing uncertainty raises EPE3D to 0.045 with Acc3DS 0.668; an MLP-based uncertainty construction gives 0.029 / 0.844, while the normalization-based version gives 0.025 / 0.883.
• Ablation, memory length. M = 3 gives 0.040 / 0.764, M = 5 gives 0.035 / 0.806, M = 6 gives 0.025 / 0.883, and M = 8 gives 0.031 / 0.823, so longer memory helps up to a point, then slightly degrades.
• Ablation, memory contents. Storing state, flow, and uncertainty together gives 0.025 / 0.883; removing state gives 0.029 / 0.834, removing flow gives 0.031 / 0.827, removing uncertainty gives 0.033 / 0.807, and removing both flow and uncertainty gives 0.066 / 0.415.
• Per-class errors. Dynamic Bucket-Normalized EPE for the first flow on nuScenes shows MESSENGER ahead of GenFlow3D for cars (0.0209 vs. 0.0234), other vehicles (0.0228 vs. 0.0257), pedestrians (0.0291 vs. 0.0353), and wheeled vulnerable road users (0.0321 vs. 0.0418).
• Three-way EPE. For the 4th future flow on nuScenes, MESSENGER reports foreground dynamic 0.051, foreground static 0.013, background static 0.024, class average 0.029, versus GenFlow3D's 0.137, 0.072, 0.086, and 0.098.
• Uncertainty visualization. The paper reports that learned uncertainty concentrates in dynamic regions such as moving cars, where retrieved memory flows are less reliable.

Methodology in Plain English

The system processes a LiDAR sequence one frame at a time. For each incoming frame it extracts a hierarchy of point features across L = 4 stages. At the coarsest stage only, the model keeps a memory buffer holding the last M = 6 historical coarse flows, the GRU-based hidden motion states that produced them, and a per-point reliability score for each.

When a new frame arrives, the network builds a query from the most recent hidden state and keys from all stacked history states, while the values combine both the explicit past flows and the implicit hidden states. Attention over these produces a forecast of the current frame's flow based purely on history, which serves as the initialization. Because this forecast ignores current geometry, it is then refined using the current frame's features, a cost volume between the previous and current frame features, and the updated hidden state. That refined coarse flow is pushed into the memory buffer in first-in-first-out order, and the oldest entry is dropped.

The uncertainty module computes the residual between the history-based forecast and the current refined flow. A large residual means history contradicted current evidence, so the point is flagged as unreliable. Residuals are mapped to a scalar between 0 and 1 by L2 normalization and clamping, and the resulting reliability factor reweights each historical flow during memory retrieval, so untrustworthy history is suppressed and the model leans on current correlation instead.

Finally, cascaded diffusion refinement decoders upsample and refine the coarse flow across the remaining stages. Training uses multi-scale supervised losses on both full flows and flow residuals, summed over both observed and future frames with stage weights of 0.02, 0.04, 0.08, and 0.16. Optimization uses AdamW with beta1 = 0.9 and beta2 = 0.999, a learning rate initialized at 0.001 and reduced by 0.7 every 40 epochs. The diffusion decoder uses DDIM with a total timestamp of 1000 and 1-step denoising at inference. For each frame, 8192 points are randomly sampled, and ground points and points farther than 35 m are filtered out.

Why This Matters

The paper reframes scene flow from a pairwise matching problem into a sequential forecasting problem, showing that explicitly stored motion memory plus reliability weighting can hold up over long prediction horizons where diffusion-based and sequence-to-sequence baselines drift. It is the first work the authors identify that combines history flows as memory priors with progressive next-frame flow forecasting in the 3D counterpart of optical flow.

Real-world applications listed or implied by the paper:

• Autonomous driving perception, where predicting future 3D motion of surrounding objects supports planning and safety.
• Motion segmentation of dynamic versus static scene elements.
• SLAM systems, where accurate 3D displacement aids localization and mapping in dynamic environments.
• Dynamic reconstruction and generation, since scene flow provides dense 4D motion cues usable in world models.
• Robotics and wheeled vulnerable road user safety, where the per-class analysis specifically targets pedestrians and cyclists.

Industry relevance centers on deployment constraints: MESSENGER's latency of 55.9 ms and constant GPU memory with respect to input frame length matter for real-time onboard systems, in contrast to the growing memory cost of sequence-to-sequence multi-frame architectures and the reliance of some prior methods (such as DeltaFlow) on ground truth rigid poses from odometry.

Future Directions

• Scaling to longer horizons. The paper states its forecasting works within a limited frame length, and suggests incorporating large-scale pretrained backbones such as Wan for long-horizon world modeling.
• Better handling of memory staleness. Performance degrades slightly from M = 6 to M = 8 because distant frames share scarce semantic similarity and act as outliers; how to select or compress memory more selectively is unresolved.
• Robustness to abrupt dynamics. Sudden non-rigid deformations, abrupt acceleration, and prolonged occlusion remain the failure modes the uncertainty module is designed to mitigate, and further improvement there is open.
• Integration with 4D world models. The authors propose that dense 4D scene flow could strengthen motion modeling in recent world models.

Target Audience

Researchers and graduate students working on 3D scene flow, point cloud sequence learning, autonomous driving perception, and video or 4D world modeling. It is also relevant to practitioners who need real-time LiDAR motion estimation under fixed memory budgets and to readers interested in memory-augmented autoregressive architectures for spatio-temporal forecasting.

Authors’ abstract

Scene flow can capture low-level 3D motion displacements in dynamic scenarios. Early pairwise estimators relying on instantaneous two-frame motion lack long-term temporal correlation and also struggle with poor extrapolation ability in future prediction. Although some recent methods attempt to explore multi-frame scene flow estimation in a sequence-to-sequence manner, they typically suffer from heavy computational overhead with increasing input frames and long-horizon prediction degradation due to ineffective motion propagation. To address these problems, we propose a novel memory-enhanced sequential scene flow pipeline, called MESSENGER. To sufficiently mine long-term temporal dependencies naturally within consecutive sequences, a memory buffer is designed by explicitly storing multiple history flow estimates and latent states. For each input frame, the temporally stored flows and states are correlated and retrieved to predict the current initialized flow in a next-frame forecasting manner. Furthermore, we develop an uncertainty-aware reweighting module to filter unreliable retrievals and mitigate accumulated errors. Extensive experiments on nuScenes and Argoverse 2 demonstrate state-of-the-art performance of our MESSENGER, reducing EPE3D by 71.6% on nuScenes and 67.7% on Argoverse 2 in long-horizon future extrapolation. This superiority can be attributed to our designed autoregressive forecasting paradigm, which naturally forces the network to progressively learn the next-frame distribution based on history observations. Code will be released at https://github.com/liujiuming123/Messenger.