LASER: Layer-wise Scale Alignment for Training-Free Streaming 4D Reconstruction

AI summary

Overview

Research area: Computer vision — feed-forward 3D/4D scene reconstruction, streaming (online) reconstruction, and camera pose estimation.

Technical level: Advanced. The paper assumes familiarity with feed-forward reconstruction transformers (DUSt3R, VGGT, π³), Sim(3) alignment, SLAM-style trajectory evaluation metrics (ATE, RPE), and point-map accuracy/completeness metrics.

Scope in one sentence: LASER is a training-free wrapper that turns an existing offline feed-forward reconstruction model into a streaming system by registering overlapping temporal windows with a proposed layer-wise scale alignment (LSA) that corrects depth-dependent scale drift.

The paper is listed as arXiv:2512.13680v3 [cs.CV], dated 28 Mar 2026, under CC BY 4.0. Authors are affiliated with Mitsubishi Electric Research Laboratories, Northeastern University, and an independent researcher affiliation; the paper notes equal contribution. Project website: https://neu-vi.github.io/LASER/

What This Paper Is About

Modern feed-forward reconstruction models such as VGGT and π³ produce high-quality point maps and camera poses, but they must see all frames at once, so their memory grows quadratically and they cannot process a live video stream. Existing streaming alternatives fix this by retraining or distillation, which is expensive and can discard the strong geometric priors the offline models already learned.

LASER instead asks whether an offline model can be converted to streaming use with no retraining at all. The core obstacle the authors identify is that a single global Sim(3) similarity transform between consecutive windows is not enough: because of monocular scale ambiguity, different depth layers in a scene (for example foreground versus background) scale inconsistently from window to window, producing distortions and metric drift when the windows are fused.

Key Contributions

1. A training-free streaming framework (LASER). It converts offline reconstruction models into streaming systems without retraining, demonstrated with both VGGT and π³ backbones, using a sliding-window scheme with overlapping windows and incremental submap registration.

2. Identification of the layer depth misalignment problem. The authors show that simple Sim(3) alignment — as used by concurrent work VGGT-Long — fails because monocular scale ambiguity causes relative depth scales of different scene layers to vary inconsistently between windows, particularly under limited camera translation.

3. Layer-wise Scale Alignment (LSA). A geometry-driven refinement that segments depth predictions into discrete layers, estimates per-layer scale factors from point-wise correspondences via robust IRLS, and propagates and aggregates those scales across both adjacent windows and timestamps using a directed layer graph.

4. State-of-the-art streaming results with high efficiency. The paper reports streaming pose estimation and reconstruction improvements of −68.6% ATE on Sintel pose estimation and −63.9% Acc on 7-Scenes reconstruction compared with the previous best, while running at 14 FPS with 6 GB peak runtime memory on an RTX A6000 GPU, scaling to kilometer-scale sequences.

Main Findings

• Sim(3)-only alignment is insufficient. After a global Sim(3) registration, surfaces at different depths still show layer-wise scale inconsistency — foreground regions appear over- or under-scaled relative to background structures across consecutive windows. This anisotropic scaling along depth accumulates over time into visible distortion and metric drift.

• LSA is the decisive component. With π³ as backbone on video depth estimation, removing LSA degrades Sintel Abs Rel from 0.247 to 0.328 and δ<1.25 from 68.8 to 51.4; on Bonn it degrades Abs Rel from 0.048 to 0.123 and δ<1.25 from 97.4 to 85.6. The paper notes LSA does not affect camera pose estimation.

• Replacing the segmentation algorithm does not help. Substituting the efficient segmentation algorithm with SAM 2 gives Sintel Abs Rel 0.251 / δ<1.25 67.8 and Bonn 0.051 / 97.4, i.e. no accuracy gain despite trading speed for better segmentation.

• Temporal propagation matters. Removing propagation through the intra-window edges (E_intra) yields Sintel 0.261 / 64.7 and Bonn 0.05 / 97.1, which the authors attribute to ignoring temporal relationships and preventing scale updates in non-overlapping frames.

• Depth estimation. On Sintel, Bonn and KITTI, π³+Ours achieves Abs Rel 0.247 / 0.048 / 0.054 and δ<1.25 68.8 / 97.4 / 98.3; VGGT+Ours achieves 0.297 / 0.07 / 0.116 and 64.6 / 92.6 / 88.4. LASER reaches the lowest Abs Rel across all three datasets compared with streaming baselines such as CUT3R, StreamVGGT and STream3Rβ, and maintains its offline backbones' performance while streaming.

• Small-scale pose estimation. π³+Ours is best in almost all metrics: Sintel ATE 0.061 / RPE_trans 0.028 / RPE_rot 0.249, ScanNet 0.031 / 0.012 / 0.339, TUM 0.016 / 0.009 / 0.308. VGGT+Ours ranks second across all metrics (Sintel 0.131 / 0.053 / 0.398; ScanNet 0.035 / 0.014 / 0.354; TUM 0.013 / 0.010 / 0.306).

• Large-scale KITTI Odometry. LASER (π³) reaches Avg. ATE 24.17 and Avg.* 14.42 (Avg.* excludes the high-speed Seq. 01), the second-lowest mean ATE among all methods, versus 27.64 / 18.28 for VGGT-Long and 30.72 / 16.45 for π³-Long. The authors report a 12–21% reduction in average ATE over these training-free concurrent methods. Offline VGGT and π³ run out of memory on all listed sequences, MASt3R-SLAM loses tracking, and CUT3R and Fast3R are reported as OOM on most sequences.

• Point map estimation. On 7-Scenes and NRGBD, π³+Ours reports Acc mean/median 0.013 / 0.005 and Comp 0.017 / 0.006 on 7-Scenes, and Acc 0.020 / 0.010 and Comp 0.012 / 0.004 on NRGBD, improving Acc and Comp over prior streaming baselines. The authors state that its Normal Consistency can be slightly lower than StreamVGGT or STream3Rβ because of the π³ backbone's limited surface-normal fidelity, though LASER still improves NC over π³ (0.607 vs 0.598 mean on 7-Scenes).

• Efficiency. With π³ as the offline model, LASER runs at approximately 14.2 FPS with only 6 GB peak memory; with VGGT it runs at approximately 10.9 FPS with 10 GB peak memory, on an RTX A6000 GPU.

• Hyperparameter robustness. Performance is stable across IoU thresholds τ from 0.2 to 0.6 and varied window sizes L; the defaults L = 20 and τ = 0.3 are chosen as a good balance. The paper reports a competitive reconstruction quality against offline models of 0.013 versus 0.011 mean accuracy on 7-Scenes while enabling online processing.

Methodology in Plain English

The system processes a video as a sequence of overlapping temporal windows, each containing L consecutive frames and sharing O frames with the next window (so window start indices advance by L − O). Each window is fed, unmodified and frozen, to an offline feed-forward reconstructor such as VGGT or π³, which returns dense per-frame 3D point maps, camera poses, and per-pixel confidence scores. These are combined into a local submap in the window's own coordinate frame.

The local submap is then registered to the global map built so far using a similarity transform in Sim(3) space. A global scale factor is estimated first with a robust IRLS optimization that enforces a shared metric across the two adjacent windows, and rotation and translation follow from the Kabsch algorithm using scaled camera anchors. This part works on camera pose, but the authors show it is not enough for the geometry.

The remaining problem is that a single global scale assumes all parts of the scene scale uniformly, which monocular reconstruction cannot guarantee, especially when the camera barely translates. LASER therefore adds Layer-wise Scale Alignment. It takes the registered point map for a frame, reads off its Z coordinates as a pseudo-depth map, and segments that depth map into spatially coherent depth layers using an efficient segmentation algorithm. Each layer is treated as a node in a directed layer graph: inter-window edges connect layers that overlap (by IoU above a threshold τ = 0.3) between the previous and current window at shared timestamps, and intra-window edges connect the same layer across adjacent frames within a window.

Per-layer scale factors are first optimized on the inter-window edges by solving, with IRLS and a Huber loss, for the scale that best maps depth values in one window's layer onto the corresponding values in the other window's layer. These scales are then propagated along the graph and combined as a weighted average, with each edge weighted by the IoU of the two connected layers. Finally, each layer's scale is applied back to the pixels it contains, correcting the point map. The whole procedure is stated as Algorithm 1 in the paper.

Why This Matters

Impact on research. The paper argues that when deep models provide strong local geometry, classical layer-based geometric reasoning can unify their outputs into consistent long-range reconstructions without any retraining. That is a different research direction from the dominant one of designing new streaming architectures and training them from scratch or by distillation. It also gives a concrete diagnosis (layer depth misalignment) for why naive Sim(3) window alignment drifts, and it is backbone-agnostic: the same framework is demonstrated on two different offline models, and the authors argue it can immediately absorb future, stronger offline models for free.

Real-world applications:

• Autonomous driving and mobile robotics, where a vehicle must consume video causally and produce globally consistent trajectories over kilometer-long routes — exactly the KITTI Odometry setting evaluated here.
• AR/VR headsets, which need low-latency, memory-bounded 3D scene tracking rather than batch reprocessing of all frames.
• Large-scale mapping and digital cultural preservation, where offline pipelines are typically applied to long captures and would benefit from online incremental fusion.
• Video post-production and content creation (the paper shows qualitative results on DAVIS and

Authors’ abstract

Recent feed-forward reconstruction models like VGGT and $π^3$ achieve impressive reconstruction quality but cannot process streaming videos due to quadratic memory complexity, limiting their practical deployment. While existing streaming methods address this through learned memory mechanisms or causal attention, they require extensive retraining and may not fully leverage the strong geometric priors of state-of-the-art offline models. We propose LASER, a training-free framework that converts an offline reconstruction model into a streaming system by aligning predictions across consecutive temporal windows. We observe that simple similarity transformation ($\mathrm{Sim}(3)$) alignment fails due to layer depth misalignment: monocular scale ambiguity causes relative depth scales of different scene layers to vary inconsistently between windows. To address this, we introduce layer-wise scale alignment, which segments depth predictions into discrete layers, computes per-layer scale factors, and propagates them across both adjacent windows and timestamps. Extensive experiments show that LASER achieves state-of-the-art performance on camera pose estimation and point map reconstruction %quality with offline models while operating at 14 FPS with 6 GB peak memory on a RTX A6000 GPU, enabling practical deployment for kilometer-scale streaming videos. Project website: https://neu-vi.github.io/LASER/