Solv3R: 3D Looped Transformer as a Multi-view Solver

AI summary

Overview

• Research area: Computer Vision — multi-view 3D reconstruction, geometric foundation models, and learned iterative solvers.
• Technical level: Advanced (assumes familiarity with transformers, attention, camera pose estimation, and structure-from-motion).
• Scope: The paper reformulates multi-view 3D reconstruction as a learned, iteration-invariant "solver" that can continue refining an existing reconstruction, absorb new views, and reuse cached geometric context across four different reconstruction settings.

What This Paper Is About

Most recent geometric foundation models reconstruct a scene in a single forward pass, or maintain a recurrent memory that only accepts frames arriving in sequence. That makes it hard to do what classical structure-from-motion does easily: keep refining an existing map, add new images later, or reuse a solved map as context. The authors ask what a network would need in order to behave like a classical iterative solver — one whose update rule does not know (or care) which iteration it is on — and they build and train such a model, called Solv3R.

Key Contributions

1. A stationary (timestep-invariant) geometric update operator. Solv3R replaces DéjàView's step-conditioned gates with learned constant gates and rescales the updated state to a fixed RMS norm, so the same transformer block can be applied indefinitely without knowing the iteration index. The initial image encoding is injected at every solver call, as a classical solver re-reads its constraints.
2. Training for solver dynamics, not just accuracy. The operator is trained on heterogeneous states — varying numbers of solver calls, Gaussian perturbations, and partial resets of views to their initial encoding — so that solved states stay fixed (stability), perturbed states are corrected, and views with different amounts of refinement can be updated jointly (continuation).
3. Reusable geometric context via a single KV cache. Because updates no longer depend on refinement stage, one key-value cache built from solved map views can be reused across all solver calls, reducing KV cache storage by 16× compared with stage-specific caching in DéjàView.
4. One model, four settings, no task-specific modules. A single learned solver (117M parameters, same as DéjàView) is evaluated on offline reconstruction, incremental reconstruction, map-based relocalization, and streaming, without streaming-specific training or a separate optimization backend.

Main Findings

• Stability under continued computation: Solv3R maintains reconstruction accuracy as its relative state update (‖z^{k+1} − z^k‖₂ / ‖z^k‖₂) shrinks, whereas DéjàView loses accuracy when iterated beyond its trained range. DéjàView is trained with up to 16 steps in the comparison shown.
• Adaptive stopping works: Using a relative-update threshold of γ = 0.02, 144 of 148 multi-view reconstruction windows meet the stopping criterion within 64 calls, and 143 of those (99.3%) have camera rotation errors below 5°. The 144 windows stop after 13 to 32 calls, 17.5 on average, with harder windows taking more calls.
• Correction from perturbations: After perturbing the state at call 8 (relative Gaussian noise, fixed-scale Gaussian noise, or resetting views), then refining for eight more calls, Solv3R recovers accurate camera poses under every perturbation type; DéjàView does not.
• Reusing prior states beats restarting: In a sliding-window warm-start setting on a KITTI sequence with four solver calls per new frame, Solv3R stays stable and reaches accurate camera estimates in fewer solver calls than restarting from scratch, while DéjàView diverges.
• Insensitivity to mixed refinement progress: When each view is initialized from a randomly sampled reconstruction stage (0–16 calls), Solv3R produces nearly identical camera rotations across independent trials, with a mean difference of only 0.08°; DéjàView remains sensitive.
• Pipelining boosts throughput: Advancing the sliding window after every solver call (rather than waiting for convergence) still refines cameras correctly. Once the first 16 calls are made, the delay between estimates per frame becomes a single call — the same latency as DéjàView at 16 calls, but 16× more frames per second (10× end to end).
• Competitive offline accuracy at a fraction of the size: With 117M parameters and 76.5 TFLOPs (fixed, 16 calls) or 91.3 TFLOPs (adaptive), Solv3R is compared against models from 411M to 1.32B parameters and up to 1150.1 TFLOPs. Adaptive stopping improves nearly all reconstruction metrics over fixed-budget inference with only 3–4 additional solver calls on average. TFLOPs are measured for 24 views across DTU, ETH3D, 7-Scenes, ScanNet++, and nuScenes.
• Incremental reconstruction revises old views: Solving four views for eight calls, then introducing twelve new views (windows from DL3DV and ScanNet++), Solv3R reconstructs the new views while improving camera estimates of the original views — an improvement not explained by extra computation, since a no-join control with the same number of calls does not show it. Expanding from 8 to 64 views reaches accuracy comparable to re-solving from scratch with fewer solver calls and lower runtime.
• Relocalization with the smallest cache: On 7-Scenes and Cambridge Landmarks, Solv3R uses 2.86 MB per map view and 51 ms per query, achieving 72.8% recall within 5 cm/5° on 7-Scenes (median error 3.58/0.95) and 24.9% within 25 cm/2° on Cambridge (median 63.5/0.74). DéjàView with a single cache collapses to 0.0% recall on both, while DéjàView with all-step caches uses 45.8 MB per view for 28.3% and 0.7%.
• Streaming without streaming-specific training: Solv3R runs at 95 ms per frame with 5.0 GB peak GPU memory, with ATE of .157 (Sintel), .015 (TUM-dyn), and .077 (ScanNet), competitive with recurrent-state and geometric-context streaming baselines. Predictions are reported at frame arrival for fair comparison.
• Ablations show stability alone is not enough: Removing input injection, state normalization, or random-depth pre-calls each causes accuracy to collapse at 1024 calls (AUC@5 of 0.0 for all three). Removing perturbation training largely preserves long-horizon accuracy (61.7 at 1024 calls) but degrades joining and cached-context inference (4.24° and 3.51° error), showing corrective behavior requires explicit training. Step-conditioned gates are sensitive to the planned horizon P.

Methodology in Plain English

The authors start from DéjàView, an existing looped transformer that repeatedly applies one shared transformer block to a multi-view latent state and decodes camera poses and depth from it. Their key move is to strip out everything that tells the block which iteration it is on. DéjàView scaled each sub-block's channels with gates computed from the step index; Solv3R replaces these with learned constant gates, initialized by averaging DéjàView's 16 step-specific gate values. It also injects the original image encoding at every call (so the network keeps seeing the constraints, not just its own evolving guess) and rescales the updated state to a fixed RMS norm so magnitudes don't drift over hundreds of calls.

To make this stationary operator actually converge, they train it the way one would train a solver. Each training iteration samples 4–64 views, runs 16–40 solver calls without gradients, then does 8 differentiable calls supervised only at the final one — with the supervised call varying between the 24th and 48th call, so the network is never told which call it is being graded on. Before the differentiable calls, with probability 0.15, they either add noise to the state or reset some views to their initial encoding, simulating errors and newly arriving frames. Supervision covers camera poses, ray maps, and depths, plus a distillation loss back to DéjàView's 16-call predictions to preserve pretrained priors (λ_dist = 0.5). Training uses AdamW at 2×10⁻⁵ with an EMA of weights, for roughly one week on 8× H100 GPUs.

The authors then treat a simple Jacobi linear solver as a reference model, extracting three properties — stability at a fixed point, correction from perturbation, and continuation when the system grows — and design experiments to test each one directly.

Why This Matters

Reconstruction in practice is rarely a single batch: image collections grow, videos stream in, robots revisit mapped spaces. Solv3R shows that a single small model can handle all of these regimes by treating reconstruction as an ongoing solve rather than a one-shot prediction, which removes the need for separate streaming architectures, task-specific heads, or a classical optimization back-end. It also gives a concrete design principle — the update rule must be independent of iteration count — for anyone building iterative learned systems beyond 3D vision.

Real-world applications:

• Robot and vehicle mapping: incremental map extension and relocalization against a frozen, compactly cached map for navigation and AR.
• Large-scale photogrammetry: progressively adding images to a reconstruction without re-solving the whole set from scratch.
• Streaming video reconstruction: monocular or multi-camera depth and trajectory estimation with low memory (5.0 GB) and 95 ms per frame.
• On-device / embedded 3D capture: the 117M-parameter model and 2.86 MB-per-view cache fit budgets that 1B+ parameter geometric foundation models do not.

Industry relevance: the work comes from UBC, Wayve Labs, SFU, University of Toronto, and Sony, and the compact model plus cheap KV caching are directly relevant to mobile, automotive, and XR products where memory and latency dominate. Code and models are slated for public release.

Future Directions

• Extending the solver beyond geometry: the authors suggest the same "maintain and refine an evolving state" view could apply to vision-language models, robotics, and world modeling.
• Longer-horizon and larger-scale stability: the paper notes that input injection, state normalization, and random-depth pre-calls are each essential at 1024 calls, leaving open how far the stationary operator can be pushed before new mechanisms are needed.
• Better corrective training: removing perturbation training costs little in long-horizon accuracy but much in joining and cached-context inference, suggesting more principled perturbation or denoising schemes could improve reuse further.
• Closing the fixed-budget gap: the ablation shows that with sufficient training, Solv3R recovers the initial drop in fixed-budget accuracy while keeping solver dynamics — an open question is how to get both immediately rather than after extended training.

Target Audience

Researchers and engineers working on 3D reconstruction, SLAM, and geometric foundation models who want to understand how to build iterative, reusable learned solvers; practitioners who need incremental or streaming mapping on constrained hardware; and machine learning researchers interested in looped/weight-shared transformers, fixed-point behavior, and test-time computation outside of language modeling.

Authors’ abstract

Classical structure-from-motion reconstruction proceeds iteratively, refining camera poses and scene geometry from a current estimate. This allows maps to easily expand without recomputing already computed views, or geometry estimates to improve as more observations are provided. Most geometric foundation models, however, predict entire geometry in a single forward pass or update a recurrent state as frames arrive sequentially, making this expansion and refinement non-trivial. We introduce Solv3R, a looped transformer that learns to continue solving multi-view geometry from its current state. Importantly, its update operator is independent of the iteration count, allowing previously refined and newly introduced views to be jointly updated regardless of how ``solved'' they are. We train the operator to preserve accurate reconstructions under continued iteration and correct inaccurate states, enabling refinement of fixed image collections and integration of new observations. Representations of solved views can also be cached once and reused as geometric context across iterations, enabling compact and efficient key-value caching. We analyze the solver dynamics and compare it with the state-of-the-art, demonstrating that a single learned solver supports offline reconstruction, incremental reconstruction, relocalization, and streaming without task-specific training or modules. The code and models will be made publicly available at https://ubc-vision.github.io/solv3r/.