Camera3R: Feed-Forward 3D Reconstruction from Heterogeneous Camera Models

AI summary

Overview

Research area: 3D computer vision — feed-forward multi-view 3D reconstruction and parameter-efficient model adaptation. Technical level: Advanced (assumes familiarity with transformer-based reconstruction backbones, LoRA, and SVD-based rank truncation). Scope: The paper proposes Camera3R, a mixture-of-LoRA-experts framework that extends perspective-pretrained feed-forward 3D reconstruction networks to fisheye and panoramic cameras, including mixed and unknown camera inputs, at 0.5% additional parameters.

What This Paper Is About

Feed-forward 3D reconstruction models (e.g., VGGT, MapAnything, π³) are trained on very large perspective-image datasets and predict camera poses and dense geometry in a single forward pass, but they break down on fisheye and panoramic images because those cameras have different fields of view and distortion profiles. The goal is to let one pretrained model ingest an arbitrary, possibly mixed and unlabeled stream of perspective, fisheye, and panoramic frames, while losing nothing on the perspective images it already handled well — without retraining from scratch.

Key Contributions

1. A mixture-of-experts extension in which each non-perspective camera type (fisheye, panoramic) gets its own set of LoRA adapters injected into the frozen perspective-pretrained backbone, with a learned camera router that selects the expert when camera metadata is unavailable.
2. An adaptive capacity allocation scheme: adapters are first trained at rank 64 for all layers, then compressed by a global-threshold truncated SVD so that capacity is redistributed toward the layer-expert pairs that need it, cutting parameters to 0.5% of the backbone.
3. State-of-the-art results on single-camera (fisheye, panoramic) and mixed-camera reconstruction across multiple indoor and outdoor datasets, demonstrated on three different backbones (VGGT, MapAnything, π³).
4. The authors' claim to be the first to enable joint feed-forward 3D reconstruction from heterogeneous perspective, fisheye, and panoramic images in a single forward pass, with backward compatibility on perspective inputs guaranteed because perspective frames bypass all adapters.

Main Findings

• Parameter and training efficiency: 0.5% additional parameters and less than a day of training adapt a foundational reconstruction model to non-perspective cameras, while fully preserving its original perspective performance.
• Compression is nearly free: after rank truncation, the adapter is approximately compressed by 22 times with only a slight performance drop.
• Router is essentially perfect: the lightweight CNN camera classifier converges in a few minutes and reaches 100% classification accuracy on the test sets, suggesting the three camera types are visually distinct enough to identify.
• Large gains over calibration tokens (CT): on ADT fisheye with the π³ backbone, Camera3R reaches RRA 0.990 / RTA 0.860 / AUC 0.610 / ATE 0.034 versus CT's 0.971 / 0.732 / 0.437 / 0.054. On Stanford2D3DS panoramic with VGGT, RRA goes from 0.457 (+CT) to 0.958 (+Ours), and on 360Loc with VGGT from 0.321 to 0.939 AUC.
• Strongest with π³: the paper reports the best overall results using the π³ backbone; on ScanNet++ fisheye it achieves RRA 0.997, RTA 0.968, AUC 0.870.
• Mixed cameras work: on KITTI360 with mixed perspective and fisheye, π³ alone scores RRA 0.200 / RTA 0.082 / AUC 0.010, while Camera3R lifts this to 1.000 / 0.881 / 0.753.
• Design ablation (Table 4, π³ point clouds): full finetuning and retraining both damage perspective performance (ADT CD degrades from 0.036 to 0.042 for finetune and to 0.748 for retrain), as do variants that give perspective its own expert (3PFP, 1PFP). Only the "1FP" and "2FP" designs, which leave perspective routed through the frozen backbone, exactly preserve the original perspective numbers; separating fisheye and panoramic experts (2FP) is generally better than a single shared expert (1FP).
• Mixed-camera geometry is complementary: in the KITTI360 example, two back-to-back fisheye cameras have little covisibility, but a front-facing perspective camera acts as an intermediate observation that links them into a coherent scene reconstruction.

Methodology in Plain English

The recipe is: freeze the big pretrained reconstruction network and bolt small learnable "patches" onto its linear layers. Each patch is a pair of low-rank matrices (a standard LoRA adapter), and there is a separate patch set for fisheye images and for panoramic images. Perspective frames activate no patch at all, so they run through the original network unchanged — that is what guarantees no regression on perspective data. A small CNN looks at each incoming image and picks which patch set to use, so camera metadata is not needed at test time.

Training is staged: the camera classifier is trained first with cross-entropy on ground-truth camera labels, then each camera's adapters are trained separately on fisheye and panoramic sequences, and finally the global-attention adapters are finetuned on mixed-camera sequences. Because perspective data can be synthesized by undistorting fisheye images and by rendering perspective views from panoramas (with depth and ray maps transformed consistently), no separate perspective dataset is required; fisheye images are also randomly re-distorted to diversify the distortions seen during training.

After training at rank 64 everywhere, each learned update matrix is decomposed with SVD and small singular values are dropped using one global threshold, keeping only the directions that matter most for each layer and camera. Because truncated SVD gives the optimal low-rank approximation, this reallocates capacity at essentially no cost in accuracy. Adding a future camera type would require only a new small adapter and one extra classifier class, avoiding catastrophic forgetting of existing cameras.

Why This Matters

Research impact: The paper reframes "generalize to new cameras" as an adaptation problem rather than a retraining problem, showing that a frozen perspective backbone already contains features that can be modulated for other projection geometries. It also provides a direct apples-to-apples comparison against calibration tokens, another adaptation approach, across three backbones.

Real-world applications:

• Extended reality headsets and spatial computing devices that fuse wide-FoV or fisheye tracking cameras with standard perspective cameras.
• Autonomous navigation and robotics, where a mixed rig (e.g., fisheye for coverage, perspective for fidelity) must be mapped in one pass on constrained compute.
• 360° and panoramic capture pipelines for indoor scene reconstruction and digital twins.
• Vehicle-mounted mapping, as in the KITTI360 example with back-to-back fisheye cameras plus a front-facing perspective camera.

Industry relevance: Backward compatibility means a deployed perspective-based 3D reconstruction system can add new sensors incrementally — a new camera needs only a lightweight adapter and one extra router class, not a retrained foundation model — which lowers compute, memory, and system complexity for spatial computing platforms.

Future Directions

• Extending beyond the three predefined camera categories: the current design assumes each frame belongs to perspective, fisheye, or panoramic, so a fourth camera type (the authors mention catadioptric or microscopic) requires incrementally learning a new class and expert first.
• Closing the synthetic-to-real gap: cross-camera synthesis from fisheye and panoramic data provides diverse projection geometries, but the authors note synthetic distortion may not fully capture real lens- and sensor-specific effects.
• Scaling the continual-learning story: the paper argues new experts can be added without retraining existing adapters, but a full study of how performance evolves as many camera types accumulate is not reported.
• Broadening backbone coverage and possibly relaxing the discrete router to handle continuous or unknown camera parameterizations, which the discrete one-hot routing currently does not address.

Target Audience

Researchers and engineers working on 3D reconstruction, SLAM, and spatial computing who already have a pretrained feed-forward reconstruction model and need it to handle fisheye, panoramic, or mixed camera rigs. It is most useful to readers comfortable with LoRA and transformer backbones; newcomers can still follow the high-level argument but the method section is equation-heavy.

Authors’ abstract

Feed-forward 3D reconstruction (3R) from unconstrained multi-view images has recently emerged as a prominent research direction in 3D computer vision. Various foundational reconstruction models have been proposed with different architectures and reconstruction paradigms. Despite being trained on large-scale perspective datasets with substantial computational resources, these models generally struggle to generalize to non-perspective images. This limitation hinders their…