E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training

AI summary

Overview

Research area: Computer vision — self-supervised representation learning for 3D perception, multi-view geometry, camera pose estimation, novel-view synthesis, and 3D Gaussian splatting.

Technical level: Advanced. The paper assumes familiarity with transformers, differentiable rendering, 3D Gaussian splatting, camera intrinsics/extrinsics, and standard 3D vision benchmarks.

Scope: The paper introduces E-RayZer, a fully self-supervised feedforward model that predicts camera parameters and explicit 3D Gaussians from unlabeled multi-view images, trained with a visual-overlap curriculum, and evaluates it as a spatial visual pre-training framework against both self-supervised and supervised baselines.

What This Paper Is About

Most 3D vision models today are trained with supervision derived from Structure-from-Motion systems such as COLMAP, which is described as inefficient, imperfect, and ultimately unscalable. Prior self-supervised work like RayZer instead learns 3D only indirectly, through latent-space view synthesis, which allows shortcut solutions such as frame interpolation rather than genuine 3D understanding. E-RayZer's goal is to learn geometrically grounded, 3D-aware representations directly from unlabeled images by performing self-supervised 3D reconstruction in explicit 3D space.

Key Contributions

1. First truly self-supervised feedforward 3D Gaussian splatting reconstruction model. E-RayZer is trained from scratch with zero 3D annotation, predicting camera parameters and explicit 3D Gaussians and rendering them for photometric self-supervision.
2. A visual-overlap-based learning curriculum. A fine-grained, sequence-level curriculum organizes training from easy to hard samples and harmonizes heterogeneous data sources without supervision. It has two instantiations: a semantic overlap metric (DINOv2 cosine similarity, self-supervised) and a geometric overlap metric (UFM covisibility, trained with 3D annotations).
3. A stronger paradigm for spatial visual pre-training. E-RayZer's learned representations outperform DINOv2, DINOv3, CroCo v2, VideoMAE V2, Perception Encoder, and RayZer on downstream 3D tasks.
4. Competitiveness with fully supervised reconstruction. E-RayZer matches or sometimes surpasses the supervised VGGT (reproduced with matched architecture and training setups as VGGT*), and its pre-trained weights further improve VGGT* when used as initialization.

Main Findings

• Pose estimation over prior self-supervised methods. In Table 1, when trained on DL3DV, E-RayZer reaches RPA@5°/15°/30° of 84.5 / 98.4 / 99.3 on WildRGB-D, 7.7 / 33.6 / 63.0 on ScanNet++, and 72.0 / 88.4 / 93.5 on DL3DV; RayZer trained on DL3DV scores 0.0 / 0.2 / 6.5, 0.0 / 0.7 / 6.2, and 0.0 / 0.6 / 6.2 respectively, despite RayZer achieving PSNR of 25.9, 20.5, and 21.4 on those datasets versus E-RayZer's 24.3, 20.1, and 20.3.
• Outperforms a partially supervised baseline. E-RayZer (trained only on RE10K) exceeds SPFSplat, which is initialized from MASt3R and is therefore not truly self-supervised, on most metrics in Table 1 — for example 40.3 / 89.4 / 96.5 versus 31.5 / 58.0 / 69.8 RPA on WildRGB-D.
• Comparable to supervised VGGT on pose.* In Table 2 (both trained on DL3DV), E-RayZer outperforms VGGT* on out-of-domain sets including WildRGB-D (51.1 / 82.3 versus 32.5 / 76.2), CamLand (18.1 / 62.9 versus 11.1 / 49.8), and BlendedMVS (22.9 / 46.8 versus 17.0 / 42.8), while VGGT* leads on others such as in-domain DL3DV (79.6 / 94.2 versus 72.0 / 88.4).
• Pre-training transfers to the supervised model. Initializing VGGT* with E-RayZer weights improves it substantially across all reported datasets, e.g. DL3DV 87.3 / 96.6, CO3Dv2 25.3 / 72.2, WildRGB-D 56.2 / 91.4, NAVI 26.9 / 64.3, and ScanNet++ 14.3 / 53.8 (RPA@5°/15°).
• Best learned representations on 3D downstream tasks. In Table 3, frozen-backbone E-RayZer on ScanNet++ achieves AbsRel 0.116 and δ<1.25 of 87.1, versus RayZer's 0.161 and 79.3 and DINOv2's 0.193 and 74.9; pose RPA@5°/15° is 13.8 / 49.5 for E-RayZer versus 4.7 / 27.4 for RayZer. Under full finetuning on BlendedMVS, E-RayZer reaches AbsRel 0.148 and δ<1.25 of 82.8 with pose RPA@5°/15° of 36.2 / 58.8, versus RayZer's 0.194, 77.7, and 26.1 / 50.2.
• Competitive but not leading on 2.5D flow. On StaticThings3D pairwise flow (Table 4), E-RayZer records EPE 1.254 with outlier ratios 16.9 / 7.8 / 3.1 at 1px/2px/5px, slightly behind RayZer (1.105; 13.4 / 6.6 / 2.8) but ahead of CroCo v2 (1.273; 17.7 / 8.7 / 3.8) and VideoMAE V2 (2.028; 42.7 / 22.1 / 6.9).
• Similar scaling behavior to supervision. Table 5 shows E-RayZer and VGGT* scale similarly: 7-dataset training improves generalization over DL3DV-only or RE10K-only training, but reducing sampling frequency of a domain slightly degrades performance on its corresponding test set (7-dataset models perform worse on DL3DV than DL3DV-only models). Training on DL3DV also yields better results than RE10K, indicating data quality matters.
• Visual-overlap curriculum beats frame-interval curriculum. In Table 6 on DL3DV, geometric overlap gives 20.3 PSNR and 72.0 / 88.4 / 93.5 RPA, semantic overlap gives 20.4 and 73.2 / 88.7 / 93.7, frame interval gives 19.8 and 56.1 / 79.3 / 86.0, and no curriculum gives 16.1 and 4.0 / 27.8 / 47.2. The same ordering holds on the seven-dataset mixture.
• Explicit 3D was previously reported to fail. The paper notes that training with explicit 3D leads to non-convergence as reported in RayZer (Table 7), which the proposed curriculum is designed to resolve.

Methodology in Plain English

E-RayZer takes a set of unlabeled multi-view images and splits them into a reference set and a hidden target set. A transformer first predicts camera intrinsics and extrinsics for every image using a VGGT-style multi-view transformer with alternating local and global attention, performing pairwise pose prediction between a canonical view and a target view. Image index embeddings are removed entirely, since the paper identifies them as a strong cue that lets RayZer learn frame interpolation shortcuts.

The posed reference views are then converted into pixel-aligned explicit 3D Gaussians — each Gaussian parameterized by distance along the ray, a quaternion orientation, spherical harmonic coefficients, scale, and opacity. Because 3D Gaussians support closed-form differentiable rendering, RayZer's learned transformer renderer is no longer needed; the modified gsplat renderer also back-propagates gradients to camera intrinsics. The Gaussians are rendered under the model's own predicted target-view cameras, and supervision comes only from photometric losses (MSE plus a perceptual loss) on the rendered target views — no 3D labels at all.

To make this stable, the authors introduce a curriculum based on visual overlap between frames. Overlap is labeled by uniformly sampling frame triplets at various spacings and averaging pairwise overlaps, producing a per-sequence spacing profile. During training, the overlap lower limit decays linearly (1.0 → 0.5 for geometric scheduling, 1.0 → 0.75 for semantic scheduling), and the model looks up the corresponding frame spacing per sequence, starting with easy, high-overlap samples and progressing to harder ones. This same metric aligns heterogeneous data sources without manual per-dataset tuning.

Training uses 10 input images (5 reference, 5 target). Single-dataset variants train on RealEstate10K or DL3DV; the multi-dataset variant mixes seven datasets: DL3DV, CO3Dv2, RealEstate10K, MVImgNet, ARKitScenes, WildRGB-D, and ACID. Evaluation covers WildRGB-D, DL3DV, ScanNet++, BlendedMVS, and StaticThings3D, plus zero-shot pose tests on RE10K, CO3Dv2, 7-Scenes, CamLand, and NAVI.

Why This Matters

Impact on research: The paper argues that data diversity and quality, rather than explicit 3D supervision, are the true drivers of scalability in large 3D vision models. It shows a self-supervised 3D model can match a supervised one and can also serve as a pre-training stage that improves that supervised model, suggesting a self-supervised-then-supervised pipeline for visual geometry.

Real-world applications:

• 3D scene reconstruction and novel-view synthesis from casually captured photos or video, without needing camera calibration or COLMAP preprocessing.
• Robotics and embodied agents that need spatial understanding of the physical world from raw visual observations.
• Augmented and virtual reality content creation, where camera poses and scene geometry must be inferred from consumer footage.
• Monocular and multi-view depth estimation for downstream 3D perception pipelines, given the depth-probing results on ScanNet++ and BlendedMVS.

Industry relevance: Eliminating dependence on SfM-derived pseudo-labels removes a slow and error-prone data pipeline step, which matters for companies training large 3D foundation models on internet-scale imagery. The demonstrated scaling behavior, the data-mixing guidance, and the finding that a self-supervised checkpoint boosts a supervised model all point toward practical recipes for 3D foundation model development.

Future Directions

• Extending to dynamic scenes. E-RayZer currently operates on static scenes, limiting training to the existing static datasets, which the authors note remain scarce; dynamic-scene support would allow learning from generic videos.
• Handling sparse or discontinuous inputs. The curriculum assumes continuous video frames with fairly uniform camera motion; sparse images or frames with drastic viewpoint changes may reduce its effectiveness.
• Closing the gap with supervision at large data scale. VGGT* still holds an advantage when trained on large data, so whether self-supervision fully matches supervision at internet scale remains open.
• Preserving 2.5D performance. E-RayZer trails RayZer on pairwise flow estimation, raising the question of how to retain correspondence ability while keeping explicit 3D grounding.

Target Audience

Researchers and engineers working on 3D foundation models, multi-view geometry, self-supervised representation learning, and neural rendering, particularly those interested in replacing SfM-derived supervision with scalable self-supervised objectives. It is most useful to readers already comfortable with transformer architectures, 3D Gaussian splatting, and camera pose evaluation metrics; the paper is not beginner-friendly.

Authors’ abstract

Self-supervised pre-training has driven rapid progress in foundation models for language, 2D images, and video, yet remains largely unexplored for learning 3D-aware representations from multi-view images. In this paper, we present E-RayZer, a self-supervised 3D vision model that learns geometrically grounded representations directly from unlabeled images. Unlike prior self-supervised methods such as RayZer, which infer 3D indirectly through latent-space view synthesis, E-RayZer operates directly in 3D space, performing self-supervised 3D reconstruction with Explicit geometry. This formulation eliminates shortcut solutions and yields representations that are 3D-aware. To ensure convergence and scalability, we introduce a fine-grained learning curriculum that organizes training from easy to hard samples and harmonizes heterogeneous data sources without any supervision. Experiments show that E-RayZer significantly outperforms RayZer on pose estimation and matches or sometimes surpasses fully supervised reconstruction models such as VGGT. Furthermore, its learned representations outperform leading visual pre-training models (e.g., DINOv3, CroCo v2, VideoMAE V2, and RayZer) on 3D downstream tasks, establishing E-RayZer as a promising paradigm for spatial visual pre-training.