Splat and Distill: Augmenting Teachers with Feed-Forward 3D Reconstruction For 3D-Aware Distillation

AI summary

Overview

Research area: Computer Vision — 3D-aware representation learning, knowledge distillation, and Vision Foundation Models.

Technical level: Intermediate. Readers will benefit from familiarity with vision transformers, student-teacher distillation, and 3D Gaussian Splatting, though the core idea is described conceptually in the paper.

Scope: The paper introduces a training framework that injects 3D geometric awareness into 2D vision foundation models by augmenting the teacher network with a feed-forward 3D reconstruction pipeline.

What This Paper Is About

Vision Foundation Models such as DINOv2 produce excellent features for 2D tasks, but perform poorly on tasks that require understanding 3D geometry, such as depth estimation and surface normals. Prior work (FiT3D) tries to fix this by optimizing a separate 3D representation per scene and then fine-tuning the model on rendered features, but this requires slow per-scene optimization and, because the input 2D features are inconsistent across views, produces a least-squares averaged, blurry "compromise." This paper instead augments the teacher network with a fast, feed-forward 3D reconstruction model during training, lifting 2D features into explicit 3D Gaussians, "splatting" them onto novel viewpoints, and distilling the resulting geometrically grounded features into a student model.

Key Contributions

1. A 3D-aware teacher augmentation scheme. The teacher network is paired with a pre-trained, feed-forward 3D reconstruction model (MVSplat) that predicts 3D Gaussians from context views, replacing the slow per-scene optimization used in prior work.

2. Mask-aware feature lifting and semantic blending. 2D teacher features are attached to 3D Gaussians via pixel-to-Gaussian correspondences, using instance segmentation masks to guide upscaling across the ×14 resolution gap between patch features and full-resolution images, and a blending step that averages rendered features within semantic regions to regularize reconstruction artifacts.

3. A dynamic distillation process. Because the teacher's weights are updated as an exponential moving average (EMA) of the student's weights, the teacher's consistency improves alongside the student's, avoiding the static averaging of inconsistent features that occurs in optimization-based pipelines.

4. Comprehensive evaluation across 3D-aware and semantic tasks. The method is evaluated on monocular depth estimation, surface normal estimation, multi-view correspondence, and semantic segmentation, including out-of-domain generalization.

Main Findings

• Monocular depth estimation improves consistently. On ViT-Small/Base backbones, the method outperforms baselines on ScanNet++, ScanNet, and NYUv2, with average relative RMSE gains of 5.90%, 5.82%, and 3.21% over the closest baseline respectively. For example, on ViT-Small, ScanNet++ RMSE improves from 0.2811 (DINOv2) and 0.2500 (Fit3D) to 0.2421 (Ours).

• Surface normal estimation improves. On NYUv2, the method achieves a 5.37% improvement over the closest baseline with DINOv2-Small (28.93 vs. 30.57 for Fit3D) and 3.93% with DINOv2-Base (29.37 vs. 30.57 for Fit3D).

• Semantic segmentation is not sacrificed — it improves. With DINOv2-Small, the method gains 0.03%, 2.77%, and 1.76% relative mIoU over Fit3D on ScanNet++, ScanNet, and NYUv2. With DINOv2-Base it improves notably on ScanNet and NYUv2 but shows a slight decrease on ScanNet++ (mIoU 34.07 vs. 34.85 for Fit3D).

• Multi-view correspondence improves across viewpoint changes. The method consistently improves recall over baselines for varying viewpoint changes on ScanNet, as shown in the paper's Fig. 7.

• Out-of-domain transfer holds. With a ViT-Base backbone, relative mIoU gains of 3.56% on ADE20K and 0.79% on Pascal VOC are reported. On KITTI depth estimation, the method improves 3.31% over Fit3D, transferring indoor 3D awareness to outdoor scenes.

• Ablations validate the design choices. On ScanNet++ with ViT-Small, removing blending (A) gives depth RMSE 0.3435 vs. 0.3299 for the full model; bilinear instead of mask-aware upscaling (B) gives 0.3309; a cosine loss instead of the distillation loss (C) gives 0.3310; a frozen teacher (D) gives 0.3444; rendering to context views instead of novel views (E) gives 0.3332 (though segmentation improves to 84.02 aAcc); SAM-extracted masks instead of manual masks (F) gives 0.3328; a direct feature rendering loss (G) gives 0.3430; and the most basic variant (H) gives 0.3520.

• Manual masks are only marginally better than SAM masks. The paper reports that manual annotation has only a minor advantage over SAM-extracted masks, and that consistent class labels across frames are not required.

• Qualitative improvements. The method produces more refined structural detail in depth maps, smoother and more geometrically correct normal maps (e.g., correctly identifying the back of a couch), cleaner object boundaries in segmentation, and less noisy feature spaces under PCA.

Methodology in Plain English

The framework, called Splat and Distill (SnD), builds on the DINOv2 student-teacher self-distillation setup. Each training iteration samples a 3D scene, drawing two "context" views and a separate "target" view.

The teacher's job is no longer to simply process a 2D image. Instead:

1. Reconstruct geometry. A frozen, pre-trained feed-forward model (MVSplat) takes the two context views and predicts a set of 3D Gaussians, each with a mean position, covariance, and opacity. The appearance parameters (spherical harmonic coefficients) are discarded, since the goal is geometry, not image rendering.

2. Extract teacher features. The teacher processes the same context views to produce low-resolution 2D feature maps.

3. Upscale with masks. Because the feature maps are 14× smaller than the images, naively upscaling blurs features across object boundaries. The method interpolates only from neighboring low-resolution points that share the same semantic label, producing sharp high-resolution features.

4. Lift into 3D. Each upscaled feature vector is attached to its corresponding 3D Gaussian using the one-to-one pixel-to-Gaussian correspondence that MVSplat provides, creating a 3D feature scene.

5. Splat to a novel view. This feature scene is rendered from the target camera's viewpoint. A blending step then averages the rendered features within each semantic mask region (using α = 0.5), smoothing out noise from imperfect reconstruction while preserving object edges.

6. Distill. The student sees only the 2D target image and produces its own features. Both feature maps pass through a shared DINO head, and the student is trained with a cross-entropy distillation loss against the teacher's blended features (downscaled to match resolution), with a stop-gradient on the teacher. The teacher's weights are updated as an EMA of the student's.

The training data is a subset of ScanNet++, deliberately matching the subset used by the closest baseline (Fit3D) for a fair comparison.

Why This Matters

Impact on research: The paper offers a scalable alternative to optimization-based 3D feature distillation. By replacing per-scene optimization with a feed-forward pipeline, it avoids the feature-averaging artifacts that limit prior work and uses far fewer Gaussians. It also argues against the direction taken by DUNE, which distills directly from teachers and thereby inherits their 3D inconsistencies — this work instead corrects inconsistencies before distilling. Notably, it achieves stronger 3D awareness while also improving semantic richness, suggesting the two objectives need not trade off.

Real-world applications:

• Robot navigation and manipulation in indoor environments, where reliable monocular depth and surface normals from a single camera matter.
• Augmented reality and 3D scene understanding, where consistent features across viewpoints support stable placement and tracking.
• Autonomous driving, since the KITTI results show indoor-learned 3D awareness transferring to outdoor scenes.
• 3D content creation and editing pipelines, where semantically meaningful, geometrically grounded features enable open-vocabulary segmentation and manipulation.

Industry relevance: The method improves off-the-shelf foundation models that many production systems already use, rather than requiring a new architecture. Because MVSplat is used frozen and off-the-shelf, and manual masks can be swapped for SAM masks with only a minor drop in performance, the approach is practical to adopt.

Future Directions

• Reducing reliance on segmentation masks. Ablation F shows SAM masks come close to manual ones, but the framework still depends on instance masks during training. Removing this dependency entirely is an open step.

• Extending beyond indoor scenes and two context views. Training uses only ScanNet++ indoor data and K=2 context views. Whether more context views, outdoor training data, or larger scene diversity would further improve results is not established here.

• Resolving the segmentation-versus-geometry tension seen in Ablation E. Rendering to context views improves segmentation while rendering to novel views improves depth, suggesting a possible scheme that benefits both.

• Scaling to larger backbones and broader task suites. Experiments cover ViT-Small and ViT-Base; scaling behavior beyond these variants, and on tasks not tested here, is not reported.

Target Audience

Researchers and practitioners working on vision foundation models, 3D-aware representation learning, and knowledge distillation. It is especially relevant to those building systems that need geometric understanding from single images or across multiple views, and to anyone who has found DINOv2-family features strong on 2D semantics but weak on depth, normals, or correspondence. Readers with background in 3D Gaussian Splatting and student-teacher distillation will get the most from the technical sections.

Authors’ abstract

Vision Foundation Models (VFMs) have achieved remarkable success when applied to various downstream 2D tasks. Despite their effectiveness, they often exhibit a critical lack of 3D awareness. To this end, we introduce Splat and Distill, a framework that instills robust 3D awareness into 2D VFMs by augmenting the teacher model with a fast, feed-forward 3D reconstruction pipeline. Given 2D features produced by a teacher model, our method first lifts these features into an explicit 3D Gaussian representation, in a feedforward manner. These 3D features are then ``splatted" onto novel viewpoints, producing a set of novel 2D feature maps used to supervise the student model, ``distilling" geometrically grounded knowledge. By replacing slow per-scene optimization of prior work with our feed-forward lifting approach, our framework avoids feature-averaging artifacts, creating a dynamic learning process where the teacher's consistency improves alongside that of the student. We conduct a comprehensive evaluation on a suite of downstream tasks, including monocular depth estimation, surface normal estimation, multi-view correspondence, and semantic segmentation. Our method significantly outperforms prior works, not only achieving substantial gains in 3D awareness but also enhancing the underlying semantic richness of 2D features. Project page is available at https://davidshavin4.github.io/Splat-and-Distill/