3D sans 3D Scans: Scalable Pre-training from Video-Generated Point Clouds

AI summary

Overview

• Research area: 3D self-supervised learning (3D-SSL) for indoor scene understanding, specifically whether 3D representations can be pre-trained without any real 3D scans.
• Technical level: Advanced. The paper assumes familiarity with self-supervised distillation (teacher–student, EMA), Sinkhorn-Knopp clustering, point cloud backbones such as Point Transformer V3, and feed-forward multi-view reconstruction models.
• Scope: The paper introduces a video-generated point cloud dataset (RoomTours) and a noise-robust self-supervised framework (LAM3C), then evaluates both on indoor semantic and instance segmentation across four benchmark datasets.

What This Paper Is About

Large-scale 3D scene scans are expensive and slow to collect, which caps how far 3D self-supervised learning can scale. The authors ask whether 3D representations can instead be learned from ordinary unlabeled room-walkthrough videos, by reconstructing point clouds from those videos with an off-the-shelf feed-forward reconstruction model and then pre-training on them. The goal is a "scan-free" pre-training pipeline that is competitive with methods trained on real 3D scans.

Key Contributions

1. A pipeline for generating large video-generated point cloud (VGPC) datasets. The authors collect unlabeled indoor walkthrough videos from the web, filter and segment them into scene sequences, reconstruct them into point clouds, and align them geometrically. This produces RoomTours, a dataset of 49,219 VGPC scenes.
2. LAM3C, a self-supervised framework with two new noise-regularized losses: a Laplacian smoothing term that encourages spatially adjacent points to share similar embeddings, and a noise consistency term that enforces stable global representations across augmented views of the same noisy point cloud.
3. Empirical demonstration that VGPC alone are sufficient for 3D pre-training. Without any real 3D scans, LAM3C outperforms several prior self-supervised methods and matches or exceeds Sonata in some settings, including on instance segmentation on S3DIS Area 5.
4. A controlled ablation of the ingredients — alignment, regularization, data scale, and choice of reconstruction model — identifying large-capacity backbone, sufficient pre-training length, and dataset scale as necessary conditions.

Main Findings

• RoomTours scale and composition. The full dataset contains 49,219 VGPC scenes; the subset built solely from the authors' own collected videos is 15,921 VGPC (referred to as RoomTours-16k). The authors collected 3,462 videos from 19 countries, with a median duration of 3.63 minutes, yielding an average of 4.59 sequences per video.
• Semantic segmentation gains over training from scratch. On ScanNet, LAM3C with 49k scenes reaches 66.0 mIoU linear probing and 77.7 full fine-tuning, versus 16.1 / 74.7 for PTv3 from scratch. Similar gains hold on ScanNet200 (25.3 / 35.1 vs 2.2 / 32.0), ScanNet++ Val (34.2 / 43.1 vs 6.9 / 40.3), and S3DIS Area 5 (65.7 / 72.9 vs 29.6 / 67.8).
• Instance segmentation gains over training from scratch. LAM3C-49k reports AP of 25.1 / 39.7 on ScanNet, 8.3 / 19.6 on ScanNet200, 11.3 / 20.5 on ScanNet++ Val, and 21.6 / 45.7 on S3DIS Area 5, compared with PTv3 from scratch at 0.2 / 26.9, 0.02 / 17.2, 0.03 / 18.5, and 11.9 / 39.6.
• A larger variant narrows the gap further. LAM3C* (PTv3 Large, 437k pre-training steps) reaches 69.5 / 79.5 semantic mIoU on ScanNet and 47.2 instance full fine-tuning on S3DIS Area 5, the latter exceeding the full Sonata result of 45.5.
• Comparison with Sonata is mixed, not uniformly favorable. The full Sonata, which uses 18k real and 121k synthetic scenes, reports higher numbers on most semantic segmentation columns (for example 72.5 linear probing on ScanNet and 37.3 on ScanNet++ Val) than LAM3C*. The paper frames its claim as outperforming Sonata trained only on real scans on several benchmarks and matching Sonata when Sonata uses both real and synthetic data; Sonata variants restricted to real data score lower, such as Sonata (all real) at 28.0 / 40.3 instance segmentation on ScanNet and 22.9 linear probing on S3DIS Area 5.
• Regularization is the key to learning from noisy reconstructions. Removing the noise-regularized loss drops ScanNet linear probing from 57.1 to 51.1; using the Laplacian smoothing loss alone gives 55.0 and the noise consistency loss alone gives 55.3, while combining both gives 57.1.
• Alignment matters most for frozen-backbone evaluation. Adding z-axis alignment improves linear probing by +5.3 points (51.8 → 57.1) and scale alignment improves it by +1.6 points (55.5 → 57.1).
• Performance scales with the number of generated scenes. Linear probing rises from 57.1 at 1k scenes to 58.9 at 16k and 65.9 at 49k; full fine-tuning rises from 75.1 to 75.6 to 77.7.
• The reconstruction model matters. With the same pre-training recipe, VGGT gives 49.3 / 75.1 and MapAnything gives 50.1 / 75.5, versus 57.1 / 75.1 for π³, which the authors attribute to π³ handling longer sequences.
• LAM3C and RoomTours are complementary. On clean ScanNet pre-training data, Sonata and LAM3C score 67.1 and 66.8; on noisy RoomTours-1k, Sonata scores 51.1 and LAM3C scores 57.1. The authors read this as LAM3C providing no inherent advantage on curated scans but a clear advantage on noisy reconstructions.
• Advantages persist under data-efficient fine-tuning. With only 1% of ScanNet training scenes, LAM3C-49k* reaches 40.5 mIoU, versus 23.6 for PTv3 from scratch and 36.0 for Sonata (ScanNet); with 20 annotated points it reaches 70.4 versus 62.2 and 66.3.
• Zero-shot behavior is partial. PCA visualizations show LAM3C clearly separating local structures such as desks without any fine-tuning, but global structure is less coherent than for a model pre-trained on real scans, with blurred floor boundaries.

Methodology in Plain English

The authors avoid 3D scanners entirely. Instead, they scrape room-walkthrough videos from YouTube (real-estate tours and apartment viewings) from multiple cities and countries. They filter out videos with heavy editing, aggressive camera motion, or dynamic subjects, and they use CLIP to classify each frame as indoor or outdoor and then as one of three coarse room types (living room, bedroom, bathroom). Scene boundaries are found where the predicted room type changes, with a 0.5-second temporal consistency constraint. Each resulting scene sequence is fed to π³, a feed-forward reconstruction model, which turns it into a colored point cloud in one pass rather than through an optimization loop.

Because raw reconstructions have inconsistent coordinate systems and scales, the authors post-process them: they subsample points, remove statistical outliers, detect a dominant plane and rotate the scene to a Z-up orientation, and rescale the scene so its bounding-box diagonal matches a scale distribution estimated from ScanNet. They then compute per-point normals with local PCA.

For pre-training, they use a teacher–student setup: two augmented views of the same point cloud go to a student network and an exponential-moving-average teacher, and the student is trained to match the teacher's prototype assignments computed via Sinkhorn-Knopp. This base clustering loss follows Sonata. On top of it, they add a Laplacian smoothing loss that builds a k-nearest-neighbor graph over each point cloud and penalizes differences in embedding between nearby points, weighted by distance, with a Huber penalty instead of squared L2 for robustness, and a noise consistency loss that pulls the student's embeddings of an augmented view toward the teacher's embeddings of a corresponding view. The smoothing weight ramps from 2e-4 to 3e-3 during training while the consistency weight is held at 0.05, so the model gradually emphasizes geometric regularization.

Why This Matters

• Impact on research. The paper reframes the bottleneck in 3D self-supervised learning as a data-acquisition problem rather than purely a modeling problem. It shows that a corpus roughly two orders of magnitude larger than typical scanned indoor datasets can be synthesized from video, which suggests a route to the kind of scaling that has driven progress in 2D vision. It also documents that prior comparisons can be confounded when pre-training data includes validation or test splits, and corrects for this in its baselines.
• Real-world applications (implied by the setting, not enumerated in the paper):
  • Indoor semantic and instance segmentation for scanned or reconstructed building interiors.
  • Robot and embodied-agent perception in indoor environments, which requires spatial understanding without pre-collected scans of each new building.
  • Real-estate and virtual-tour products that already generate 3D walkthroughs from video and could reuse such representations.
  • Augmented and virtual reality content pipelines that need labeled structure from casually captured video.
• Industry relevance. The pipeline's inputs are commodity video and its reconstruction step runs at about 5 minutes per scene on eight NVIDIA H200 GPUs, which makes data collection a matter of video sourcing and compute rather than scanner deployment and manual annotation. That shifts the cost structure for any organization that needs 3D scene understanding at scale.

Future Directions

• Closing the global-coherence gap. The authors observe that LAM3C learns less globally consistent geometry than models trained on real scans, attributing this to coordinate frame and scale variation in π³ reconstructions. Reducing that variability, or making the objective explicitly invariant to it, is a natural next step.
• Scaling RoomTours further. Since the data-scaling ablation shows performance still rising from 1k to 16k to 49k scenes, it is open whether continued growth in video-sourced scenes yields continued gains or saturates.
• Better reconstruction backbones. π³ outperformed VGGT and MapAnything in the ablation, largely for its longer-sequence handling. Improved feed-forward reconstruction is a direct lever on the quality of the pre-training data.
• Extending beyond indoor scenes and beyond these benchmarks. The paper evaluates on ScanNet, ScanNet200, ScanNet++ Val, and S3DIS Area 5 for semantic and instance segmentation. Whether video-generated point clouds transfer to outdoor scenes, detection, or navigation tasks is not reported.

Target Audience

Researchers and engineers working on 3D representation learning, point cloud perception, and dataset construction for computer vision, particularly those interested in self-supervised pre-training, scalable data pipelines, and reducing dependence on scanned or manually annotated 3D data. Readers who need only the high-level message can take the dataset construction and pre-training recipe from the main text; readers who want to reproduce the numbers should expect to consult the supplementary material for the alignment, augmentation, and fine-tuning hyperparameters, which the main text references but does not fully specify.

Authors’ abstract

Despite recent progress in 3D self-supervised learning, collecting large-scale 3D scene scans remains expensive and labor-intensive. In this work, we investigate whether 3D representations can be learned from unlabeled videos recorded without any real 3D sensors. We present Laplacian-Aware Multi-level 3D Clustering with Sinkhorn-Knopp (LAM3C), a self-supervised framework that learns from video-generated point clouds reconstructed from unlabeled videos. We first introduce RoomTours, a video-generated point cloud dataset constructed by collecting room-walkthrough videos from the web (e.g., real-estate tours) and generating 49,219 scenes using an off-the-shelf feed-forward reconstruction model. We also propose a noise-regularized loss that stabilizes representation learning by enforcing local geometric smoothness and ensuring feature stability under noisy point clouds. Remarkably, without using any real 3D scans, LAM3C achieves better performance than previous self-supervised methods on indoor semantic and instance segmentation. These results suggest that unlabeled videos represent an abundant source of data for 3D self-supervised learning. Our source code is available at https://ryosuke-yamada.github.io/lam3c/.