Vernata: Self-Supervised Learning of LiDAR Point Representations

AI summary

Overview

Research area: Self-supervised representation learning for 3D point clouds, specifically outdoor LiDAR perception for robotics and autonomous driving.

Technical level: Intermediate. The paper assumes familiarity with self-supervised learning (contrastive methods, self-distillation, student-teacher frameworks), semantic segmentation, and 3D backbone architectures. The core ideas are explainable, but the method builds directly on the Sonata framework and references architectures such as PTv3, DINOv2, and LoftUp.

Scope in one sentence: The paper presents Vernata, a self-supervised pretraining framework for outdoor LiDAR point clouds that extends Sonata with sparse view augmentation, a Sinkhorn-Knopp memory bank, and cross-modal distillation from 2D image features, and evaluates it on GrandTour, TartanGround, Waymo, and a custom in-house dataset.

What This Paper Is About

Deep learning on 3D LiDAR point clouds is held back by the cost of manually annotating 3D data, so self-supervised learning (SSL) offers a way to learn general-purpose features from unlabeled scans. Prior point-cloud SSL methods suffer from representation collapse, and the Sonata framework addressed this by preventing models from latching onto trivial geometry, but Sonata was designed for indoor scans with uniform density and auxiliary modalities. The goal of this work is to adapt that paradigm to outdoor LiDAR, where point density drops sharply with range, resolution is lower, and color or reliable surface normals are often missing.

Key Contributions

1. Sparse view augmentation: A subsampling strategy that presents the student with sparsely subsampled views of the scene while the teacher sees dense views, creating a sparse-to-dense matching objective to make representations invariant to the range-dependent density variations of LiDAR.

2. Sinkhorn-Knopp memory bank for 3D: An adaptation of the SwAV-style memory bank to point-cloud representation learning, which concatenates a FIFO queue of prototype scores with the current batch before Sinkhorn-Knopp normalization, decoupling normalization statistics from batch size so training stabilizes on 4 GPUs instead of the 32 GPUs used by Sonata.

3. Cross-modal distillation from 2D vision foundation models: A point-wise distillation objective that uses frozen DINOv2-S features, upsampled with LoftUp (224 px short side) and backprojected into the point cloud, aligned to student features via an L2-normalized cosine-style loss, providing fine-grained semantic guidance especially at long range.

4. A public TartanGround benchmark protocol: Since TartanGround lacked a canonical evaluation protocol, the authors define one by accumulating point clouds over 3 frames, mapping 1496 scene-specific raw labels to 7 canonical semantic classes, and defining a stratified train/validation split, which they release publicly along with the code.

Main Findings

• Large gains over Sonata baselines on both LiDAR datasets: Linear probing with Vernata reaches 54.7 mIoU on TartanGround (+5.9 points, +12.1% over the finetuned Sonata baseline) and 57.1 mIoU on Waymo (+7.3 points, +14.7%). Full metrics: TartanGround mAcc 69.0 and Acc 87.0; Waymo mAcc 69.0 and Acc 91.5.

• Cross-modal distillation drives most of the improvement: In the ablation, adding CMD alone yielded 53.4 mIoU on TartanGround and 56.5 mIoU on Waymo, the single largest gain of any component. Sparse view augmentation alone gave 49.6 mIoU on TartanGround and 50.9 on Waymo; the memory bank alone gave 50.8 and 51.0; combining both gave 51.6 and 51.8. CMD was slightly less effective on TartanGround (+4.6 mIoU) than Waymo (+6.7 mIoU), attributed to the domain gap between synthetic imagery and DINOv2's real-world pretraining data.

• Robustness when modalities are missing: A CMD-less Vernata variant trained and evaluated with coordinates only reached 49.4 mIoU on TartanGround and 50.2 mIoU on Waymo. Removing only normals gave 50.2 and 51.2; the full coordinate/color/normal configuration gave 51.6 and 51.8.

• SSL closes the gap to supervised learning as data shrinks: Against a fully supervised PTv3 baseline, PTv3 led by 11.2 mIoU on Waymo (68.3 vs 57.1, n=23,691). On TartanGround (n=6,501) the gap narrowed to 1.5 mIoU (56.2 vs 54.7), with Vernata higher on mAcc by 3.6 (69.0 vs 65.4). On the custom dataset (n=220), Vernata outperformed PTv3 by 2.5 mIoU and 4.7 mAcc (22.1/29.3 vs 19.6/24.6), despite PTv3 using a 7.5M-parameter decoder versus Vernata's 9–37k-parameter linear head.

• High-resolution distillation helps at long range: Against Concerto's patch-based distillation, results were on par overall (TartanGround 53.4 mIoU for both; Waymo 56.9 patch vs 56.5 Ours), but beyond a 20-meter radius Vernata was better: Waymo 20 m+ mIoU 50.7 vs 50.1, and TartanGround 20 m+ mIoU 48.2 vs 48.0.

• Real-world generalization with very little labeled data: The CMD-less model, finetuned on an in-house dataset of just 311 frames, performed robust path detection and captured dynamic obstacles such as humans on a GrandTour trail and on a trajectory from RAI's Ultra Mobility Vehicle (UMV), the latter using only point coordinates.

• Inference throughput reported: On an NVIDIA A5000, FP32 / FP16 rates were 8.9 / 16.1 Hz on Waymo, 7.2 / 13.3 Hz on TartanGround, and 8.8 / 16.8 Hz on the custom dataset.

Methodology in Plain English

Vernata keeps the Sonata student-teacher setup: a teacher network (a PTv3 encoder updated by exponential moving average of the student) processes large "global" views covering 40%–100% of a scene, while the student processes partial views—masked global views and small local crops covering 5%–40%. The student must predict the teacher's prototype assignments, computed by Sinkhorn-Knopp normalization over learnable prototypes, using a cross-entropy loss. Sonata's key trick, which Vernata inherits, is deliberately hiding fine-grained geometry—dropping the decoder and using parameter-free up-casting—so the model cannot take the "geometric shortcut" of regressing trivial cues like point height.

Three changes adapt this to outdoor LiDAR. First, the student also receives a uniformly subsampled version of a global view, with the subsampling ratio scheduled to get progressively harsher from (0.9, 1.0) down to (0.5, 0.7) over training; matching this sparse view to the teacher's dense view teaches density invariance. Second, because Sinkhorn-Knopp needs large batches to estimate the global prototype distribution, a FIFO memory bank of 50,000 prototype scores (GrandTour and TartanGround) or 100,000 (Waymo) is concatenated with the current batch before normalization, with only the batch-sized slice kept as targets. Third, a frozen DINOv2-S image teacher provides features that are upsampled with LoftUp and backprojected onto points using known camera intrinsics and extrinsics; the student minimizes L2 distance between its normalized projected features and these normalized image features. The final loss is simply the sum of the uni-modal and cross-modal losses, with no extra hyperparameter tuning.

Evaluation uses linear probing for semantic segmentation (mIoU, mAcc, Acc), initializing from a ScanNet-pretrained Sonata checkpoint, pretraining self-supervised on the target datasets for 20,000 steps, then training a linear head for 10,000 steps (TartanGround, custom) or 20,000 steps (Waymo). Point clouds are cropped to a 50 m × 50 m region around the sensor and voxelized at 0.1 m during training. TartanGround pretraining used 24,722 samples, with linear probing restricted to 6,501 curated samples; the custom dataset contains 311 labeled frames over four trail trajectories (two from GrandTour, two self-collected) with 30 classes and one self-collected trajectory held out.

Why This Matters

Impact on research: This work shows that SSL recipes designed for indoor point clouds do not transfer for free to outdoor LiDAR, and identifies specific reasons—density decay with range, lower resolution, missing modalities—along with concrete fixes. It also releases a standardized TartanGround segmentation protocol and a large-scale pretraining recipe that runs on 4 GPUs rather than 32, lowering the compute barrier for 3D SSL research. The finding that SSL-pretrained models surpass fully supervised ones as labeled data shrinks is a strong argument for pretraining-first pipelines in 3D.

Real-world applications:

• Off-road and trail autonomy, where robots must segment paths, vegetation, people, and terrain features without dense annotation (evaluated on GrandTour trails and a Ultra Mobility Vehicle trajectory).
• Urban autonomous driving perception, where better road and lane-marker segmentation in both dense and sparse regions matters (evaluated on Waymo).
• Robots with degraded sensor suites, such as platforms without calibrated omnidirectional cameras or without reliable surface normals, which can still run the coordinate-only model.
• Deploying as a pretrained backbone for downstream 3D tasks where labeled data is scarce, such as rare-class detection in unstructured environments.

Industry relevance: On-device inference rates reported on an NVIDIA A5000 (7.2–16.8 Hz depending on dataset and precision) and training on 4 H100-class GPUs make the approach plausible for practical robotics pipelines. The public code release at https://github.com/rai-opensource/vernata supports adoption, and the low-data advantage speaks directly to industrial settings where annotation budgets are the limiting factor.

Future Directions

• Temporal context: The authors state the method currently operates strictly on a per-frame basis, and future work will integrate temporal context to ensure inference consistency. This is also relevant to the paper's own observation that walkways and structural elements are occasionally misclassified.
• Deployment on mobile robots: The authors plan to deploy these representations for execution on mobile robots, moving from segmentation to acting on the representations.
• Architectural trade-offs in distillation: The comparison with Concerto's patch-based distillation showed near-parity overall but an advantage for high-resolution matching beyond 20 meters; the authors explicitly leave a comprehensive investigation of these trade-offs to future work.
• Domain gap in cross-modal distillation: CMD was less effective on the synthetic TartanGround imagery than on real Waymo data, suggesting the interaction between synthetic data and real-world vision foundation model features needs further study.

Target Audience

Robotics and autonomous driving perception researchers working on 3D representation learning; practitioners who need LiDAR segmentation models but lack large annotated datasets; engineers building perception stacks for off-road or unstructured environments where color and normals may be unavailable; and researchers interested in cross-modal distillation between 2D vision foundation models and 3D backbones. Readers should have some background in self-supervised learning and point-cloud architectures to follow the method section, though the problem framing and results are accessible to a broader computer vision audience.

Authors’ abstract

LiDAR serves as a primary sensing modality for robots operating in outdoor environments. However, the performance of deep learning models in this domain is severely limited by the scarcity of labeled data, a direct result of the high cost of 3D annotation. Self-supervised learning addresses this scarcity by learning general-purpose features from unlabeled data. In this work, we present a multi-modal, multi-teacher distillation framework for self-supervised learning on outdoor LiDAR point clouds. Building upon the Sonata architecture, we introduce Vernata, consisting of three extensions: sparse view augmentation to improve robustness against varying point densities, a memory bank mechanism to stabilize resource-constrained training, and cross-modal distillation utilizing dense, high-resolution 2D image features to enable fine-grained semantic guidance. We evaluate our method on the GrandTour, TartanGround, and Waymo datasets, as well as data collected from our own robotic platforms. Our experiments demonstrate a significant performance improvement over Sonata baselines, yielding mIoU scores of 54.7 on TartanGround (+5.9 points, +12.1%) and 57.1 on Waymo (+7.3 points, +14.7%). Finally, we show that the self-supervised approach maintains strong performance even in reduced-modality settings (lacking color or normals), achieving competitive mIoU scores of 49.4 and 50.2 on the respective datasets.