Multi-Modal Assistance for Unsupervised Domain Adaptation on Point Cloud 3D Object Detection

AI summary

Overview

Research area: Unsupervised domain adaptation (UDA) for LiDAR-based 3D object detection in autonomous driving, combining point clouds with camera images and vision-language models.

Technical level: Advanced. Understanding the paper requires familiarity with teacher-student self-training, 3D detection backbones (PV-RCNN, SECOND-IoU, PointPillars), pseudo-labeling, contrastive/embedding alignment losses, and vision-language models.

Scope: The paper proposes MMAssist, a multi-modal training-time framework that uses image and text features as bridges to align 3D features across the source and target domains, and that also injects image-derived 3D boxes into the target-domain pseudo labels.

What This Paper Is About

LiDAR-based 3D detectors trained on a labeled source dataset usually degrade when deployed on a different dataset, because of differences in LiDAR beam counts, environments, and sensor setups. Existing state-of-the-art UDA methods for 3D detection rely almost entirely on point clouds, even though autonomous driving datasets normally capture images and point clouds at the same time. MMAssist aims to exploit those images, and text descriptions of objects generated from them by a large vision-language model, to make the 3D features of the source and target domains agree and to improve the quality of target-domain pseudo labels — while still using point clouds only at test time.

Key Contributions

1. A multi-modal assistance framework for LiDAR-based 3D UDA that targets both cross-domain feature alignment and pseudo-label quality, rather than only one of the two.
2. A feature-alignment method that uses image features (extracted with RoIAlign from a pre-trained vision backbone) and text features (extracted from LVLM-generated object descriptions with a pre-trained text encoder) as bridges between source-domain and target-domain 3D features, since similar objects in different domains are expected to have a smaller gap in image and text space than in point cloud space.
3. A weighted feature-fusion scheme that combines the 3D feature with its image-aligned and text-aligned features using learned weights, plus alignment between the student and teacher branch features in the target domain.
4. A pseudo-label enhancement procedure that generates 2D boxes with the off-the-shelf detector GroundingDINO, lifts them into 3D with geometric reasoning, and adds them to teacher-generated pseudo labels in long-range areas, discarding image-derived boxes that substantially overlap teacher boxes.
5. An implementation integrated into three detectors (PV-RCNN, SECOND-IoU, PointPillars) and evaluated on three adaptation tasks (Waymo→nuScenes, Waymo→KITTI, nuScenes→KITTI), reported to perform best in seven of the nine resulting subtasks "in general."

Main Findings

• Best-in-most-subtask results: Across the combination of three detectors and three adaptation tasks (nine subtasks), MMAssist is reported to perform best in seven of them in general.
• PV-RCNN, nuScenes→KITTI: MMAssist reaches 86.8/78.1 (AP_BEV/AP_3D), surpassing the second-best method CMT (85.8/75.5) by 1.0% AP_BEV and 2.6% AP_3D.
• PV-RCNN, Waymo→KITTI: MMAssist reaches 87.6/72.7, which is 1.7% AP_BEV better than CMT (85.9/74.5), while CMT is 1.8% AP_3D better than MMAssist.
• PV-RCNN, Waymo→nuScenes: MMAssist reaches 45.5/27.0, outperforming the second-best CMDA (44.4/26.4) by 1.1% AP_BEV and 0.6% AP_3D.
• PointPillars, nuScenes→KITTI: MMAssist and GroupEXP-DA both achieve the best 81.9% AP_BEV, but MMAssist leads by 7.6% AP_3D (60.4 vs. 52.8 for GroupEXP-DA).
• PointPillars, Waymo→KITTI: MMAssist (81.4/56.8) beats the second-best GroupEXP-DA (78.4/54.1) by 3.0% AP_BEV and 2.7% AP_3D.
• PointPillars, Waymo→nuScenes (a loss for MMAssist): GroupEXP-DA performs best (44.3/22.2) and exceeds MMAssist (42.5/21.5) by 1.8% AP_BEV and 0.7% AP_3D.
• SECOND-IoU, nuScenes→KITTI: MMAssist (84.8/69.8) beats the overall second-best CMT (83.0/68.1) by 1.8% AP_BEV and 1.7% AP_3D.
• SECOND-IoU, Waymo→KITTI: MMAssist and GroupEXP-DA share the best AP_BEV (86.9), but GroupEXP-DA leads by 3.3% AP_3D (73.7 vs. 70.4).
• SECOND-IoU, Waymo→nuScenes: MMAssist (45.1/24.6) surpasses the second-best AP_BEV of GroupEXP-DA (43.8) by 1.3% and shares the best AP_3D with CMDA.
• Ablation on Waymo→KITTI with PointPillars: baseline (a) 76.7/52.7; adding image-derived pseudo labels (b) 79.1/53.2; adding image-feature alignment (c) 80.5/54.4; adding text-feature alignment (d) 80.0/54.5; both (e) 80.9/55.7; full model with student-teacher alignment (f) 81.4/56.8.
• Second stage alone is not the source of the gain: compared with vanilla PointPillars (the baseline, 76.7/52.7), simply adding a second stage raises AP_BEV to 78.3% but lowers AP_3D to 51.9%.
• Bridge effect confirmed: applying image and text alignment in both the pre-training and self-training stages (81.4/56.8) gives a notably larger gain than applying them in only one stage (79.7/55.3 for pre-training only; 79.8/54.6 for self-training only), relative to the 79.1/53.2 result without them.
• Weighted fusion is the best fusion: learned weighted element-wise sum (WSum) gives 81.4/56.8, versus concatenation (Concat) 80.5/55.4 and plain element-wise sum (Sum) 78.8/50.8.
• Model choices: LLaVA + SLIP (80.2/55.7 for Qwen2-VL + SLIP; 80.6/52.7 for LLaVA + LLaMA; 81.4/56.8 for LLaVA + SLIP) shows LLaVA is the better LVLM choice and SLIP the better text encoder choice among those tested.
• Long-range pseudo labels: adding image-derived pseudo labels changes distances 0–30 m from 81.0/58.5 to 81.4/58.4, distances 30–60 m from 50.7/24.6 to 51.9/27.9, and distances 60–150 m from 3.2/0.2 to 5.0/0.7.
• Speed cost is small: on a single RTX 4090, SECOND-IoU runs at 52.04 FPS (51.40 with MMAssist), PV-RCNN at 6.67 FPS (6.65), and PointPillars at 82.55 FPS (79.43). Image and text information is used only during training; testing uses point clouds only.

Methodology in Plain English

MMAssist builds on the teacher-student self-training recipe of DTS. In a pre-training stage, a detector is trained on the labeled source domain. In a self-training stage, that model initializes both a teacher and a student; the student learns from pseudo labels on the unlabeled target domain, and the teacher is updated as an exponential moving average of the student. Point clouds from both domains are re-sampled so they have similar beam densities.

The novel part is how the two domains are brought together. Instead of trying to directly align source and target 3D features — hard, because the two training processes are separate — the method uses image and text features as a common intermediate. For every ground-truth or pseudo-labeled 3D box, the researchers project it into the camera image to get a 2D box, extract that box's image feature with RoIAlign from a pre-trained vision backbone (the GroundingDINO backbone), and obtain a text description of the object from a large vision-language model (LLaVA) using a prompt that names the class and the box location. The description is encoded into a text feature with a pre-trained text encoder (SLIP).

When a predicted 3D box matches a label or pseudo label above an IoU threshold, the method maps its 3D feature into the image and text feature spaces and pulls it toward the matching image and text features while pushing image features of the box away from randomly sampled background boxes. Cosine-similarity losses drive this. The 3D feature and its aligned image and text features are then combined using weights learned by a small network, and the fused feature refines the detector's output (for PV-RCNN, refining the second-stage proposal; for PointPillars, refining the initial detection; for SECOND-IoU, predicting the IoU score). In the target domain, matched student and teacher detections have their 3D features aligned to each other as well.

To improve pseudo labels, an off-the-shelf 2D detector (GroundingDINO) finds 2D boxes in the target images, and those are lifted to 3D using geometric reasoning. Because the source-trained 3D detector usually gives more accurate pseudo labels but struggles at long range, image-derived 3D boxes are kept only when they are beyond a distance threshold and do not overlap teacher boxes beyond an IoU threshold; the final pseudo-label set is the union of the kept image-derived boxes and the teacher's boxes.

Why This Matters

Impact on research. Most 3D UDA work treats LiDAR alone, even though images are captured alongside point clouds in the major datasets. This paper shows that image and text features can act as a shared space that makes cross-domain alignment easier than aligning point-cloud features directly, and it demonstrates that a large vision-language model can be useful not just for pre-training but inside a 3D UDA self-training loop. It also shows a cheap way to improve pseudo labels at ranges where the 3D teacher is weak.

Real-world applications.

• Autonomous driving: adapting a detector trained in one city, sensor rig, or LiDAR beam configuration to a vehicle fleet with different sensors and environments, without new 3D annotation.
• Robot perception: transferring 3D perception to robots with different depth or LiDAR sensors than the data they were trained on.
• Fleet-scale data efficiency: reducing the cost of labeling new 3D datasets by relying on image data, which is already collected and cheaper to process with 2D detectors and vision-language models.
• Long-range detection for safety: improving detection of distant objects, the regime where image-derived pseudo labels gave the most visible change in the reported range breakdown.

Industry relevance. The method adds no test-time cost — the reported FPS drops are small (for example, 82.55 to 79.43 FPS for PointPillars) and inference remains point-cloud-only, which matters for deployment. The components it relies on (GroundingDINO, LLaVA, SLIP) are off-the-shelf, and the method plugs into three widely used detectors, which lowers the barrier to adoption.

Future Directions

• The paper's provided content is truncated at the distance-range analysis, and no explicit future-work statement is included; the directions below are open questions the work raises rather than commitments made by the authors.
• Extending the image-and-text bridge to other detection architectures and sensor configurations beyond PV-RCNN, SECOND-IoU, and PointPillars, and to other adaptation tasks beyond the three evaluated (W→N, W→K, N→K).
• Reducing dependence on specific LVLM and text encoder choices, since the reported comparisons show LLaVA beats Qwen2-VL and SLIP beats LLaMA on the tested task, leaving open how sensitive the method is to these choices and to prompt wording.
• Investigating why the method underperforms on certain subtasks, such as PointPillars and SECOND-IoU on Waymo→nuScenes and the 3.3% AP_3D gap on SECOND-IoU Waymo→KITTI, to determine when multi-modal assistance helps and when it does not.
• Using image and text information at inference or in a semi-supervised setting, since the current design deliberately restricts multi-modal use to training.

Target Audience

Researchers and graduate students working on 3D object detection, domain adaptation, and autonomous driving perception; practitioners who need to deploy LiDAR detectors across sensors or environments without new annotation; and researchers interested in applying off-the-shelf vision-language models to 3D perception tasks. Readers should already be comfortable with teacher-student self-training, pseudo-labeling, and 3D detector architectures.

Authors’ abstract

Unsupervised domain adaptation for LiDAR-based 3D object detection (3D UDA) based on the teacher-student architecture with pseudo labels has achieved notable improvements in recent years. Although it is quite popular to collect point clouds and images simultaneously, little attention has been paid to the usefulness of image data in 3D UDA when training the models. In this paper, we propose an approach named MMAssist that improves the performance of 3D UDA with multi-modal assistance. A method is designed to align 3D features between the source domain and the target domain by using image and text features as bridges. More specifically, we project the ground truth labels or pseudo labels to the images to get a set of 2D bounding boxes. For each 2D box, we extract its image feature from a pre-trained vision backbone. A large vision-language model (LVLM) is adopted to extract the box's text description, and a pre-trained text encoder is used to obtain its text feature. During the training of the model in the source domain and the student model in the target domain, we align the 3D features of the predicted boxes with their corresponding image and text features, and the 3D features and the aligned features are fused with learned weights for the final prediction. The features between the student branch and the teacher branch in the target domain are aligned as well. To enhance the pseudo labels, we use an off-the-shelf 2D object detector to generate 2D bounding boxes from images and estimate their corresponding 3D boxes with the aid of point cloud, and these 3D boxes are combined with the pseudo labels generated by the teacher model. Experimental results show that our approach achieves promising performance compared with state-of-the-art methods in three domain adaptation tasks on three popular 3D object detection datasets. The code is available at https://github.com/liangp/MMAssist.