Geometry-to-Image Synthesis-Driven Generative Point Cloud Registration

AI summary

Overview

Research area: 3D computer vision — point cloud registration, combining diffusion-based 2D image generation (Stable Diffusion, ControlNet) with 3D geometric matching.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, ControlNet conditioning, self-attention, rigid transformation estimation (SE(3)/SO(3)), and point cloud descriptors.

Scope in one sentence: The paper proposes "Generative Point Cloud Registration," a paradigm that synthesizes cross-view-consistent RGB images from geometry-only point clouds so that color features can be fused into 3D descriptors for more robust matching, with separate generative models for depth-camera and LiDAR settings.

What This Paper Is About

Standard point cloud registration matches source and target clouds using geometry alone, which becomes unreliable under low overlap, repetitive patterns, or noise. Prior RGB-D registration work shows that real color and semantic cues make descriptors far more distinctive, but in geometry-only registration those RGB images simply do not exist. This paper's goal is to generate the missing color: it builds matching-specific 2D generative models that produce a source image and a target image aligned to the two point clouds, then fuses those generated colors into geometric descriptors to improve correspondence and pose estimation.

Key Contributions

1. A new paradigm — Generative Point Cloud Registration. Instead of single-image generation, the framework performs pairwise generation, producing one image per point cloud. The generated pair must satisfy two stated criteria: 2D-3D geometric consistency (images preserve the spatial layout of their own point clouds) and cross-view texture consistency (corresponding regions look alike, so feature similarity is not degraded).

2. DepthMatch-ControlNet for depth-camera registration. Built on ControlNet, it converts point clouds into depth maps via the intrinsic matrix and conditions generation on them. It adds two designs — coupled conditional denoising and coupled prompt guidance — for cross-view texture consistency, and supports both zero-shot operation and few-shot fine-tuning.

3. LiDARMatch-ControlNet for LiDAR registration. This extends the framework to 360-degree LiDAR by projecting each point cloud into an equirectangular range map (via spherical coordinates: elevation from arccos(z/||p||₂), azimuth from arctan2(y, x)) as the generation condition, producing panoramic RGB images. The authors state this is the first successful realization of LiDAR point cloud-to-panoramic image generation.

4. A general, plug-and-play framework. A zero-shot geometric-color fusion mechanism uses pretrained large vision models (DINOv2 and Stable Diffusion) to extract representations from generated images and enhance geometric descriptors by weighted concatenation. An XYZ-RGB fusion scheme is also introduced. The framework is claimed to be integrable with a wide range of existing registration methods.

Additionally, the paper provides a theoretical analysis of the coupled denoising diffusion mechanism (Section III-E), showing that it models the joint distribution of cross-view images and enables consistent generation.

Main Findings

• Independent ControlNet denoising produces texture inconsistency. When source and target images are generated by two separate denoising passes, neither is aware of the colors the other produces, so corresponding regions differ (illustrated in Fig. 3, left).

• Coupling the latent representations enables cross-view message passing with no architectural change. The noisy latent of shape [H′, W′, d] is expanded to [2H′, W′, d], and the source and target depth maps are vertically concatenated into a [2H, W, 1] condition. The self-attention over this coupled latent lets each element attend to all elements of both images, giving long-range cross-view dependency and texture-aware message passing while reusing the original conditional denoiser.

• A specific "coupled prompt" unlocks zero-shot pairwise consistency. The authors report that a prompt instructing the model to "generate two vertically stacked images that are captured from the different viewpoints in a same scene" with the same environment and "very subtle differences," with the same layout and key elements, guides the denoiser to recover consistent textures without any fine-tuning. They state they are the first to uncover and use this inherent capability of pretrained ControlNet for zero-shot pairwise generation.

• Few-shot fine-tuning improves consistency further. Only the learnable encoder copy of DepthMatch-ControlNet is fine-tuned, not all parameters, preserving Stable Diffusion's generative ability. Fine-tuning uses a standard denoising objective (Eq. 5) on coupled RGB/depth pairs, and approximately 3K samples are reported as sufficient to improve consistency quality.

• Panoramic generation cannot be done zero-shot. Because no off-the-shelf pretrained ControlNet is conditioned on range maps for panoramic image generation, LiDARMatch-ControlNet requires consistency fine-tuning instead.

• Dataset field-of-view analysis drove the LiDAR data choice. Table I compares LiDAR datasets: KITTI (1 stereo camera, <360° azimuth, <180° polar), KITTI-360 (1 stereo + 2 fisheye, 360°, 120°), Waymo (5 perspective, <360°, <180°), nuScenes (5 perspective + 1 fisheye, 360°, 40°), and Dur360BEV (1 spherical camera, 360°, 180°). Dur360BEV was selected for fine-tuning because it uniquely provides genuinely panoramic images with a complete 360° horizontal and 180° vertical field of view.

• Approximately 10K panoramic pairs from Dur360BEV are reported as sufficient for consistent panoramic image generation during fine-tuning.

• Effectiveness is claimed on 3DMatch and ScanNet (depth-camera settings) and Dur360BEV (LiDAR settings). The provided content describes these experiments as validating the approach but does not report the quantitative tables or metric values.

• Publication provenance: a preliminary version was accepted at ICML 2025, covering only the depth-camera case; the extended version adds the LiDAR path, the theoretical analysis, and Dur360BEV comparisons.

Methodology in Plain English

The authors treat missing color as a generation problem rather than an acquisition problem.

1. Turn geometry into an image-like condition. For depth cameras, the point cloud is projected into a depth map using the camera intrinsics. For LiDAR, each 3D point is converted into elevation and azimuth angles and rasterized into an equirectangular range map that stores the radial distance per pixel (nearest point kept when several map to the same pixel).

2. Condition the generator on that map. ControlNet's zero convolutions encode the condition map, and the resulting features are injected into the denoiser's skip connections, so the generated RGB image preserves the geometry of the point cloud — satisfying 2D-3D geometric consistency. This follows ControlNet's existing depth-conditioned generation capability.

3. Generate the two images together rather than separately. The source and target condition maps are stacked vertically into one map, and their noisy latents are stacked into one tensor. Running the unmodified denoiser on this stacked tensor means the internal self-attention spans both images, so the two views exchange texture information — satisfying cross-view texture consistency.

4. Tell the model what is expected. A hand-written "coupled prompt" describes two vertically stacked views of the same scene with subtle differences. This steers the denoiser toward consistent layouts without training. For LiDAR, a panoramic coupled prompt describes two surround-view panoramas from a self-driving vehicle in the same urban street environment.

5. Optionally fine-tune lightly. When zero-shot consistency is not enough (perspective case) or is not available at all (panoramic case), only the learnable encoder copy is trained on coupled image/condition pairs using the standard diffusion noise-prediction loss.

6. Fuse color into matching. Zero-shot representations from pretrained vision models (DINOv2, Stable Diffusion) are extracted from the generated images and combined with geometric descriptors by weighted concatenation, or via the XYZ-RGB fusion scheme, before correspondences are estimated and the rigid transformation recovered by solving the least-squares alignment over putative correspondences.

Why This Matters

Impact on research. The paper argues that geometry-only registration has been artificially limited by the absence of RGB data, and that generative models can supply that missing modality. It also claims a first in cross-modal generation (LiDAR point clouds to panoramic RGB) and provides a theoretical account of why coupled denoising models the joint cross-view distribution. Because the framework is described as plug-and-play, its value is as a general enhancement layer on top of existing registration pipelines rather than a replacement for them.

Real-world applications (as named in the paper):

• 3D reconstruction
• LiDAR SLAM
• Object localization
• Autonomous driving / self-driving contexts (implicit in the panoramic, surround-view generation setup and the Dur360BEV dataset)

Industry relevance. The work targets the two dominant sensing modalities in robotics and autonomous systems: depth cameras (partial, single-view observations typical of indoor scanning datasets such as 3DMatch and ScanNet) and 360-degree LiDAR (sparse, omnidirectional sweeps). Enhancing registration accuracy without acquiring new color sensors would reduce hardware and calibration burden, and the LiDAR-to-panorama generation route is directly relevant to surround-view perception stacks.

Future Directions

1. Unifying the two variants. The authors explicitly note that a unified point cloud-to-image generative model covering both depth-camera and LiDAR registration is inherently challenging because of the single-view versus omnidirectional disparity and the perspective versus panoramic image formats. Whether one model can serve both remains open.

2. Extending beyond the current fine-tuning data. LiDARMatch-ControlNet must be fine-tuned because no pretrained range-map-conditioned panoramic ControlNet exists. Building or discovering such a prior could bring zero-shot panoramic generation within reach.

3. Broadening the dataset and sensor coverage. Evaluation is reported on 3DMatch, ScanNet, and Dur360BEV. Extending to other LiDAR datasets (KITTI, KITTI-360, Waymo, nuScenes are discussed but only for field-of-view analysis) and to outdoor depth-camera scenarios would test generality.

4. Pushing consistency further. The paper acknowledges that even after fine-tuning, some corresponding regions may still show geometric or texture inconsistency. Reducing residual inconsistency without enlarging the fine-tuning set is a natural next target.

Target Audience

This paper is best suited to researchers and graduate students working on 3D registration, point cloud processing, SLAM, and multimodal perception, as well as practitioners who want to strengthen existing matching pipelines without adding color sensors. It will be most useful to readers already comfortable with diffusion models and ControlNet conditioning; readers without that background will need to consult the cited Stable Diffusion and ControlNet work first. Those specifically interested in 360-degree/panoramic representation learning and LiDAR-to-image generation will find the LiDARMatch-ControlNet design the most distinctive part.

Authors’ abstract

In this paper, we propose a novel 3D registration paradigm, Generative Point Cloud Registration, which bridges advanced 2D generative models with 3D matching tasks to enhance registration performance. Our key idea is to generate cross-view consistent image pairs that are well-aligned with the source and target point clouds, enabling geometry-color feature fusion to facilitate robust matching. To ensure high-quality matching, the generated image pair should feature both 2D-3D geometric consistency and cross-view texture consistency. To this end, we introduce DepthMatch-ControlNet and LiDARMatch-ControlNet, two matching-specific, controllable 2D generative models. Specifically, for depth camera-based 3D registration with point clouds derived from the depth maps, DepthMatch-ControlNet leverages the depth-conditioned generation capabilities of ControlNet to synthesize perspective-view RGB images that are geometrically consistent with depth maps, ensuring accurate 2D-3D alignment. Additionally, by incorporating a coupled conditional denoising scheme and coupled prompt guidance, it further promotes cross-view feature interaction, guiding texture consistency generation. To address LiDAR-based 3D registration with point clouds captured by LiDAR sensors, LiDARMatch-ControlNet extends this framework by conditioning on paired equirectangular range maps projected from 360-degree LiDAR point clouds, generating corresponding panoramic RGB images. Our generative 3D registration paradigm is general and can be seamlessly integrated into a wide range of existing registration methods to improve their performance. Extensive experiments on the 3DMatch and ScanNet datasets (for depth-camera settings), as well as the Dur360BEV dataset (for LiDAR settings), demonstrate the effectiveness of our approach.