SemGeo-Gen: Unsupervised Generation of Approximate Cross-Instance Semantic-Geometric Correspondences

AI summary

Overview

Research area: Computer Vision — cross-instance semantic correspondence, 3D shape registration, and synthetic supervision generation for image matching.

Technical level: Advanced. The paper assumes familiarity with point clouds, non-rigid registration (Chamfer distance, thin-plate splines), PCA feature reduction, vision-transformer features (DINOv2), and PCK evaluation.

Scope: A fully unsupervised pipeline that turns weakly aligned 3D object collections into large-scale approximate semantic-geometric correspondences, and a demonstration that these synthetic correspondences work as pretraining data for real-image correspondence models.

What This Paper Is About

Matching points across different instances of the same object category (two different airplanes, two different chairs) is hard, because instances differ in shape, proportion, and texture while sharing only partially overlapping parts. Training models to do this requires human-annotated keypoints, which are expensive, sparse, and inherently ambiguous — AP-10K needed 13 trained annotators, and the Human Correspondence Consensus dataset used 80 professional annotators.

The authors propose to stop chasing exact point-level matches and instead automatically generate approximate correspondences that are semantically meaningful, geometrically consistent, and abundant. Their method, SemGeo-Gen, needs only coarse category-level rotation alignment of 3D objects — no point labels, no part labels, no correspondence labels.

Key Contributions

1. An unsupervised correspondence generator. A fully automatic pipeline that produces dense approximate cross-instance semantic-geometric correspondences from weakly aligned 3D object collections, with no point, part, or correspondence supervision at any stage.

2. A three-way coupling of geometry and semantics. The method combines multi-view DINOv2 features lifted into 3D, a deformable continuous piecewise-affine (CPA) registration, and semantic consistency pruning, so matches are constrained by both shape alignment and feature compatibility.

3. Projection into rendered views for combinatorial scale. Because correspondences are established in 3D, they can be projected through calibrated cameras to yield 3D–3D, 3D–2D, or 2D–2D supervision. A category with N objects yields up to binom(N,2) cross-instance pairs, each producing many rendered view pairs.

4. Evidence that approximate supervision transfers to real images. The generated data improves recent state-of-the-art models on SPair-71k after real-data fine-tuning, with gains of up to 5 PCK@0.10 points, and agrees well with sparse human KeypointNet annotations.

Main Findings

• Agreement with human annotations (3D–3D): On KeypointNet, generated correspondences reach 93% PCK@0.10, 97% PCK@0.15, and 99% PCK@0.20, with a median GT-used rate of 57.6%.
• Per-category spread: The lowest category is Vessel (0.71 PCK@0.10, 0.45 PCK@0.05); the highest include Laptop (0.99 PCK@0.10) and Table/Mug/Skateboard/Cap (0.96–0.97 PCK@0.10). Mean across 16 categories at PCK@0.05 is 0.75.
• Pretraining beats joint training: Naive joint training with the synthetic data gives mixed results, attributed to a synthetic-to-real domain gap. Pretrain-then-fine-tune consistently helps.
• GeoAware-SC improvement: Mean PCK on the nine rigid SPair-71k categories rises from 79.36 to 83.73 (+4.37) with the pretrain-and-fine-tune regime.
• MARCO improvement: Mean PCK rises from 86.34 to 89.30 (+2.96) under the same regime.
• Generalization beyond rigid categories: Pretraining GeoAware-SC on Rigid-9 synthetic data and fine-tuning on all 18 SPair-71k categories improves overall PCK@0.10 from 82.09 to 84.48 (+2.39), with +3.72 on rigid categories, +1.29 on non-rigid categories, and +0.23 on the two symmetric categories — even though non-rigid and symmetric objects were excluded from synthetic pretraining.
• Comparison with a 3D-mesh competitor's data: DenseMatcher pretraining under the same protocol yields 81.83 overall PCK@0.10 versus 82.09 for real-only training, i.e. roughly no gain, whereas SemGeo-Gen reaches 84.48.
• 3D–2D projection holds up: Across 16 categories, projected correspondences reach 94.2% PCK@0.10 in 2D with an average GT-used rate of 45.3%; on the same visible correspondences 3D PCK@0.10 is 95.1%, slightly below the 93%-level general evaluation is attributed to geometric inaccuracies in the approximate 3D matches.
• Semantic filtering matters: Geometric registration alone gives over 84% PCK@0.10; adding correspondence-level pruning raises it to about 91%; adding pair-level filtering raises it to about 93% on retained pairs.
• Registration choice: Global affine performs substantially worse at all PCK thresholds; thin-plate splines come close to CPA but CPA is comparable or slightly better at most thresholds and roughly 3× faster.
• Rotation sensitivity: Mean PCK@0.10 falls from 93% to 89% and 85% under 5° and 10° rotation perturbations, and drops substantially for larger rotations.

Methodology in Plain English

The pipeline takes pairs of 3D objects that are already roughly rotated into a common canonical pose and normalized to the unit sphere. Everything else is automatic.

1. Paint each 3D point with semantics. Each object is rendered from several virtual cameras. Dense DINOv2 features are extracted from those renderings and back-projected onto the visible 3D points using known camera and depth information. Multiple views are averaged so every surface point ends up with one semantic descriptor.
2. Compress and compare. Descriptors are jointly reduced to 64 dimensions with PCA, and semantic similarity between two points is the cosine similarity of their reduced vectors. Points count as compatible if similarity exceeds 0.575 (equivalent to a squared Euclidean distance of 0.85 between L2-normalized descriptors).
3. Optionally reject badly matched object pairs. Before registering, the method measures bidirectional semantic coverage: the fraction of source points that have at least one compatible point in the target, and vice versa. Pairs falling below 0.6 in either direction are discarded.
4. Deform one object onto the other. A continuous piecewise-affine transformation is defined over a regular 3D control grid, split into tetrahedra by Delaunay triangulation. Each grid vertex carries a displacement vector, and a point's displacement is barycentrically interpolated from its containing tetrahedron, giving a deformation that is locally affine but continuous across cells. The deformation is optimized with a symmetric Chamfer loss plus a smoothness term over neighbouring grid vertices (weight 0.3).
5. Nearest-neighbour assignment. After alignment, each transformed source point is matched to its nearest target point. The stored correspondence links the original source point to that target point, which is why the authors describe matches as approximate.
6. Prune semantically implausible matches. Candidate pairs whose endpoint similarity falls below 0.575 are dropped, removing matches that are geometrically close but semantically wrong.
7. Project to images. With calibrated render cameras, a 3D correspondence projects to a 2D pair, keeping only those whose endpoints are visible in their views.

For SPair-71k experiments the authors generated 12K synthetic 2D training pairs per category from Objaverse-OA instances across nine rigid categories (aeroplane, bicycle, boat, bus, car, chair, motorbike, train, tvmonitor), deliberately skipping pair-level semantic filtering to preserve diversity. All experiments ran on a single NVIDIA RTX A6000 GPU.

Why This Matters

Impact on research. The paper reframes correspondence annotation as a generation problem rather than a labelling problem. It shows that approximate, automatically derived supervision — not exact point labels — is enough to improve strong models, which challenges the assumption that manual keypoints are the bottleneck. It also provides a way to obtain correspondence data for categories that no existing benchmark covers.

Real-world applications:

• Robotics and manipulation: transferring grasp or interaction points from one object instance to another within a category.
• Augmented reality and retail: fitting a template texture or product model onto a photographed instance.
• Animation and asset pipelines: reusing rigs, skeleton weights, or material assignments across a library of 3D assets.
• Image editing and alignment: pose-aware copy-paste and content transfer between images of the same object class.

Industry relevance. Companies sitting on large libraries of 3D assets (marketplaces, game studios, CAD catalogues) can convert them into training data without an annotation workforce. The requirement of approximate canonical alignment is a real constraint, but the authors point to Objaverse-OA (over 100 rigid categories after curation) and recent scalable canonicalization work as evidence that aligned assets are becoming available.

Future Directions

• Stretching the deformation model. The CPA registration cannot fully capture large cross-instance structural differences without supervision, which limits alignment quality on dissimilar shapes; stronger unsupervised deformation models are an obvious next step.
• Better semantic features. The pruning stage inherits DINOv2's failures. Because the pipeline is modular, swapping in stronger feature extractors is a direct upgrade path.
• Handling non-rigid and pose-varying objects. The method assumes rigid, approximately aligned objects. The authors suggest clustering objects by pose and applying CPA within each cluster to absorb residual variation.
• Closing the synthetic-to-real gap. The domain gap is what makes joint training fail while pretrain-then-fine-tune succeeds; growing collections of real, canonically aligned 3D assets could narrow it, and improved 3D geometry could also reduce the small accuracy loss observed when projecting correspondences into 2D.

Target Audience

Researchers and graduate students working on semantic correspondence, 3D shape matching, and non-rigid registration; practitioners who need correspondence supervision for categories without existing benchmarks; and applied engineers in 3D asset pipelines, robotics, or AR who already hold collections of aligned 3D models and want to turn them into training data. Readers should be comfortable with point-cloud registration and deep visual features, though the high-level intuition is accessible to anyone familiar with keypoint matching benchmarks.

Authors’ abstract

We introduce SemGeo-Gen, an unsupervised method for generating approximate cross-instance semantic-geometric supervision from weakly aligned 3D object collections. Our goal is not to recover exact point-level correspondences, which are often ambiguous across different instances and would require expensive, impractical manual supervision at scale. Instead, we automatically produce approximate correspondences that are semantically meaningful, geometrically consistent, and diverse enough to train modern correspondence models. Given only coarse category-level rotation alignment, SemGeo-Gen lifts multi-view DINOv2 features to 3D, aligns object instances using continuous piecewise-affine registration, and prunes candidate matches using semantic consistency. The resulting 3D correspondences can be projected into rendered views, yielding scalable 3D-3D, 3D-2D, and 2D-2D supervision without manual annotation. We validate the generated correspondences against sparse human annotations from KeypointNet, obtaining 93% PCK@0.10 and 97% PCK@0.15. More importantly, we show that using the generated correspondences for synthetic pretraining improves recent state-of-the-art semantic correspondence models on SPair-71k after fine-tuning on real data, with gains of up to 5 PCK@0.10 points. Together, these results indicate that approximate semantic-geometric supervision generated from 3D assets can improve real-image correspondence learning and serve as a scalable alternative to costly manual annotation. Project page: https://bgu-cs-vil.github.io/SemGeo-Gen/.