Gaussian-Constrained LeJEPA Representations for Unsupervised Scene Discovery and Pose Consistency

AI summary

Overview

Research area: Computer vision — unsupervised 3D scene reconstruction, self-supervised representation learning, and Structure from Motion (SfM), evaluated on the Image Matching Challenge 2025 (IMC2025).

Technical level: Advanced. The paper assumes familiarity with self-supervised learning losses, image embeddings, covariance/eigenvalue analysis, and SfM pipelines such as COLMAP and OpenMVG.

Scope: The paper empirically tests whether Gaussian constraints from the LeJEPA framework, implemented as Sliced Isotropic Gaussian Regularization (SIGReg) on image embeddings, improve unsupervised scene discovery and camera pose estimation within an IMC2025 competition pipeline.

What This Paper Is About

Structure from Motion pipelines work well on controlled collections but break down when a dataset mixes images from several unrelated scenes, contains outliers, and has ambiguous visual content. The Image Matching Challenge 2025 poses exactly this setting: images must first be partitioned into scenes, then camera poses must be estimated per scene. The paper asks whether theoretically motivated Gaussian constraints on learned image embeddings can replace the hand-engineered heuristics (contrastive losses, negative sampling) that typical self-supervised methods rely on, and whether this helps in practice.

Key Contributions

1. One of the first empirical studies applying Gaussian-constrained self-supervised representations to unsupervised scene discovery, rather than only to standard representation benchmarks.
2. Three progressively refined pipelines: a score-optimized traditional approach, a generalized robust solution, and a LeJEPA-enhanced solution, with the third replacing heuristic similarity measures with Gaussian-derived ones.
3. An implementation of LeJEPA's SIGReg principle as a clustering validity check, using the eigenvalue ratio and mean norm of cluster embeddings to decide whether a cluster satisfies isotropic Gaussian constraints.
4. A quantitative comparison of all three pipelines on IMC2025 public and private leaderboard scores, scene counts, outlier counts, and average scene sizes.

Main Findings

• Private score tied, not improved, against the generalized baseline: the LeJEPA-enhanced pipeline scored 0.87 public and 0.54 private, identical to the generalized robust pipeline. The score-optimized pipeline scored 0.96 public but only 0.19 private, indicating heavy overfitting to the public split.
• Better scene separation: the LeJEPA-enhanced pipeline produced 7 scenes versus 4 for the generalized pipeline and 6 for the score-optimized pipeline.
• Fewer outliers: the LeJEPA-enhanced pipeline had 7 outliers (9.6%), compared with 12 (16.4%) for the generalized pipeline and 10 (13.7%) for the score-optimized pipeline, on an average scene size of 9.4 versus 15.2 and 10.5 respectively.
• Gaussian cosine similarity is derived, not learned: the similarity used is s_ij = 1/2 (1 + z_i^T z_j / (||z_i|| ||z_j||)), justified by the claim that for isotropic Gaussians inner products correlate with the probability of common origin.
• Characteristic function matching as an alternative similarity: s_ij = 1 - |φ_i(t) - φ_j(t)|, where φ_i(t) = E[e^{itz_i}] is the characteristic function, inspired by LeJEPA's Epps-Pulley test.
• Cluster validation thresholds: a cluster is accepted as satisfying the isotropic condition when λ_max(Σ)/λ_min(Σ) < 10 and ||μ|| < 1.0, where Σ is the cluster covariance and μ the cluster mean.
• Pipeline design differences drive most of the variance: Pipeline 1 used RootSIFT with CLAHE preprocessing, FLANN matching with geometric verification, DBSCAN ensemble clustering with consensus, and circular trajectory pose heuristics. Pipeline 2 used normalized SIFT with bilateral filtering, adaptive FLANN plus brute-force matching, data-driven parameter estimation, and scene-type inference (planar/linear/object-centric).
• Test set size limits statistical confidence: the hidden test set contained roughly 1,300 images, which the paper lists as a limitation.

Methodology in Plain English

The author built three versions of the same competition pipeline. Each one loads a set of images, extracts local features, compares images to build a similarity matrix, groups the images into scenes, generates camera poses, and formats a submission.

The first pipeline is tuned aggressively for the competition metric, which is the harmonic mean of mAA (a recall-style pose measure) and clustering precision, defined as |S ∩ C| / |C|. The tuning targets cluster size balance and outlier removal. The second pipeline deliberately avoids dataset-specific tuning, inferring parameters from the data and guessing whether a scene is planar, linear, or object-centric before generating poses.

The third pipeline adds the LeJEPA idea. An encoder passes images through a backbone, normalizes the resulting features to unit length, and then rescales them by the square root of the embedding dimension. This is the SIGReg step, which is meant to push embeddings toward an isotropic Gaussian distribution. Matching between images then uses either the Gaussian cosine similarity or the characteristic function matching described above. Finally, before accepting a cluster as a valid scene, the pipeline checks whether the cluster's embeddings look isotropic: the ratio of the largest to smallest covariance eigenvalue must be below 10, and the norm of the mean embedding must be below 1.0.

Why This Matters

Impact on research. The paper argues that self-supervised methods such as SimCLR, MoCo, and BYOL depend on engineered heuristics — negative sampling and contrastive losses — that LeJEPA identifies as theoretically unnecessary. By testing Gaussian constraints on a messy real-world task rather than a clean benchmark, it provides evidence about how far those theoretical claims transfer. The result is a mixed picture: the constraints coincided with better scene separation and fewer outliers but did not raise the private leaderboard score above the simpler generalized pipeline.

Real-world applications named in the paper:

• Cultural heritage preservation and museum digitization projects.
• Urban planning from large collections of user-generated imagery.
• Autonomous navigation.
• Augmented reality systems.

Industry relevance. Platforms such as Flickr and Instagram, along with museum digitization efforts, produce millions of unstructured images daily. Pipelines that can automatically sort mixed, outlier-laden image collections into coherent scenes and recover camera poses are directly useful for map building, 3D asset creation, and any workflow that ingests uncurated photo sets.

Future Directions

• Train proper LeJEPA encoders end-to-end, rather than applying Gaussian constraints as a post-hoc normalization and validation step.
• Scale the approach to larger datasets, since the hidden test set here was only about 1,300 images.
• Extend the Gaussian framework to more complex distribution models beyond the isotropic Gaussian assumption.
• Investigate whether the improvement in scene separation and outlier rate can be converted into private leaderboard gains, given that the LeJEPA-enhanced and generalized pipelines tied at 0.54 private and 0.87 public.

Target Audience

Researchers and practitioners working at the intersection of self-supervised learning and 3D reconstruction, particularly those familiar with SfM pipelines and image matching. Competition participants in IMC2025 or similar benchmarks will find the pipeline progression and the leaderboard breakdown instructive. Readers looking for a rigorous theoretical treatment should note that the paper explicitly states it does not introduce new theoretical guarantees and positions itself as an empirical evaluation.

Authors’ abstract

Unsupervised 3D scene reconstruction from unstructured image collections remains a fundamental challenge in computer vision, particularly when images originate from multiple unrelated scenes and contain significant visual ambiguity. The Image Matching Challenge 2025 (IMC2025) highlights these difficulties by requiring both scene discovery and camera pose estimation under real-world conditions, including outliers and mixed content. This paper investigates the application of Gaussian-constrained representations inspired by LeJEPA (Joint Embedding Predictive Architecture) to address these challenges. We present three progressively refined pipelines, culminating in a LeJEPA-inspired approach that enforces isotropic Gaussian constraints on learned image embeddings. Rather than introducing new theoretical guarantees, our work empirically evaluates how these constraints influence clustering consistency and pose estimation robustness in practice. Experimental results on IMC2025 demonstrate that Gaussian-constrained embeddings can improve scene separation and pose plausibility compared to heuristic-driven baselines, particularly in visually ambiguous settings. These findings suggest that theoretically motivated representation constraints offer a promising direction for bridging self-supervised learning principles and practical structure-from-motion pipelines.