Self-Supervised Learning by Curvature Alignment

AI summary

Overview

Research area: Self-supervised visual representation learning (non-contrastive / redundancy-reduction methods) with a discrete-geometry twist.

Technical level: Intermediate. The paper assumes familiarity with self-supervised learning objectives (Barlow Twins, VICReg), encoder–projector architectures, and linear probe evaluation, and it introduces curvature concepts from differential and discrete geometry.

Scope: The paper proposes CurvSSL and kernel CurvSSL, two curvature-regularized self-supervised learning objectives that augment a Barlow Twins-style redundancy-reduction loss with a regularizer on the local manifold geometry of embeddings, evaluated on MNIST and CIFAR-10 with a ResNet-18 backbone.

What This Paper Is About

Modern non-contrastive self-supervised methods shape the first- and second-order statistics of learned embeddings (means, variances, cross-correlations), but they largely ignore the local geometry of the data manifold. Two augmentations of the same image can therefore land close together in Euclidean space while their local neighborhoods, tangent directions, or bending behave differently, distorting the structure that nearest-neighbor retrieval and clustering rely on. The goal of this paper is to add an explicit geometric constraint — discrete curvature computed from each embedding's k nearest neighbors — on top of a standard redundancy-reduction objective, so that the learned representation preserves consistent local manifold bending across views.

Key Contributions

1. A curvature score for embeddings. Each projected embedding is treated as a vertex of a hypothetical polyhedron whose faces are spanned by its k nearest neighbors. Neighbor differences are translated to the origin, normalized onto the unit hypersphere, and aggregated through pairwise cosine similarities to yield a scalar curvature score (a relaxation of Descartes' angular defect, which sums face angles around a vertex relative to 2π).

2. CurvSSL: a curvature-regularized SSL objective. A standard two-view encoder–projector architecture with a Barlow Twins loss on projected features is augmented with a Barlow-style loss applied to a curvature-derived matrix. This aligns curvature scores across augmentations of the same sample (diagonal entries near 1) and decorrelates curvature patterns across different samples (off-diagonal entries near 0). The total loss is the embedding-level term plus a weighted curvature term.

3. Kernel CurvSSL: an RKHS extension. Both the k-NN search and the curvature computation are moved into a reproducing kernel Hilbert space. The kernel curvature score is computed from a normalized local Gram matrix of neighbor edges, allowing nonlinear curvature patterns to be captured. An RBF kernel was used in experiments.

4. Empirical evaluation and geometry inspection. Linear evaluation on MNIST and CIFAR-10 with a frozen ResNet-18 encoder, plus UMAP visualizations of the learned features, showing competitive or improved accuracy over Barlow Twins and VICReg.

Main Findings

• Linear probe accuracy on MNIST: VicReg 95.9%, Barlow Twins 94.9%, CurvSSL 97.9%, Kernel CurvSSL 98.4%.

• Linear probe accuracy on CIFAR-10: VicReg 74.5%, Barlow Twins 73.6%, CurvSSL 75.1%, Kernel CurvSSL 76.5%.

• Kernel variant outperforms the Euclidean variant. The paper attributes this to the RKHS handling nonlinearities better.

• The curvature term does not prevent learning discriminative features. The paper states that on the harder CIFAR-10 benchmark the curvature term "does not prevent the model from learning discriminative features and can improve class separation compared to using redundancy reduction alone."

• Curvature regularization produces more coherent clusters. On MNIST, UMAP plots of CurvSSL and kernel CurvSSL show well-separated digit clusters with relatively smooth transitions between visually similar digits such as '3' and '5', with tighter and more coherent class clusters attributed to geometrically consistent local neighborhoods across augmentations.

• CIFAR-10 embeddings form more complex structures, reflecting higher intra-class variability of natural images, but classes still occupy distinct regions with meaningful local neighborhoods.

• Hyperparameters were fixed across datasets. The projection dimension is 128, k = 10 neighbors, mini-batch size 256, and λ_emb = λ_curv = α_curv = 1 were selected once and reused for both datasets. The model was trained for 100 epochs on MNIST and 500 on CIFAR-10 with Adam at learning rate 10^-3 and weight decay 10^-4. UMAP used n_neighbors = 15 and min_dist = 0.1.

Methodology in Plain English

The authors keep the standard two-view self-supervised recipe: take an image, apply two random augmentations to it, push both through a shared ResNet-18 encoder and a two-layer MLP projection head, and get two embeddings. They then train with a loss that has two parts.

The first part is the familiar Barlow Twins idea. Normalize the projected embeddings per feature dimension across the batch, compute the cross-correlation matrix between the two views, push its diagonal entries toward 1 (so the two views agree feature by feature) and its off-diagonal entries toward 0 (so features carry distinct information).

The second part is new and operates on geometry rather than coordinates. For each embedding in the batch, find its k nearest neighbors in the projected space. Imagine that embedding as the corner of a shape, and treat the lines from it to each neighbor as edges. Translate those edges so they start at the origin, shrink them all to unit length, and measure the cosine of the angles between every pair of edges. Summing those cosines gives a single number per embedding: the curvature score, which reflects how sharply the neighborhood bends around that point. Do the same for the embeddings from the second view.

Now stack the scores of the two views into two vectors, normalize them, and multiply them into a batch-by-batch matrix. Apply the same diagonal-pull-to-1, off-diagonal-push-to-0 treatment as before. This makes the curvature of an image's two augmented views agree, while making curvature patterns of different images unrelated — so the model not only learns invariant, non-redundant features but also consistent local bending.

In the kernel version, all of this neighbor-finding and angle-measuring happens in a higher-dimensional feature space defined by a kernel function, using a normalized local Gram matrix instead of raw cosine similarities. Intuitively, the kernel lets the method detect curvature patterns that are nonlinear in the original embedding space.

After pretraining, the encoder is frozen and the projection head is discarded. A small classifier (a linear layer with one hidden layer and batch normalization) is trained with cross-entropy on top of frozen features for a fixed number of epochs (e.g., 50) with SGD, and top-1 test accuracy is reported.

Why This Matters

Impact on research. The paper argues that self-supervised learning has been dominated by statistical regularizers on embedding coordinates, and that explicitly shaping local geometry is a simple and largely unexplored complement. It offers a concrete, drop-in regularizer that works with existing two-view pipelines, connecting SSL to discrete differential geometry (angular defect, Euler-Poincaré characteristic) and to kernel methods. The claim is not that curvature replaces invariance or redundancy reduction, but that it adds a dimension of control those objectives lack.

Real-world applications the paper motivates:

• Nearest-neighbor retrieval, where distorted local neighborhoods directly degrade which items are returned for a query.
• Clustering, which depends on the local structure of the representation rather than global statistics alone.
• Semi-supervised learning, where limited labels must be propagated through a well-behaved feature manifold.
• Transfer learning with frozen encoders, since the evaluation protocol here (freeze the encoder, train a small classifier) matches how pretrained SSL features are often reused downstream.

Industry relevance. Self-supervised pretraining followed by a linear probe is a standard industrial workflow for vision systems where labels are expensive. A regularizer that improves the geometry of frozen features without changing the architecture is attractive because it can be added to an existing training pipeline with minimal engineering cost. The paper demonstrates this only at the MNIST and CIFAR-10 scale, so the practical industrial claim remains untested on large-scale data.

Future Directions

• Scaling to larger datasets and architectures. Only MNIST and CIFAR-10 with ResNet-18 were tested; the paper explicitly names larger datasets and architectures as the next step. No ImageNet-scale or transformer-based results are reported.
• Manifold-sensitive downstream tasks. The authors propose semi-supervised learning and retrieval as settings where preserving local geometry should pay off most, but they do not report experiments on those tasks — only linear classification and UMAP visualizations.
• Hyperparameter and kernel selection. k = 10 neighbors, the projection dimension of 128, the loss weights, and the RBF kernel were fixed rather than tuned; how sensitive the method is to these choices, and whether adaptive or learned kernels help, is not reported.
• Theoretical grounding. The curvature score is a deliberate relaxation of exact polyhedron angular defect (it sums cosines over all edge pairs rather than only pairs sharing a face). The paper offers an informal justification for why this relaxation is valid but no formal guarantees about what the resulting regularizer optimizes or when it provably improves representation quality.

Target Audience

Researchers and graduate students working on self-supervised representation learning, particularly those interested in non-contrastive methods such as Barlow Twins and VICReg, and anyone exploring geometric or manifold-aware objectives for deep learning. It is also relevant to practitioners who build frozen-encoder feature extractors and care about retrieval, clustering, or semi-supervised pipelines, though they should note the evaluation is limited to MNIST and CIFAR-10. Readers without a background in self-supervised learning or discrete geometry will find the curvature derivations in Sections 2 and 3 demanding.

Authors’ abstract

Self-supervised learning (SSL) has recently advanced through non-contrastive methods that couple an invariance term with variance, covariance, or redundancy-reduction penalties. While such objectives shape first- and second-order statistics of the representation, they largely ignore the local geometry of the underlying data manifold. In this paper, we introduce CurvSSL, a curvature-regularized self-supervised learning framework, and its RKHS extension, kernel CurvSSL. Our approach retains a standard two-view encoder-projector architecture with a Barlow Twins-style redundancy-reduction loss on projected features, but augments it with a curvature-based regularizer. Each embedding is treated as a vertex whose $k$ nearest neighbors define a discrete curvature score via cosine interactions on the unit hypersphere; in the kernel variant, curvature is computed from a normalized local Gram matrix in an RKHS. These scores are aligned and decorrelated across augmentations by a Barlow-style loss on a curvature-derived matrix, encouraging both view invariance and consistency of local manifold bending. Experiments on MNIST and CIFAR-10 datasets with a ResNet-18 backbone show that curvature-regularized SSL yields competitive or improved linear evaluation performance compared to Barlow Twins and VICReg. Our results indicate that explicitly shaping local geometry is a simple and effective complement to purely statistical SSL regularizers.