Can Local Learning Match Self-Supervised Backpropagation?

AI summary

Overview

Research area: Self-supervised learning (SSL), biologically plausible credit assignment, and theory of local learning rules as alternatives to backpropagation (BP) in deep networks.

Technical level: Advanced. The paper contains formal theorems with proofs (in appendices) for deep linear and semi-orthonormal networks, a unified loss formalism, and large-scale empirical benchmarks; the conceptual framing, however, is readable for non-specialists.

Scope: The paper asks whether layer-wise, backprop-free self-supervised learning rules can be made mathematically and empirically equivalent to end-to-end self-supervised backpropagation, and tests this on image benchmarks up to ImageNet.

What This Paper Is About

Backpropagation requires a structured backward pass that biological networks do not appear to have, so researchers have proposed "local" learning rules in which each layer is updated using only signals available at that layer. In the supervised setting these local rules can approach BP, but in self-supervised learning they perform considerably worse, and no theory connected them to global representation learning. This paper develops such a theory for deep linear networks, uses it to derive modified local-SSL algorithms, and shows that the resulting rule (CLAPP++) matches a comparable global BP-SSL baseline on several image benchmarks.

Key Contributions

1. A theory of exact equivalence. For deep linear networks with orthonormal weight matrices, the authors prove (Theorem 3.1) that a broad class of layer-wise contrastive local-SSL losses produces weight updates identical to those of global BP-SSL on the last-layer loss, provided the feedback/reference matrix is either fixed to identity or is optimally learned and convexity holds.
2. A theory of direct feedback. When the orthonormality condition fails — specifically when the number of neurons shrinks across layers — they show (Theorem 3.3) that adding "direct feedback" from the last layer (local-SSL DFB) yields updates provably closer to global BP-SSL gradients than standard local-SSL.
3. A theory-guided convnet modification. For convolutional networks they show theoretically (Proposition 3.5) and numerically that feedback weights need a structured 2D spatial dependence, replacing global average pooling of features with patch-wise pooling, so the gradient is shared only over a patch rather than the whole image.
4. A new state of the art for local-SSL. The resulting variants (CLAPP++, CLAPP++DFB and CLAPP++both) improve accuracy on CIFAR-10, STL-10, Tiny ImageNet and ImageNet, reaching the performance of a comparable global BP-SSL baseline on CIFAR-10, STL-10 and Tiny ImageNet.

Main Findings

• Exact match in the ideal linear case. In a 6-layer linear network with 128 neurons per layer and orthonormal weights, trained on a fixed batch of MNIST inputs, the local-SSL gradient step and the global BP-SSL gradient step were numerically identical, achieving the maximum cosine similarity of one.
• Every dropped condition reduces alignment. Removing learnability of B (fixed random B), orthonormality of W, or linearity (ReLU MLP) each lowered gradient alignment, with the drop being more significant in earlier layers when orthonormality and linearity were removed simultaneously.
• Direct feedback helps when widths shrink. On a 6-layer linear network whose width halves each layer (from 128 down to 4), local-SSL with direct feedback improved alignment with BP-SSL gradients, both for the linear f(x) = -x used in the theorem and for the convex, decreasing softplus f(x) = log(1 + exp(-x)). The same trend appeared in a 6-layer ReLU MLP on MNIST with standard initialization, batched inputs and B optimized online, where training B mattered (fixed random B performed worse).
• Local-SSL DFB approaches the best achievable local update. The authors define a "theoretical optimum" local update (Definition 3.4) whose feedback matrices are optimized directly to match BP gradients, an upper bound that itself requires BP to compute. In the ReLU MLP, the margin between local-SSL DFB and this upper bound was narrow.
• Spatial dependence improves convnet alignment. In a 4-layer linear convnet (kernel and stride of length 2), 2D spatial dependence improved gradient alignment with BP-SSL, and combining it with direct feedback improved alignment further. In VGG models trained on STL-10, the alignment gain persisted during training (measurements taken after 100 epochs).
• Benchmark accuracy of the new rules. On CIFAR-10 / STL-10 / Tiny ImageNet / ImageNet: CLAPP++ reached 80.51 ± 0.27 / 78.66 ± 0.16 / 36.63 ± 0.32 / 42.55; CLAPP++DFB reached 80.65 ± 0.18 / 79.38 ± 0.17 / 36.70 ± 0.24 / 44.16; CLAPP++both reached 81.18 ± 0.18 / 79.62 ± 0.28 / 37.78 ± 0.67 / 42.49. Confidence intervals are 95% intervals from 4 random seeds.
• Comparison to BP baselines. BP-CLAPP++ scored 80.49 ± 0.20 / 80.36 ± 0.26 / 37.55 ± 0.61 / 48.52 and BP-InfoNCE scored 80.69 ± 0.84 / 81.97 ± 1.23 / 36.78 ± 0.73 / 55.19 across the same four datasets (ImageNet values reported without intervals). The paper reports that CLAPP++ reaches BP performance on the first three datasets but not on ImageNet.
• Comparison to previous local-SSL state of the art. Reported figures include CLAPP 73.6 on STL-10; LPL 59.4 on CIFAR-10 and 63.2 on STL-10; SoftHebb 80.31 on CIFAR-10, 76.23 on STL-10 and 27.3 on ImageNet; Stochastic FF 76.96 on CIFAR-10; SCFF 80.60 on CIFAR-10, 77.14 on STL-10 and 35.67 on Tiny ImageNet. The authors state that all CLAPP++ variants with spatial dependence exceed these on all datasets.
• Memory and time footprint. With batch size 256 on an A100 GPU using a 6-layer VGG on STL-10, peak VRAM was 5.47 GB (146 ms) for BP-CLAPP++, 3.38 GB (125 ms) for CLAPP++ without 2D spatial dependence, 3.38 GB (146 ms) for CLAPP++ and 3.45 GB (141 ms) for CLAPP++DFB. With an 8-layer VGG on ImageNet the corresponding values were 11.10 GB (384 ms), 4.60 GB (330 ms), 4.60 GB (376 ms) and 4.69 GB (360 ms).
• Ablation attribution. Ablations indicate that the largest performance gains on the four datasets came from introducing spatial dependence; without it, CLAPP++ scored 73.21 / 75.10 / 28.18 / 38.31.
• Naive BP-free alternatives underperformed. Naive CLAPP trained with Direct Feedback Alignment scored 52.30, and naive CLAPP with Predictive Coding scored 46.52 and 36.75 (the reported table places these under the CIFAR-10 and STL-10 columns; the truncated content does not clearly map them), which is why the authors focused on existing local-SSL rules.
• The theory does not cover normalized losses. Theorem 3.1 could not be extended to linear networks that include normalization of layer activities, which is required for Forward-forward algorithms, so the analysis focuses on CLAPP-like normalization-free rules with a learnable feedback matrix.

Methodology in Plain English

The authors first define one shared notation that can express several existing local-SSL algorithms (CLAPP, Forward-forward, PhyLL and SCFF) as a single contrastive scheme: each layer compares a positive sample against a negative sample using a score computed with a matrix B, and a decreasing function f turns the difference into a loss. Forward-forward uses the identity for B and normalization; CLAPP uses a learnable projection and no normalization.

They then restrict attention to deep linear networks with orthonormal weight matrices and ask when the layer-wise gradient equals the gradient that backpropagation would send to that layer from the final-layer loss. The key step is showing that the local layer's optimal B equals the product of the downstream weight matrices transposed and multiplied by the last-layer B, which holds exactly when the weights are orthonormal (since the products cancel) or when the loss is strictly convex in B so its minimizer is unique. They verify this numerically and then break one assumption at a time to see how much alignment survives.

Because orthonormality forces every layer to have the same number of neurons, they next study networks that shrink in width and propose feeding the last layer's activity back as the reference for every layer (direct feedback). They prove a Frobenius-norm inequality showing this variant is closer to the BP gradient, and confirm it in simulations, including a 6-layer ReLU MLP on MNIST where B is optimized by gradient descent rather than solved exactly. They also construct an "upper bound" local update whose B is trained to directly minimize the difference to BP gradients, to see how much room is left.

For convolutional networks they note that averaging feature maps over all spatial locations forces the same gradient everywhere in the map, which conflicts with what BP does. They therefore pool features in small patches before applying B, letting B depend on relative spatial location. Finally, they pretrain VGG-style convolutional networks on CIFAR-10, STL-10, Tiny ImageNet and ImageNet with the modified rules, freeze the network, and read out a supervised linear classifier trained on concatenated representations from multiple layers. Relative to the original CLAPP they also swap non-overlapping image patches for SimCLR-style data augmentations.

Why This Matters

Research impact. The paper gives the first clear set of conditions under which a layer-wise self-supervised rule is mathematically indistinguishable from global BP-SSL, and it shows that a specific violation of those conditions (shrinking width) can be repaired by direct feedback. It also proposes a unifying notation and reports that better gradient alignment correlates with better downstream accuracy, echoing a pattern previously seen in supervised local learning.

Real-world applications (as motivated by the paper):

• On-chip learning on neuromorphic and event-based digital hardware, where a global backward pass is impractical.
• Training on cost-efficient or energy-efficient physical computing substrates, including analog or physical neural networks.
• Reducing training memory for large pretrained vision backbones, since layer activations need only be retained locally: peak VRAM was 3.38 GB versus 5.47 GB on STL-10 and 4.60 GB versus 11.10 GB on ImageNet in the reported settings.
• Layer-parallel or block-distributed training pipelines, where gradient flow is deliberately stopped at block boundaries.

Industry relevance. If local rules can match end-to-end SSL accuracy on vision benchmarks while using roughly half the activation memory reported for BP-CLAPP++ on ImageNet, they become plausible candidates for hardware-constrained or memory-constrained pretraining, and for hardware that cannot support a symmetric backward pass. Note that CLAPP++ still trails BP-InfoNCE on ImageNet (44.16 versus 55.19), so the practical case is strongest for mid-scale datasets at present.

Future Directions

• Extending the equivalence theorem beyond linear networks and beyond the orthonormality requirement, since the paper states the theorem does not carry over to nonlinear networks.
• Handling normalized layer activities, which are needed for Forward-forward algorithms but block the current proof; the paper explicitly notes this gap.
• Closing the ImageNet accuracy gap, where CLAPP++ variants reached between 42.49 and 44.16 while BP-InfoNCE reached 55.19.
• Scaling the spatial-dependence idea toward the ideal case of patch sizes k1 = k2 = 1, which Proposition

Authors’ abstract

While end-to-end self-supervised learning with backpropagation (global BP-SSL) has become central for training modern AI systems, theories of local self-supervised learning (local-SSL) have struggled to build functional representations in deep neural networks. To establish a link between global and local rules, we first develop a theory for deep linear networks: we identify conditions for local-SSL algorithms (like Forward-forward or CLAPP) to implement exactly the same weight update as a global BP-SSL. Starting from the theoretical insights, we then develop novel variants of local-SSL algorithms to approximate global BP-SSL in deep non-linear convolutional neural networks. Variants that improve the similarity between gradient updates of local-SSL with those of global BP-SSL also show better performance on image datasets (CIFAR-10, STL-10, and Tiny ImageNet). The best local-SSL rule with the CLAPP loss function matches the performance of a comparable global BP-SSL with InfoNCE or CPC-like loss functions, and improves upon state-of-the-art for local SSL on these benchmarks.