Fisher information and the geometry of memorization in neural networks

AI summary

Overview

Research area: Machine learning — loss-landscape geometry, the Fisher information, and the separation of memorization from generalization in trained networks.

Technical level: Intermediate. The experimental sections are accessible to anyone who knows classification and eigendecompositions; the closed-form "barcode" model in Section 5 requires comfort with linear regression theory and random matrix results.

Scope: The paper tests the claim that the eigenspectrum of the Fisher information separates memorized content from generalizing features, across Gaussian-image noise, label noise, and a synthetic in-context learning task, plus an analytically solvable linear model.

What This Paper Is About

Neural networks can drive training error to zero by memorizing individual samples, yet they also learn features that generalize. The paper asks whether the local geometry of the loss — specifically the Fisher information matrix — can tell these two kinds of stored information apart. The authors define and test a "de-memorization" operation: projecting a layer's weights onto the span of the top k Fisher eigenvectors, and checking whether performance on memorized samples collapses while performance on structured data survives.

Key Contributions

1. Low-eigenvalue Fisher subspaces encode memorized data across settings. The authors show this for injected Gaussian input noise, for randomly flipped labels, and for memorized latent tasks in an in-context linear regression problem, and they map how the effect differs by layer.

2. The separation is data-dependent. Computing the Fisher on a denoised dataset yields cleaner de-memorization than computing it on the full training distribution; on the full set, the top eigenspaces mix generalizing and memorized information, and the whole-network (block-diagonal) picture breaks down.

3. De-memorization can be lifted to latent tasks and can improve generalization. In the in-context regression setting, projecting the MLP of the second transformer block recovers roughly half the gap to the best early-stopped test performance, at the cost of accuracy on memorized training tasks; the same projection applied to that block's attention layer only hurts performance.

4. A closed-form linear "barcode" model reproduces the effect. The model treats uninformative features as a block that correlates with label noise only in finite samples, giving exact expressions for the risk reduction from deleting that block and predicting when an interior-optimal projection rank exists.

Main Findings

• First layer separates, last layer does not (Gaussian images). With the Fisher computed over the full training set, projecting the first layer of a ResNet-20 onto the top k eigenvectors raises CIFAR-10 train and test accuracy in tandem, peaking at k ≈ 45, while accuracy on the 500 memorized Gaussian images stays at approximately 30%. In the final layer, all three curves rise together with no separation at any k, indicating the representations have merged.

• Almost all final-layer predictive information is in a few directions. Nearly all relevant predictive information in the final layer's weights lies within the top approximately 5% of its eigendirections.

• True-label recovery under label noise. With 10% of CIFAR-10 labels flipped uniformly among the nine incorrect classes, the top Fisher subspace raises accuracy on the poisoned samples' true labels well above chance in the first layer. In the penultimate convolutional layer, de-memorization is successful when the Fisher is computed on clean data only — the red (memorized-label) curve has not reached its maximum when train and test accuracy do — but the decoupling weakens when the Fisher is computed on the full training set.

• Whole-network block-diagonal Fisher is less informative. With a block-diagonal Fisher computed on clean-label data, top eigenspaces selectively capture the true classification task, and true-label accuracy on memorized samples (orange dashed line) behaves non-monotonically. Computed on a sample of the full training set, the top eigenspaces mix the two tasks, and separability is only visible at the level of individual layers.

• Participation ratios distinguish the two noise types. In the input-noise case, total participation ratios drop by roughly one order of magnitude from top to bottom modes, with clean CIFAR-10 images dominating the top modes and Gaussian samples relegated to the bottom. In the label-noise case, corrupted and uncorrupted samples contribute in a constant ratio across the entire spectrum.

• Single-sample projections are mildly targeted. Projecting out top eigenvectors of a single sample's Fisher degrades that sample more than others in the input-noise case (Gaussian-set accuracy after Gaussian-Fisher projection falls from 100.0 at k = 0 to 75.6 at k = 9, versus 84.3 to 83.1 for CIFAR-10 test), but the broader effect is small; the notable exception is significant degradation of Gaussian-set performance after projecting out a Gaussian sample.

• Memorization localizes to one MLP in the transformer. Of all layers examined in the three-block transformer, only the Block 2 MLP improves held-out task performance when restricted to a top Fisher subspace; every other layer performs worse on both train and test tasks under the same projection.

• Theory matches simulation in the barcode model. The predicted risk reduction ΔR = Tr(Σ_ξ)·d/(n − r − d − 1), valid for n − r > d + 1, agrees with simulations at (n, r, d) = (500, 4, 250) with 200 replicates. An informative feature with population variance below the spectral separation threshold 1 + √(d/(n−r)) merges into the barcode bulk, so at higher label-flip rates an interior-optimal retained rank emerges, whereas at low flip rates full rank is best.

Methodology in Plain English

The authors deliberately force models to memorize. In the first CIFAR-10 variant they add 500 synthetic images — 1% of the training set — each drawn from a Gaussian whose mean and variance match a randomly chosen CIFAR-10 class but labeled with an independently chosen random class, so no generalizable feature can solve them. In the second variant they reassign 10% of training labels uniformly among the nine incorrect classes. In both, a ResNet-20 is trained to 100% training accuracy.

They then compute the exact per-layer Fisher information matrix (rather than the K-FAC approximation used in prior work) from the network's predictive probabilities, diagonalize it, and evaluate the model after projecting each layer's weights onto the span of the top k eigenvectors, sweeping k. Because BatchNorm statistics shift under projection, the authors re-estimate them after each projection and include a matched unchanged-weight control at k = 0.

To see which samples feed which eigenmodes, they compute per-sample Fisher contributions c_i^k = u_k^T F_i u_k and participation ratios Ψ^k, plus "restricted" versions counting only CIFAR, only Gaussian, only corrupted, or only uncorrupted samples. They also run the reverse experiment: build a single sample's Fisher, project out its top eigenvectors, and measure which samples suffer.

The transformer experiment uses a three-block GPT-2-style model trained on in-context linear regression: contexts of 20 (x, y) pairs with y_i = w_τ^T x_i and x in R^5, with training contexts drawn from a small set of M task vectors so the model overfits to those tasks. The final piece is a hand-solvable linear model in which informative features and an uninformative Gaussian "barcode" block are fitted jointly to corrupted labels, letting the authors derive exactly how much risk is removed by discarding the barcode block afterwards.

Why This Matters

The work gives a concrete geometric account of where memorized content lives in a trained network and why it can sometimes be removed without retraining — and equally importantly, documents where that account fails, namely in the last layer and when the Fisher is computed on noisy data.

Impact on research: It extends and stress-tests a specific prior claim (that top Fisher modes encode generalizing structure and low-curvature modes encode memorization) by removing the K-FAC approximation, adding layer resolution, and varying the dataset used to compute the Fisher. It also connects to benign overfitting and data coarse-graining, and supplies a closed-form model for a phenomenon previously described only empirically.

Real-world applications:

• Mitigating reproduction of pretraining data in language models, which raises copyright and privacy concerns.
• Defending against data poisoning, where adversaries exploit a model's tendency to memorize injected samples.
• Recovering performance lost to mislabeled training data without relabeling or retraining from scratch.
• Building "intelligent yet ignorant" models — systems that retain task competence while not storing particular information.

Industry relevance: The method is a post-hoc weight edit requiring only a Fisher computation and a projection, so it is cheap relative to retraining. The paper's caveats matter commercially: the effect is layer-specific, depends on having a clean reference dataset for the Fisher, does not support fine-grained deletion, and the authors explicitly warn it is not a scale-independent editing prescription.

Future Directions

• Test whether projection is a small move in function space. The authors note the method is geometrically odd: identifying subspaces with a local quantity but then applying a non-local projection that can perturb parameters substantially. They raise the possibility that the projection lands near the same model within a "hyperribbon" of soft directions, and leave this to future work.

• Scale to larger models. The empirical work uses ResNet-20 on CIFAR-10 variants and a three-block transformer with hidden dimension 32; the authors state that the layerwise separation may change qualitatively in larger models.

• Extend beyond simple object classification and synthetic regression. The authors flag that semantic, compositional, or cross-example distributed memorization — and other data modalities — may behave differently.

• Incorporate cross-layer correlations. The analysis ignores them, using a block-diagonal approximation for the whole-network Fisher, and the authors note they may contribute meaningfully to memorization.

Target Audience

Machine learning researchers working on interpretability, loss-landscape geometry, memorization and privacy, noisy-label learning, and unlearning; theoretically inclined readers interested in solvable models of overfitting; and practitioners considering Fisher-based model editing who need to know its layer- and data-dependence before applying it.

Authors’ abstract

A fruitful approach towards understanding generalization in machine learning involves characterizing how models separate generalizable structure from idiosyncratic or even corrupted features of training data. Fisher information has emerged as a diagnostic tool for this separation, leveraging loss geometry to distinguish generalization from memorization. Here we investigate this connection in vision models trained on CIFAR-10 with controlled noise that can be learned only through memorization. We show that projecting model weights onto the top Fisher eigenvectors decouples general task performance from noisy sample memorization, and we investigate how this decoupling presents differentially across layers. We show that the same phenomena are present in small transformers trained to solve a synthetic in-context learning task, and are reproduced in closed form in a simple, analytically solvable linear model. Our results reveal that Fisher information captures aspects of the geometric structure underlying how neural networks allocate capacity between generalization and memorization.