Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks

AI summary

Overview

Research area: Deep learning theory and optimization — specifically the geometry of neural network loss landscapes, mode connectivity, and the stochastic dynamics of SGD.

Technical level: Advanced. The paper combines statistical physics (entropic forces, effective temperature, Boltzmann distributions) with spectral analysis of the Hessian of real networks, and assumes familiarity with concepts such as minimum energy paths, the Fisher Information Matrix, and linear mode connectivity.

Scope: The paper argues that the apparent contradiction between "minima are connected by low-loss paths" and "SGD never explores those paths" is explained by entropy, not energy — curvature rises away from minima along connecting paths, and the resulting entropic forces confine optimization to the endpoints.

What This Paper Is About

Neural network loss landscapes in the overparameterized regime contain distinct minima that are joined by paths of nearly constant, low loss — yet standard SGD training converges to one minimum and essentially never wanders onto those connecting paths. The paper asks why energetic connectivity does not translate into dynamical connectivity, and proposes that variations in curvature along the paths, combined with SGD noise, generate entropic forces that push optimizers back toward the flatter endpoints.

Key Contributions

1. Empirically showing that curvature along minimum-loss paths between minima increases away from the endpoints — measured both as the trace of the Hessian and as its maximum eigenvalue.
2. Arguing that this curvature "bump" constitutes an entropic barrier, so that regions which are energetically connected are effectively disconnected for noisy optimization dynamics.
3. Demonstrating that entropic forces confine models to specific regions of parameter space despite low-loss connecting paths, including cases where entropy overrides energy and drives the model up the loss gradient.
4. Showing that in networks sharing the first $k$ epochs of training, entropic barriers persist longer than energetic barriers, implying that both energetic and entropic forces shape the final region of parameter space a model occupies.

Main Findings

• Curvature rises along connecting paths: For Wide ResNet-16-4 on CIFAR-10, both the Hessian trace and $\lambda_{\max}(\mathcal{H})$ increase as one moves from the endpoints into the interior of minimum energy paths (MEPs), while the cross-entropy loss along the MEP is often lower than at the endpoints.
• The whole spectrum shifts: An SVD of the score matrix computed on $E = 1024$ training examples shows that the entire Hessian spectrum shifts upward in the MEP interior, not just the leading eigenvalue.
• The effect is not a loss artifact: Although loss drops between the first and second pivots and then stays roughly constant, both sharpness metrics continue to rise — so the increase in curvature is a genuine change in landscape geometry rather than a byproduct of lower loss or implicit regularization.
• Entropic forces push models back to endpoints: With projected SGD constrained to the MEP (batch size $B = 16$, learning rate $\eta = 0.02$), models initialized at various points drift back toward the nearest, flatter endpoint; models starting deeper along the path take longer to relax.
• Entropy can beat energy: Models are driven back toward the first pivot even though the loss increases along that direction — a case where the entropic force dominates the energetic one, consistent with the system minimizing free energy rather than energy.
• Force scales with effective temperature: Relaxation is faster for smaller minibatches (Figure 3B) and for larger learning rates (Figure 3C). The appendix shows the entropic force scaling with batch size directly, and the characteristic relaxation time (time for the relative distance along the MEP to fall by a factor of $e$) is described as a growing exponential.
• Optimizer dependence: Projected Adam and SGD with Nesterov momentum respond more strongly to curvature variations than vanilla SGD, suggesting entropic effects matter more with adaptive optimizers or momentum.
• Entropic barriers persist late in training: Reproducing the linear mode connectivity setup with ResNet-20, the average loss along linear paths between sibling networks drops rapidly as the splitting epoch $k$ grows, but curvature barriers persist for larger $k$. Using an instability measure (max/min relative change along the path), small $k$ shows larger loss instability while larger $k$ shows larger curvature instability.
• Trends generalize across data and architecture: A ResNet-110 trained on CIFAR-100 shows similar loss and curvature behavior, and similar late-training dominance of entropic barriers; the appendix reports the same for ResNet-20 on CIFAR-100.
• Entropic barriers are statistical, not absolute: The authors emphasize the model is not dynamically forbidden from crossing them — crossing is simply statistically extremely unlikely, so the model is "effectively forbidden."

Methodology in Plain English

The authors train many image classifiers on CIFAR-10 from different random seeds, producing a diverse set of distinct minima. For each pair of minima they construct a low-loss connecting path using the AutoNEB algorithm, which places intermediate "pivots," moves them as if joined by elastic springs while minimizing loss perpendicular to the path, and inserts new pivots wherever the straight segment between existing pivots has too much loss.

They then characterize the path in two ways. First, they measure curvature at points along it using three tractable proxies for the Hessian spectrum: the maximum eigenvalue estimated by power iteration (requiring only Hessian-vector products), the Hessian trace estimated through the Fisher Information Matrix (which coincides with the Hessian at a loss minimum), and the singular values of a score matrix of shape $N \times (C \cdot E)$ computed on a small subset of $E$ training examples.

Second, they run optimization constrained to the path using a variant called $k$-step projected SGD: the model takes $k$ ordinary SGD steps on minibatches, then is projected back onto the closest segment of the path. Taking multiple steps between projections preserves the noise-driven multi-step dynamics that generate entropic forces, while keeping the model on the path so its motion can be measured. Experiments are compared against an effective time $t_{\mathrm{eff}} = (\text{optimizer updates}) \times \eta$ so runs at different learning rates are comparable.

For the linear mode connectivity analysis, they follow prior work: train $M$ networks on a shared data order up to a splitting epoch $k$, then let each continue with its own independent data ordering until convergence, and measure loss and curvature along the straight line between the converged siblings.

Training defaults: Wide ResNet-16-4 on CIFAR-10, SGD with momentum $\beta = 0.9$, weight decay $w = 5 \times 10^{-4}$, initial learning rate $\eta = 0.1$, 200 epochs, batch size 256, learning rate divided by 5 at 30%, 60%, 80% and 90% of training, with random horizontal flips and random crops with 4-pixel padding resized to $32 \times 32$.

Why This Matters

The paper reframes a central puzzle in deep learning theory. The common picture of a single broad "valley" of low-loss solutions connected by flat paths is replaced by a picture in which that valley is carved into effectively disconnected regions by curvature-induced entropic barriers. This gives a mechanistic explanation for why SGD localizes, why the basin a network lands in is so sensitive to early perturbations, and why models trained with SGD keep generalizing rather than diffusing into overfitting regions that are energetically reachable.

Areas the findings bear on:

• Model merging and weight-space ensembling: Techniques that average weights across minima, including Stochastic Weight Averaging, may be combining solutions that are energetically connected but effectively separated by entropic barriers — implying the averaged solution may not be reachable by diffusive optimization at the bottom of the valley.
• Optimizer and hyperparameter design: Since entropic force strength grows with the effective temperature $T \propto \eta/B$, batch size and learning rate are not just convergence knobs but controls on which region of parameter space a model settles into.
• Generalization diagnostics: The authors propose that generalizing minima may be shielded from overfit minima by entropic barriers, offering a possible answer to why SGD does not overfit despite a flat energetic landscape.
• Understanding late-stage training: The two-phase picture — energetic forces early, entropic forces late — suggests where to intervene to control basin selection.

Industry relevance: The results are directly relevant to anyone applying weight averaging, checkpoint merging, or model soups, and to practitioners tuning batch size and learning rate, since those choices set the effective temperature that governs entropic confinement. The paper notes the codebase will be released once the author list is unblinded.

Future Directions

• Principled path sampling: The authors acknowledge that AutoNEB and linear interpolation bias which paths get studied, and identify sampling the space of low-loss paths more rigorously as an important and promising direction.
• Connectivity of averaged minima: Analyzing the connectivity properties of weight-averaged minima, to understand how ensembling constructs solutions with favorable generalization.
• Generalization and entropic shielding: Testing the hypothesis that generalizing minima are effectively disconnected from overfitting minima by entropic barriers.
• Generalization across settings: The analysis is repeated for ResNet-20 and ResNet-110 on CIFAR-10 and CIFAR-100; extending it to further architectures, datasets, and optimizers, and refining the relationship between the effective temperature and quantities like learning rate and batch size, remain open.

Target Audience

Machine learning researchers working on optimization, loss landscape geometry, and mode connectivity; physicists moving into deep learning theory who are comfortable with entropic forces and effective temperatures; and practitioners of weight-space ensembling or model merging who want a mechanistic account of why averaged solutions behave the way they do. Readers without prior exposure to Hessian spectral methods, the Fisher Information Matrix, or stochastic thermodynamics will need to consult the cited background literature.

Authors’ abstract

Modern neural networks exhibit a striking property: basins of attraction in the loss landscape are often connected by low-loss paths, yet optimization dynamics generally remain confined to a single convex basin and rarely explore intermediate points. We resolve this paradox by identifying entropic barriers arising from the interplay between curvature variations along these paths and noise in optimization dynamics. Empirically, we find that curvature systematically rises away from minima, producing effective forces that bias noisy dynamics back toward the endpoints - even when the loss remains nearly flat. These barriers persist longer than energetic barriers, shaping the late-time localization of solutions in parameter space. Our results highlight the role of curvature-induced entropic forces in governing both connectivity and confinement in deep learning landscapes.