Topological Signatures of ReLU Neural Network Activation Patterns

AI summary

Overview

Research area: Topological data analysis (TDA) applied to deep learning theory — specifically the polyhedral geometry of feedforward ReLU neural networks.

Technical level: Advanced. The paper assumes familiarity with graph Laplacians, spectral graph theory, cell complexes, Betti numbers, filtrations, and persistent homology.

Scope: The paper investigates whether topological features of the polytope decomposition induced by a ReLU network's activation patterns correlate with network performance during binary classification (via the Fiedler partition of the dual graph) and during regression (via homology and cell counts of a random filtration).

Authors and affiliations: Vicente Bosca (University of Pennsylvania), Tatum Rask (Colorado State University), Sunia Tanweer (Michigan State University), Andrew R. Tawfeek (University of Washington), Branden Stone (Georgia Tech Research Institute). The first four authors are listed as having equal contribution. The work was supported in part by the Georgia Tech Research Institute (GTRI) through their summer research internship program (GRIP). arXiv:2510.12700v2 [cs.LG].

What This Paper Is About

A ReLU network partitions its input space into polytopes, where every point in a given polytope produces the same pattern of "on" and "off" neurons (the same binary state vector). This paper asks whether the shape and connectivity of that partition carries information about how well the network is learning.

The authors pursue two lines of inquiry: for binary classification, whether a spectral partition (the weighted Fiedler partition) of the graph of neighboring polytopes aligns with the decision boundary; and for regression, whether the homology and cell counts of the polytope complex track the training loss across epochs.

Key Contributions

1. A vertex-weighted Laplacian for the dual graph. The authors propose weighting each node of the dual graph by one plus the number of training data points contained in the corresponding polytope, leaving edges unweighted (identity), and show this weight choice makes the Fiedler partition match class labels in two small binary classification experiments where the unweighted version fails.

2. Empirical link between topological complexity and training loss. Using a random filtration of the polyhedral cell complex, the authors compute averaged Betti curves per epoch and report moderate-to-strong correlations between topological complexity and training loss, robust to detrending and confirmed by first-difference analysis.

3. A computational shortcut via the f-vector. The paper shows that the peaks of the Betti curves recover the f-vector of the decomposition: max(β₀) equals f₀ (number of vertices) and max(β₁) equals f₂ (number of 2-cells) once all edges but no faces have been added, offering an efficient alternative to full homology computation, which becomes intractable for networks with thousands of cells per dimension.

4. Evidence that loss spikes are topological events. Moments of training instability correspond to a transient increase in the filtration value at which the maximum Betti number is achieved, and the authors argue these are not merely numerical artifacts but signs of a deeper reorganization of the network's internal topological representation.

Main Findings

• Unweighted Fiedler partition is inaccurate. In the Two Circles experiment, the unweighted Fiedler vector misclassified 19.05% of polytopes with an L₂ error of 2. In the Two Moons experiment, it misclassified 10% of polytopes with an L₂ error of 1.20.

• Weighted Fiedler partition matches the decision boundary. In the same experiments, the training-point-weighted partition misclassified 0% of polytopes, with L₂ error 0 (Circles) and 0.34 (Moons).

• Both classification models reached perfect accuracy. The Two Circles model used architecture (2, 6, 6, 2), trained to 4000 epochs, with training loss 0.00002 and test loss 0.00016, reaching 100% accuracy on both training and test data. The Two Moons model used architecture (2, 5, 5, 5, 2), trained to 2000 epochs, with training loss 0.00002 and test loss 0.00001, also reaching 100% accuracy on both.

• Grokking has a geometric signature. Grokking (delayed generalization) manifests as an increase in polytope regions around training points once training error becomes negligible, alongside a greater concentration of smaller polytopes around decision boundaries. Those smaller boundary polytopes receive smaller weights in the dual graph, which the authors argue is why the weighted Fiedler vector "cuts" near the decision boundary.

• Cell counts decrease over training. The downward shift of peak Betti values in early epochs — from approximately 25% to 11% for β₀ and from 75% to 35% for β₁ — indicates substantial evolution in the number of cells across dimensions. The overarching trend across all 25 trials is a reduction in the total number of cells.

• Loss spikes coincide with cell-count fluctuations. Immediately prior to the final large spike in loss, the number of cells tends to increase, followed by a decrease. This reorganizing behavior after a big loss spike was observed consistently across trials.

• Correlations are strong across chaotic and periodic dynamics. For the chaotic system (n = 25 trials), raw Pearson correlation between loss and 0-cells was 0.69 ± 0.13 (p < 10⁻⁶, 25/25 significant), detrended 0.82 ± 0.11 (25/25 significant), with maximum cross-correlation at lag = 25 of 0.70 ± 0.13. For the periodic system, raw Pearson was 0.61 ± 0.09 (25/25 significant), detrended 0.61 ± 0.10 (25/25 significant), with maximum cross-correlation at lag = 100 of 0.74 ± 0.14.

• First differences show the relationship is dynamic. First-difference correlations were 0.88 ± 0.14 (25/25 significant) for the chaotic system and 0.58 ± 0.31 (23/25 significant) for the periodic system. Spearman rank correlations were weaker: 0.29 ± 0.20 (20/25 significant) for chaotic and 0.13 ± 0.22 (15/25 significant) for periodic.

• Topology appears to lead the loss. Maximum cross-correlation occurred at a lag of +25 epochs for the chaotic system and +100 epochs for the periodic system, suggesting geometric reorganization of the polyhedral decomposition precedes shifts in the loss landscape.

• Euler characteristic constrains the f-vector. For the 2D square domain, χ = f₀ − f₁ + f₂ = 1, which together with fᵢ ≥ 0 implies f₀ ∼ f₂ ∼ ½f₁, verified experimentally.

• Theoretical justification is still missing. The authors state they have not proven that the weighted Laplacian approximates a weighted minimum cut, and list this as future work.

Methodology in Plain English

Setup. The authors work with (L+1)-layer feedforward ReLU networks. For any input point, each hidden neuron is either "on" (pre-activation greater than zero) or "off" (less than or equal to zero), producing a binary state vector. Stacking all hidden layers gives one long binary vector per input point. All points sharing a binary vector form a single polytope, so the input space is carved into polytopes. The experiments use the input space ℝ².

Classification branch. They build a dual graph whose vertices are the binary vectors and whose edges connect vectors differing in exactly one entry (Hamming distance 1). This graph is the adjacency structure of polytopes. They compute the graph Laplacian, and specifically its Fiedler vector — the eigenvector for the smallest non-zero eigenvalue λ₁ (the algebraic connectivity). Because the Fiedler vector's entries sum to zero, its signs give a natural binary partition that approximates a minimum cut. They then compare an unweighted Laplacian against a weighted version where each vertex weight equals one plus the number of training data points in that polytope (obtained with a function in the GoL_Toolbox). Accuracy is scored two ways: the proportion of polytopes assigned the wrong class, and the L₂ norm of the vector of differences between average class labels within polytopes and predicted values.

Regression branch. They treat the polyhedral decomposition as a cell complex: d-dimensional cells are the polytopes, (d−1)-dimensional cells are facets, lower-dimensional cells are hyperplane intersections, and 0-dimensional cells are vertices where hyperplanes meet. Rather than computing Betti numbers of the whole complex directly — which would just recover the Betti numbers of the input space (β₀ = 1, β₁ = 0 for a 2D square) — they build a random filtration: add 0-cells uniformly at random, then 1-cells, then 2-cells, and so on, recording Betti numbers at each step. They run multiple random trials and average the Betti curves, expressing the filtration parameter as the percentage of total cells added so different epochs are comparable. Sign vectors of the decomposition come from edge subdivision; the cell complex is constructed via a perturbation method from the same work; and homology is computed with ℤ₂ coefficients using the PHAT package.

Modeling task. The regression experiments use a Physics-Informed Neural Network for the Duffing oscillator. The network takes a concatenated vector [t, x(t)], passes it through a fully connected ReLU layer with 50 neurons, then three consecutive linear transformations each followed by ReLU, then a linear output layer producing a scalar prediction of displacement at the next timestep. The Duffing equation parameters were δ = 0.0, α = −1.0, β = 1.0, γ = 0.0, and ω = 1.2 (a periodic regime), with initial conditions x(0) = 0 and v(0) = 1 over t ∈ [0, 20]. The loss is the sum of the mean squared error of data predictions and the mean squared error of the governing differential equation residual.

Visualization. Results are shown as Betti curves and as heat maps where the x-axis is training epoch, the y-axis is cells added divided by the maximum cell count across all epochs, and color encodes the average Betti curve value. Experiments ran on an Intel Xeon Silver 4214R CPU at 2.40 GHz base clock, with 24 CPUs, 700 KB L1 cache, 12 MB L2 cache, and 16.5 MB shared L3 cache.

Why This Matters

This work adds to a growing body of evidence that the geometry of a ReLU network's input-space partition is not incidental but tracks learning dynamics. The distinctive claim is that the relevant signals are topological rather than purely algebraic — the Fiedler partition captures classification structure, and the homology of the cell complex tracks the loss. The authors also suggest the homology could serve as a proxy for loss without requiring access to training data, though they explicitly leave this to future work.

Potential application areas suggested by the work:

• Physics-informed modeling and scientific simulation. The pipeline was demonstrated on a PINN for the Duffing oscillator, a canonical nonlinear dynamical system, indicating relevance to any setting where neural surrogates model dynamical systems.

• Training diagnostics and monitoring. If topological complexity is a leading indicator of loss changes (as the +25 and +100 epoch lags suggest), it could serve as an early-warning signal for instability during training.

• Robustness and decision-boundary analysis. Characterizing where small, boundary-concentrated polytopes appear could inform how models behave near class boundaries.

• Model architecture design. The authors suggest these findings may inform more effective neural network architectures and training methods, since geometric and topological structure is emphasized over purely algebraic properties.

Industry relevance. The scalability analysis matters for practitioners: the number of polytopes grows polynomially with network width, exponentially with depth, and exponentially with input dimension, and computing graph Laplacian eigenpairs is itself computationally expensive under current algorithms. As presented, the methods are practical only for small networks and low-dimensional input spaces. That limitation bounds near-term industrial adoption but defines a clear target for approximation and sampling approaches.

Future Directions

• Prove the theoretical claim. Establish that the proposed vertex-weighted Laplacian approximates a weighted minimum cut, either directly or by leveraging existing results for edge-weighted and vertex-weighted Laplacians.

• Address scalability. Project activation patterns onto principal components to reduce effective dimensionality, apply approximation algorithms for homology computation or sampling-based approaches, and use faster algorithms such as TRACEMIN-Fiedler for computing the Fiedler vector — all while accepting some loss of precision in topological measurements.

• Replace random filtrations with geometry-aware ones. Use geometrically-informed or "boundary-aware" filtrations to track polytope birth and death near decision boundaries across epochs, potentially building on work showing that the dual complex of ReLU networks forms a cubical complex and enables the use of GUDHI.

• Extend to multiclass classification and test broader invariance. Develop analogues of the Fiedler partition analysis for multiclass problems, study the topological invariance of the Euler characteristic, and explore the suggested link between grokking and circuit formation, where larger polytopes would correspond to "cleaner", less complex circuits.

Target Audience

Researchers in topological data analysis and deep learning theory will get the most from this paper, particularly those already comfortable with persistent homology, cell complexes, and spectral graph theory. It is also useful for practitioners working with physics-informed neural networks who want a diagnostic lens on training dynamics, and for graduate students looking for a worked example of applying algebraic topology machinery to neural network interpretability. Readers seeking immediately deployable tools will find the scalability limitations and the unproven theoretical claim the main obstacles.

Authors’ abstract

This paper explores the topological signatures of ReLU neural network activation patterns. We consider feedforward neural networks with ReLU activation functions and analyze the polytope decomposition of the feature space induced by the network. Mainly, we investigate how the Fiedler partition of the dual graph and show that it appears to correlate with the decision boundary -- in the case of binary classification. Additionally, we compute the homology of the cellular decomposition -- in a regression task -- to draw similar patterns in behavior between the training loss and polyhedral cell-count, as the model is trained.