Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities

AI summary

Overview

Research area: Deep learning theory — specifically the geometry and topology of the parameter space of feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures, combining algebraic geometry, network flow theory, and optimization dynamics.

Technical level: Advanced. The paper is written for readers comfortable with differential geometry, algebraic varieties, quadric hypersurfaces, incidence matrices, and network flow theory. The intuition behind each result is explained, but the proofs rely on specialized mathematics.

Scope: The paper characterizes when the gradient-flow-invariant parameter space of a DAG ReLU network is disconnected, when it contains singularities, and how those singularities can be deliberately induced for pruning. (arXiv:2602.00693v1 [cs.LG], published 31 Jan 2026.)

What This Paper Is About

When a ReLU network is trained, the rescaling symmetry of the ReLU activation forces the parameters onto a restricted algebraic surface — the "invariant set" — defined by a system of quadratic equations determined by the network's topology and its initialization. The paper asks two questions about that surface: is it connected (can gradient flow move freely between all its regions?), and does it contain singularities (points where the surface pinches or degenerates)? Prior work answered these questions only for shallow networks; this paper extends the analysis to arbitrary DAG architectures and connects the answers to practical pruning.

Key Contributions

1. A compact, general formulation of the conservation laws. The authors derive the conservation laws that arise during gradient flow training of ReLU networks from rescaling symmetries, expressed in the general case of DAG-based architectures using a reduced incidence matrix. The local neuron-wise condition ⟨θ, g(θ)⟩_v = 0 for every hidden neuron v is shown to be equivalent to B̃(θ ⊙ g(θ)) = 0, where B̃ is the incidence matrix with input and output rows removed.

2. A full characterization of connectedness of the invariant set in general architectures. Extending results previously known for shallow networks (Nurisso et al., 2024), the authors give necessary and sufficient conditions for connectedness in terms of bottleneck neurons (hidden neurons with exactly one incoming or exactly one outgoing edge) and balance ("supply/demand") constraints on specific subsets of nodes.

3. Identification and analysis of singularities. The authors show that singularities of the invariant set correspond to disconnected sub-networks — groups of neurons cut off from both inputs and outputs — and prove that such singularities are generically absent at initialization and unreachable in finite time under gradient flow.

4. A regularizer for inducing singularities, enabling differentiable, structure-agnostic pruning. Because singularities correspond to reduced Jacobian rank, the authors propose penalizing the nuclear norm of the Jacobian as a smooth surrogate for rank minimization. They report that L1 regularization empirically induces similar singular behavior even though it does not explicitly target neuron sparsity, and that reaching singular configurations enables lossless pruning.

Main Findings

• Conservation laws are graph-flow conditions. The invariant set is H_G(c) = {θ : B̃θ² = c}, where c is determined by the initialization via c = B̃θ(0)². Written per hidden neuron v, each equation reads Σ_{i→v} θ²_{(i,v)} − Σ_{j←v} θ²_{(v,j)} = c_v, a hyperbolic quadric hypersurface in the local parameter space Θ_v. The authors interpret θ² as a fluid flowing through the edges of G, with input/output nodes acting as unconstrained sources/sinks and hidden nodes supplying or demanding flow according to c.

• Every balance configuration is feasible. Proposition 3 states that for all c ∈ ℝ^{|Ṽ|}, H_G(c) ≠ ∅ — any supply/demand pattern on the hidden neurons can be realized by some parameter initialization, provided no neurons are excluded from the computation.

• Connectedness is governed by bottlenecks and forward/backward-stable sets. Theorem 1 states that H_G(c) is connected if and only if: for every out-bottleneck v and every subset T of its pure ancestors that is stable by forward edges, Σ_{u∈T} c_u ≥ 0; and for every in-bottleneck v and every subset T of its pure descendants stable by backward edges, Σ_{u∈T} c_u ≤ 0. Intuitively, disconnection occurs when cutting a bottleneck's single edge makes the balances of its pure ancestors/descendants unfeasible.

• Bottleneck-free networks are always connected. Corollary 1 states that if G has no bottleneck neurons, H_G(c) is connected. The authors note that bottleneck neurons are rare in MLPs because they imply a layer with only one node — the common exception being binary classification or scalar regression, where the single output neuron makes the last layer's neurons out-bottlenecks.

• The shallow-network result carries over to multilayer MLPs. Corollary 2: for a fully connected MLP in which all hidden layers contain more than one neuron, H_G(c) is connected if and only if c_v ≥ 0 for all out-bottlenecks and c_v ≤ 0 for all in-bottlenecks.

• Disconnection can destroy expressivity at initialization. Some functions become immediately unreachable regardless of the chosen loss function or dataset, reducing the network's universal approximation capability before training even begins.

• Singularities are sub-networks. The Jacobian of the invariant set is J_G(θ) = 2 B̃ diag(θ), so rank reduction can only occur when some edge weights θ are zero. Theorem 2 gives rank J_G(θ) = |Ṽ| + 1 − CC(G'_{−E(θ)}), where G' is the undirected graph obtained by gluing input and output nodes and neglecting edge direction, E(θ) is the set of zero-weight edges, and CC counts connected components. For regular parameters the rank equals |Ṽ|.

• Singularities, once reached, are preserved. Proposition 5 shows that if a set of nodes W is disconnected at time t₀, it remains disconnected at all later times t > t₀ under gradient flow: once a network module is killed, it cannot be revived.

• Singularities are rare and unreachable. Proposition 6 shows that if H_G(c) admits singularities, then some subset W ⊆ Ṽ of hidden neurons must satisfy Σ_{v∈W} c_v = 0 — an exact equality condition that has probability zero under any initialization scheme where each parameter is sampled independently from ℝ. Proposition 7 strengthens this: rank J(θ(0)) = rank J(θ(t)) for all 0 ≤ t < ∞, so gradient flow cannot fall into a singularity in finite time. Together, the authors conclude that singularities generally do not exist in the optimization space under common initializations and are effectively unreachable even with the special choice c = 0.

• Nuclear norm regularization drives the model toward singularities. On the Breast Cancer dataset (Elson et al., 2007), training MLPs with SGD at a step size of 0.001, the nuclear norm of the Jacobian consistently pushes the model toward singular configurations. For example, a shallow network disconnected around 18 of its 20 hidden units, whereas L2 and unregularized training left all neurons active.

• L1 behaves surprisingly similarly to the nuclear norm. Although L1 is traditionally associated with parameter-level rather than neuron-level sparsity, it empirically induces singular behavior comparable to the dedicated regularizer and therefore also fosters effective lossless pruning. The authors state that a precise theoretical understanding of this is outside the scope of the paper and provide an empirical analysis instead.

• Reaching singular configurations enables lossless pruning. While L2 is reported as already quite robust to pruning, disconnecting active neurons introduces modifications to the implemented function. Shaded regions in the pruning and singular-value experiments denote confidence intervals over 50 independent trainings that converged to a low-loss solution.

• Numerical illustration of the connectedness theorem. On a toy DAG with three hidden nodes, neuron 4 is the out-bottleneck of interest, with c = (c₂, c₃, c₄) = (θ_e² − 6, 1, 3). The largest forward-stable set is T = {2, 3, 4} with Σ_{k∈T} c_k = θ_e² − 4, so if θ_e(0) < √2 at initialization the optimization space disconnects at neuron 4, forbidding a sign switch of θ_{(4,5)}. The target function f: (x₁, x₂) → −(x₁ + x₂) for positive inputs requires θ_{(4,5)} < 0, so the optimum is unreachable for θ_e(0) = 1 < √2, while it is reachable and reached for an initialization with θ_e(0) greater than √2 as depicted in the paper's figure.

Methodology in Plain English

The authors model a feed-forward network as a directed acyclic graph whose nodes are neurons and whose edges carry parameters. They single out input and output neurons, call everything else hidden, and encode the graph's wiring in an incidence matrix B, which is then stripped of its input and output rows to give B̃.

Because ReLU is positively homogeneous, rescaling a hidden neuron's incoming weights by α > 0 and its outgoing weights by 1/α leaves the network's function unchanged. Since the loss depends on the parameters only through the network output, the gradient must be orthogonal to this rescaling orbit. Writing that orthogonality condition at every hidden neuron and using the incidence matrix turns the neuron-wise conservation laws into a single matrix equation, B̃(θ ⊙ g(θ)) = 0, which integrates to B̃θ(t)² = c with c fixed by the initialization.

That gives the central object of study: the invariant set H_G(c), the solution set of a system of degree-two polynomial equations. Each equation involves only squared parameters, with no mixed terms, so the system consists of coaxial quadric hypersurfaces — a structure that lets the authors import existing topological results (notably de Medrano, 2023) about when such quadrics and their intersections are connected or empty.

For connectedness, they use the criterion that the solution set is connected exactly when deleting any single edge still leaves a nonempty solution set, which forces any disconnection to originate at neurons with a single incoming or outgoing edge. They then translate feasibility of the remaining constraint system into network flow conditions on subsets of nodes, yielding Theorem 1.

For singularities, they compute the Jacobian of the constraint system, J_G(θ) = 2 B̃ diag(θ), observe that the only way to lose rank is for edge weights to vanish, and relate the rank deficiency to the number of connected components of a modified, undirected version of the graph. This connects singularities to disconnected sub-networks and lets them prove invariance and unreachability.

Empirically, they (i) simulate the toy DAG disconnection scenario using dedicated software (Boccato et al., 2024a) in discrete settings, and (ii) train shallow and deep MLPs, with and without biases and skip connections, on the Breast Cancer dataset with SGD at step size 0.001 to approximate continuous gradient flow, comparing nuclear-norm, L1, and L2 regularization while tracking the Jacobian's rank.

Why This Matters

Impact on research. The paper shows that the algebra of conservation laws, previously understood for shallow networks, generalizes cleanly to arbitrary DAG topologies and is tightly coupled to graph structure — bottlenecks, pure ancestors/descendants, and stable subsets. It also supplies a reachability result: singularities of the invariant set are generically nonexistent and cannot be entered in finite time by gradient flow, while any that are entered are permanent. This constrains what training can and cannot achieve with common initializations and suggests that some functions are excluded at initialization rather than during training.

Real-world applications:

• Model compression and pruning. Because singularities correspond to effective sub-networks, driving the model toward them yields lossless pruning; the nuclear norm regularizer is structure-agnostic and applies to arbitrary DAG topologies.
• Sparse and pruned architectures. Networks that emerge from unstructured pruning or from lottery-ticket-style sparse subnetwork sampling are DAGs, which is precisely the setting this theory addresses.
• Architecture design. The bottleneck characterization gives a concrete criterion — whether single-input or single-output hidden neurons exist, and what their balance values c_v are — for whether a given design has a connected optimization space.
• Diagnosing training failures. Disconnectedness explains situations in which a network cannot reach an optimum no matter how long it trains or which loss is chosen, because the optimum lies in a different connected component of the invariant set.

Industry relevance. The results bear on the development of differentiable pruning tools, the analysis of compressed deployments where the effective architecture is a DAG, and design choices around output-layer width (scalar regression and binary classification routinely produce out-bottleneck neurons). The authors caution, however, that current computational limitations restrict the practical use of the nuclear norm regularizer.

Future Directions

• Making the nuclear norm regularizer practical. The authors explicitly note that computational limitations restrict its practical use; scaling the surrogate to large networks is an open engineering and theoretical problem.
• Explaining why L1 induces singularities. The paper reports that L1 empirically behaves like the nuclear norm regularizer but states that a precise theoretical understanding is outside its scope, leaving this as an open question (an empirical analysis is provided in the paper's appendix).
• From gradient flow to discrete-time training. The analysis is conducted in continuous time; the authors note that the discrete-time setting of gradient descent has also received attention in the literature, and reconciling the reachability results with finite-step optimizers is a natural extension.
• Extending beyond ReLU and to other homogeneous activations and modules. The framework applies to any positively homogeneous activation (Leaky ReLU with 0 ≤ γ ≤ 1 is given as an example) and handles biases through a virtual input neuron fixed to 1, suggesting room to broaden the class of architectures analyzed.

Target Audience

This paper is aimed at machine learning theorists and mathematically inclined researchers working on the geometry of neural network optimization, particularly those already familiar with conservation laws and rescaling symmetries in ReLU networks. It is also relevant to researchers in algebraic geometry and topology looking for applications to deep learning, to practitioners of pruning and model compression who want a principled rather than heuristic foundation, and to graduate students with a strong mathematics background who are studying training dynamics. Readers seeking implementation-ready recipes will find the paper's own caution about the impracticality of its regularizer relevant; readers seeking intuition alone can follow the bottleneck and singularity explanations without the full proofs.

Authors’ abstract

Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments.