The Pulse

Study Finds a Mathematical Limit to AI Reproducibility

A study proposes that intertwined training basins can make deep-learning outcomes difficult to predict even when external randomness is controlled. Its tests find near-zero uncertainty exponents in a minimal model and in deep networks, mean

AI.info Team · Written with AI · fact-checked · 2026-11-06

In a minimal neural-network model, the uncertainty exponent was 0.0126 ± 0.0002. This near-zero value means, the study says, that increasing the precision of an initialization yields only marginal gains in predicting the training outcome. Published 5 November 2026 in Nature Machine Intelligence, the research proposes riddled basin geometry as a dynamical-systems account of unpredictability and irreproducibility in deep learning.

Why smaller initialization changes may not help

Training outcomes are often difficult to predict and reproduce, even when external sources of randomness are controlled. Earlier research has examined effects such as random initialization, mini-batch ordering, data augmentation and numerical errors introduced by computing platforms. The article notes evidence that changing a parameter by as little as one bit can produce variability comparable to that associated with multiple noise sources combined. It asks what intrinsic limits on predictability may remain after those external sources are eliminated.

The proposed mechanism concerns the basins of attraction associated with different training outcomes. These basins are intertwined, so an initialization leading to one solution lies arbitrarily close to initializations leading to another. The authors describe the resulting structure as a “fat” fractal, with detail at arbitrarily fine scales.

The article distinguishes this outcome-level uncertainty from chaos. In chaotic dynamics, small errors in initial conditions grow over time and limit the prediction of a trajectory. Riddling, by contrast, concerns which qualitative destination training reaches: uncertainty can persist even under arbitrarily small changes to the initial conditions.

A two-layer network reveals the mechanism

To examine how riddling can arise in learning dynamics, the researchers use a two-layer network, described as the smallest architecture that admits a non-trivial symmetry. It uses a hyperbolic tangent activation and is trained on a regression task with eight input-output pairs whose inputs and outputs are drawn from a standard normal distribution. The model is trained by deterministic gradient descent to minimize mean squared error.

The paper analytically identifies symmetry-induced invariant subspaces in the model. These include planes where the two hidden neurons have identical weights or weights that differ by a sign, as well as planes where one neuron’s weights vanish. The researchers find that prominent competing destinations among random initializations are the two permutation-invariant planes. At large learning rates, training can also diverge, an outcome the study treats as an attractor at infinity.

At a learning rate of 2.5, one invariant plane contains a chaotic attractor. The paper reports a positive maximal longitudinal Lyapunov exponent, indicating chaos within the plane, while the transverse exponents are negative overall. It also finds positive fluctuations in finite-time transverse exponents, evidence for transversely unstable periodic points. Together with competing attractors, these features meet theoretical conditions sufficient for riddling.

A destination map samples a 2,047 × 2,047 grid of initializations and shows paths to the two permutation-invariant planes or to infinity. The map reveals interwoven destinations, including at finer magnifications of areas that look uniform at coarser scales. The measured uncertainty exponent, 0.0126 ± 0.0002, quantifies how slowly the fraction of uncertain outcomes falls as the allowed perturbation shrinks.

Similar signatures in deep networks

The study extends its analysis to deep networks, reporting intertwined training destinations and near-zero parameter-space uncertainty exponents across the tested configurations. The researchers also introduce functional uncertainty exponents, which assess changes in learned functions rather than only changes in network parameters. Their results show near-zero functional exponents as well, indicating that the scale-insensitive uncertainty appears in function space.

The reported signatures are consistent across the tested architectures, optimizers, regularization methods and activation functions. The paper also presents riddling as a framework that can account for observations including irreducible model variability and trade-offs between performance and reproducibility. It puts forward riddled basin geometry as a candidate organizing principle for irreproducibility in deep learning, with implications for optimization and reliable AI deployment.