Paradoxical noise preference in RNNs

AI summary

Overview

Research area: Computational neuroscience and neural network theory — specifically the dynamics of continuous-time recurrent neural networks (CTRNNs) trained with injected noise.

Technical level: Advanced. The paper relies on stochastic differential equations, fixed-point/attractor analysis, stationary distributions, and Ornstein–Uhlenbeck process theory.

Scope: The paper demonstrates, explains mechanistically, and bounds the conditions under which noisy recurrent networks perform best at a nonzero test-time noise level rather than at zero noise.

Authors and affiliation: Noah Eckstein and Manoj Srinivasan, Department of Mechanical and Aerospace Engineering, The Ohio State University. Posted as arXiv:2601.04539v2 [cs.NE], 01 Jun 2026.

What This Paper Is About

Noise is routinely injected into recurrent neural networks during training, both to emulate biological variability and to regularize learning. The standard intuition is that noise is a training-time crutch: removing it at test time should preserve or improve performance. This paper shows the opposite happens in a large class of networks — CTRNNs trained with noise injected inside the activation function often perform best at test time when the same nonzero noise level is applied, and performance degrades if the noise is removed. The authors set out to explain why this happens, why it happens only for certain noise-placement schemes, and whether it reflects a genuine computational dependency rather than a regularization artifact.

Key Contributions

1. Demonstrates the phenomenon across diverse tasks. Noise preference is shown in function approximation, maze navigation, 2D path integration, a multi-cognitive task suite, a single-neuron regulator task, and even in feedforward multilayer perceptrons — establishing that it is not an artifact of one architecture or task.

2. Identifies noise placement as the decisive variable. Networks trained with noise injected inside the activation function ("noise-in", pre-activation noise) develop a preference for nonzero test noise. Networks trained with noise injected outside the activation function ("noise-out", post-activation noise) perform best at zero noise.

3. Provides a mechanistic explanation via noise-induced fixed-point shifts. The paper shows analytically and numerically that noise shifts the mean of the stationary distribution (the effective fixed point) of the underlying stochastic dynamics, and that these shifts predict the input-specific error bias observed at zero test noise.

4. Explains why only noise-in networks develop the preference. A single-neuron regulator analysis shows that noise-in networks have a performance incentive to place their stationary distributions near activation-function saturation boundaries (where noise is asymmetrically attenuated for variance reduction), while noise-out networks have the opposite incentive — resolving the asymmetry.

Main Findings

• Noise-in networks prefer the training noise level. For noise-in CTRNNs trained on function computation with sufficiently high training noise, root mean squared error (RMSE) was minimized when test noise approximately matched training noise. This contradicts the assumption that noise acts purely as a regularizer.

• Noise-out networks prefer zero noise. Noise-out CTRNNs performed best with zero test noise, and their systematic error appeared unaffected by changes in the noise level — consistent with a purely regularizing role.

• The effect is driven by bias, not variance. Error variance grew roughly linearly with the variance of injected noise. Systematic error (bias) changed non-monotonically, being high both at zero noise and at noise much higher than training noise. In the noise-in function computation networks, bias was minimized at a slightly higher noise level than total error.

• Sharper preference with more training noise, worse peak performance. Noise preference became more pronounced as training noise increased, with very low training noise producing essentially zero noise preference. However, performance at the preferred noise level got worse as training noise increased — suggesting the noise was not conferring the test-performance advantage expected of a regularizer.

• The effect generalizes beyond simple tasks and beyond recurrence. Noise-in networks on the maze navigation, multi-cognitive, and path integration tasks also performed best at nonzero noise levels close to training noise. For the multi-cognitive suite and path integration, optimal performance was farther from — and generally smaller than — the training noise level. Notably, the error bias in these tasks was still optimal at nearly the training noise level, while error standard deviation increased monotonically; together these imply a total-error minimum to the left of the training noise.

• Feedforward networks show it too. Noise-in multilayer perceptrons trained on the sine function computation task also developed a noise preference driven by error bias. Unlike the RNNs, the MLP bias was minimized at slightly higher than the training noise level.

• Fixed-point shifts quantitatively predict zero-noise error. For noise-in function computation networks, the authors computed the difference between zero-noise and training-noise fixed-point locations, projected that shift through the output transform (z = W_out h + B_out), and found the projected shifts closely tracked the input-specific error bias observed at zero test noise.

• The mechanism depends on proximity to nonlinear boundaries. For a single noise-in neuron, if the mean synaptic input μ_s is large and negative relative to σ_in, noise rarely excites the neuron; if large and positive, noise acts as excitation and inhibition in equal proportion with zero net effect. But when |μ_s| is on the order of σ_in, the argument regularly fluctuates into and out of the saturated region, selectively attenuating inhibitory noise samples and producing a σ_in-dependent net excitatory shift. This is why fixed points near boundaries between linear regions become noise-level dependent.

• Noise-out networks can also shift, but have no incentive to. A piecewise Ornstein–Uhlenbeck process with different decay rates above and below a boundary acquires a positive shift in its stationary distribution as noise grows, because positive deviations decay more slowly. However, adding a large neural bias inside the activation function moves the piecewise boundary away from the fixed point, eliminating the shift. This demonstrates that a shift is possible but not incentivized.

• The single-neuron regulator reveals the tradeoff. With only one neuron, no inputs, and direct neural output, the only trainable parameters are the scalar recurrent weight and scalar neural bias. The task is parameterized by two dimensionless quantities: σ_in/σ_out and r/σ_out. When pre-activation noise dominates, networks tend to land in regions exhibiting a noise preference; when post-activation noise dominates, they land squarely outside those regions. Placing the stationary distribution near saturation reduces degradation from pre-activation noise while exacerbating degradation from post-activation noise — a good trade for noise-in networks, a bad one for noise-out networks.

• The phenomenon is not restricted to rectifying activations. Noise preference appeared in both ReLU and tanh single-neuron regulator experiments. Because tanh saturates at both high and low inputs, noise preference was found when the setpoint was high or low. The paper frames the requirement as saturating rather than specifically rectifying.

• It is mechanistically distinct from stochastic resonance. The authors explicitly state that the phenomenon differs from stochastic resonance, in which nonzero noise enhances signal processing. Here, noise preference arises from learned dynamical structure and output bias, not signal enhancement.

• Noise may be required for stable training in harder tasks. For the maze navigation network, the training noise level used represented an approximate lower bound for stable training; lower noise caused dynamical instability and frequent catastrophic explosions in the performance cost. Similar instability was observed in the multi-cognitive task suite, which the authors initially tried to train at a noise level half that presented.

• There was no train/test split in these experiments. For all tasks explored, batches were sampled generatively from the same task distribution during both training and testing, and that distribution lacked particularly heavy tails. The authors note that the training performance deficit one expects from regularization therefore also applies at test time, and that in such cases the best hope from noise as a regularizer is improved training stability.

Methodology in Plain English

The authors used continuous-time recurrent neural networks whose internal state evolves according to a differential equation: neural activity decays toward a nonlinear function of the recurrent input, external input, bias, and noise. They integrated this equation with Euler's method, using a neural time constant τ = 0.1 and a discrete timestep Δt = 0.02, giving a non-dimensional ratio γ = 0.2. Noise was sampled independently for each neuron at each timestep.

The central experimental manipulation was where the noise is injected. In the standard "noise-in" formulation, Gaussian noise is added to the summed, biased synaptic inputs before the activation function is applied. In the alternative "noise-out" formulation, the noise is added after the activation function. Most experiments used rectifying activations — ReLU or its smooth approximation, SoftPlus — with tanh also used in the single-neuron regulator experiments.

They then trained networks on a battery of supervised tasks and, crucially, swept the test-time noise level rather than just testing at zero. By decomposing test error into bias (mean error) and variance (error standard deviation) across an ensemble of simulations with different noise realizations, they could determine whether performance deficits away from the training noise came from systematic error or from trial-to-trial variability. This decomposition turned out to be the key diagnostic.

The tasks spanned several levels of complexity. Two univariate function computation tasks required outputting sin(x) on [0, 2π] or tanh(x) on [−4, 4], with either a fixed readout time or a readout time sampled uniformly from a final interval. A maze navigation task cast the network as an instantaneous velocity controller moving a particle between maze vertices sampled uniformly at random, while rejecting random velocity perturbations. A path integration task followed the version studied by Cueva & Wei (2018), requiring the network to estimate a particle's position in a 2D square room from speed and heading inputs. A multi-cognitive task suite used 6 of the 20 tasks popularized by Yang et al. (2019), with names taken from Driscoll et al. (2024): ReactPro/ReactAnti, MemoryPro/MemoryAnti, and DelayPro/DelayAnti — all involving 2D directional stimuli and saccadic eye-movement-style responses. A single-neuron regulator task stripped the problem to one self-recurrent neuron with no inputs and direct neural output, to isolate the mechanism with minimal parameters. Finally, feedforward MLPs were trained on the sine task to test whether temporal dynamics were necessary.

To connect behavior to mechanism, the authors computed the fixed points of the trained networks. Since true fixed points do not exist in a stochastic system, they defined "fixed point" as the mean of the stationary distribution near a point attractor. They then measured how the zero-noise fixed point differs from the training-noise fixed point, projected that difference through the output readout, and compared it against the actual input-specific error bias measured at zero test noise. Separately, they constructed a single-neuron piecewise Ornstein–Uhlenbeck system with two different decay rates — one for each side of a boundary between linear regions — to derive the noise-dependent stationary distribution shift analytically.

Why This Matters

For computational neuroscience: The results suggest a concrete hypothesis about biological circuits. Neural populations that operate near nonlinear activation boundaries may leverage — or become functionally dependent on — synaptic variability to stabilize activity and reduce variance. This implies that some noise sensitivity observed in biological systems could reflect learned dynamical structure rather than incidental biological variability. The finding also reframes how one should interpret the widespread practice of training RNN models of neural circuits with noise: the noise may not be a neutral modeling detail but something the network co-opts as part of its computation.

For machine learning: The work identifies a subtle failure mode of noise injection — overfitting to the stochastic training environment itself, rather than only to the input–output data. This is distinct from classic overfitting and from the ordinary regularization view. It warns that a noisy-trained model's robustness cannot be assumed and that test-time noise conditions become a hyperparameter of the deployed model, not an incidental detail.

Real-world applications. The paper itself does not enumerate specific deployed applications; the following follow from its stated implications:

• Neuromorphic and analog computing hardware, where device-level noise is intrinsic and cannot simply be "switched off" at inference — networks trained with noise-in schemes may functionally depend on a noise level that hardware does not reproduce.
• Brain–machine interfaces and neural prosthetics that use RNN controllers trained with noise to match biological variability, where mismatched inference-time noise statistics could introduce systematic output bias.
• Simulation and modeling of biological circuits, where fitting a noisy-trained RNN and then analyzing it at zero noise could produce misleading conclusions about the circuit's learned dynamics.
• Robust control and robotics, where a velocity-controller network trained with noise rejection (as in the maze task) may have its behavior silently tuned to the training noise amplitude.

Industry relevance: Any production pipeline that adds Gaussian noise during training — whether for regularization, data augmentation, or variance reduction — should be aware that the resulting model may have a nonzero noise optimum, and that this optimum can be measured with a simple test-time noise sweep. The paper also notes that in some tasks noise is required for training stability at all: the maze network's training noise was an approximate lower bound, and the multi-cognitive network was initially attempted at half that level before instability forced an increase. That makes noise level a training-critical hyperparameter, not merely an augmentation knob.

Future Directions

• Testing whether biological circuits exhibit the predicted dependency. The paper proposes that circuits operating near nonlinear activation boundaries may become functionally dependent on synaptic variability. This is stated as a hypothesis and is not tested against experimental neural data here; direct empirical validation remains open.

• Characterizing the phenomenon in higher-dimensional and non-point attractors. The path integration task demonstrates the effect in a network using higher-dimensional attractors rather than simple point attractors, but the fixed-point analysis in Sections 3.3 and 3.4 is worked out for point-attractor-like settings and a single-neuron piecewise Ornstein–Uhlenbeck model. Extending the mechanistic account to continuous attractors and to networks whose computation is not fixed-point-based is an open direction.

• Understanding the discrepancy between RNNs and MLPs. In noise-in RNNs on the simple function tasks, bias was minimized at slightly higher than the training noise; in the feedforward MLPs, bias was also minimized at slightly higher than the training noise. Yet for the multi-cognitive suite and path integration, the optimal total performance was notably farther from training noise and generally smaller. Reconciling when the optimum sits at, above, or below the training level is not resolved.

• Quantifying the tradeoff against training stability. The paper reports that the maze navigation noise level was an approximate lower bound for stable training, and that the multi-cognitive network destabilized at half that level, but it does not characterize the shape of that stability boundary or how it interacts with the noise-preference optimum. Mapping this tradeoff would be practically valuable.

• A fuller treatment of the regularization question. The paper notes that its tasks had no train/test split and that the distribution lacked heavy tails, so the usual generalization benefit of regularization could not be assessed. Whether noise-in networks show an analogous preference under a genuine distribution shift — with heavy-tailed or out-of-distribution test data — is left open.

Target Audience

This paper is best suited to researchers in computational neuroscience who build RNN models of neural population dynamics, and to machine learning researchers studying training dynamics, noise injection, and robustness. It will be particularly useful to anyone who trains recurrent networks with injected noise and then evaluates them at zero noise, since the paper identifies that practice as potentially invalid. Readers will need comfort with stochastic differential equations, fixed-point and stationary-distribution reasoning, and the bias–variance decomposition of error. It is less suited to practitioners looking for an applied recipe: the paper provides a mechanism and a diagnostic, not a drop-in method, and reports no test-set generalization results. Graduate students and postdocs in dynamical systems approaches to neuroscience are likely to get the most out of it.

Authors’ abstract

In recurrent neural networks (RNNs) used to model biological neural networks, noise is typically introduced during training to emulate biological variability and regularize learning. The expectation is that removing the noise at test time should preserve or improve performance. Contrary to this intuition, we find that continuous-time RNNs (CTRNNs) often perform best at or near the training noise level. This noise preference typically arises when noise is injected inside the neural activation function; networks trained with noise injected outside the activation function perform best with zero noise. The phenomenon arises robustly in diverse tasks for large enough training noise; we also show the phenomenon arising in feedforward neural networks, not just in RNNs. Our analyses show that the phenomenon stems from noise-induced shifts of fixed points (stationary distributions) in the underlying stochastic dynamics of the RNNs. These fixed point shifts are noise-level dependent and bias the network outputs when the noise is removed, degrading performance. Analytical and numerical results show that the bias arises when neural states operate near activation-function nonlinearities, where noise is asymmetrically attenuated, and that performance optimization incentivizes operation near these nonlinearities; such performance incentives exist for networks with noise inside, but not outside, the activation function, explaining why only noise-in networks show the preference. Thus, networks can overfit to the training noise itself rather than just to the input-output data. The phenomenon is distinct from stochastic resonance, wherein nonzero noise enhances signal processing. Our findings reveal that training noise can become an integral part of the computation learned by neural networks, with implications for understanding neural population dynamics and for the design of robust artificial RNNs.