Continuous-Depth and Implicit Neural Networks

These views allocate computation and representation depth differently. An explicit residual stack applies a finite sequence of learned transformations. The layer count is predictable and debugging is straightforward. Activation memory grows with depth, and weights may differ by layer. A neural ODE integrates a learned derivative field over continuous time. Solver steps adapt, the trajectory is continuous, and solver error reaches the output; stiff dynamics can be costly. A deep equilibrium model solves for a state unchanged by a learned transformation — an implicit infinite-depth interpretation, with constant-memory differentiation options. It requires the fixed point to converge, and multiple equilibria can exist. An unrolled iterative network runs a shared block for a chosen number of steps. The approximation is simple and the compute controllable, truncation error remains, and training and inference step counts may differ.

The second row arrived in 2018. Instead of stacking layers, learn how the state changes: “instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network”. The output is then “computed using a black-box differential equation solver”. What the paper claims for such models is operational rather than aesthetic. They “have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed”.

The caveats usually listed under that column are operational too — adaptive steps, solver error, stiffness. They invite the assumption that a better solver would clear them. One of the caveats is a theorem. Augmented Neural ODEs, in 2019, opens its abstract with the result itself: “We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent.” The feature map of a neural ODE is a homeomorphism. Concretely, the flow cannot represent the function g with g(x) = -1 for ||x|| <= r1 and g(x) = +1 for r2 <= ||x|| <= r3, where 0 < r1 < r2 < r3. That is an annulus that has to be separated from the ball it surrounds, and a continuous invertible deformation of the space cannot do it. A capability gap, not a training difficulty. No tolerance setting reaches it.

The repair is dimensional rather than numerical. A 2020 ICML paper proves the complementary half from the other side. Any homeomorphism on p-dimensional Euclidean space can be approximated by a neural ODE operating on 2p dimensions. Cap the model with a single linear layer and it becomes a universal approximator for non-invertible continuous functions. Read the two together and the comparison table gains a row it did not have: what the architecture can express at all, before any question of how well it is solved.

The network defines how the hidden state should change as a function of state and time. A numerical integrator then approximates the resulting trajectory between the specified endpoints.

Tolerance, step size, solver family, and stiffness determine runtime and numerical error. The solver is therefore part of the deployed computation, not an invisible detail of the code.

The idea did not arrive quietly, and the rest of this lesson is easier to read if you know that. Neural Ordinary Differential Equations was named one of the four Best Papers at NeurIPS 2018, announced on 3 December 2018. Two of those four went to Vector Institute researchers.

Everything that follows here — unstable adjoint gradients, evaluation counts that climb during training, functions the flow provably cannot represent — was published about that paper, mostly within two years of it. It came from authors who took it seriously enough to measure it. The caveats are not a verdict on the idea. They are what a field does to a construction it intends to keep.

The representation is defined by a fixed-point equation rather than by a fixed number of distinct layers.

The computation runs as a search, not as a pass. Initialize the state from an input-derived or zero representation. Apply the shared transformation to compute the next candidate state. Measure the residual by comparing that candidate with the previous state. Iterate or accelerate, using fixed-point or root-finding methods rather than plain repetition. Accept the equilibrium once a state satisfies the tolerance criterion.

Nothing in that loop fixes its own length. The number of passes is decided at run time by the residual and the tolerance. That is why the sections that follow treat iteration count as a measured quantity, not as a property of the architecture.

The benefit depends on dynamics or iterative structure, not on novelty of terminology. The first of these motivations has a named public benchmark and published numbers attached to it. That is the standard the others should be held to as well.

The benchmark is an ICU dataset. “We evaluated our model on the PhysioNet Challenge 2012 dataset [Silva et al., 2012], which contains 8000 time series, each containing measurements from the first 48 hours of a different patient's admission to ICU.” That comes from the 2019 latent ODE paper. The task is in-hospital mortality, with 13.75% positive labels. PhysioNet documents the benchmark itself: records from 12,000 ICU stays, 4,000 of them in training set A, up to 42 variables recorded at least once in the first 48 hours, and ICU stays of under 48 hours excluded. PhysioNet published it on 20 January 2012.

The results are the useful part, and they are modest. Test AUC came out at 0.833 +/- 0.009 for ODE-RNN and 0.829 +/- 0.004 for the Latent ODE. The discrete-time GRU-D baseline reached 0.818 +/- 0.008. The authors conclude that the three achieved similar classification AUC. That is a margin of about 1.5 AUC points over a discrete-time model, on the task continuous depth is supposed to be built for. Real, reported with its error bars, and small. It only exists as a fact because someone ran the discrete baseline alongside.

Separate what the model buys from what a cleverer solver buys. Five steps, and the published literature has already run two of them hard enough to show what they catch.

1. Establish an explicit baseline. Compare a residual or unrolled network at matched quality and compute. The Multiscale DEQ paper did this in 2020: 77.5% ImageNet top-1 with 63M parameters against ResNet-101's 77.1% with 52M, and MDEQ-XL at 79.2% with 81M. The number that makes the comparison informative is the one the authors did not have to publish. The same MDEQ-large, truncated into a 5-layer unrolled network, reaches only 75.9%. Without that truncated run, the contribution of the implicit formulation is simply unmeasured.

2. Log solver work. Record iterations, function evaluations, residuals, and failure codes per input.

3. Sweep tolerances. Measure quality and runtime sensitivity instead of selecting one arbitrary value.

4. Test difficult regimes. Include long horizons, stiffness, noise, and distribution shift.

5. Validate gradients on small controlled problems, comparing implicit, adjoint, or unrolled estimates. This is the step that has already caught a headline method. The constant-memory adjoint recovers activations by solving the forward ODE backwards in time. The ANODE paper, in 2019, examined that trick and put the finding in its abstract: “However, we will show that this approach may lead to several problems: (i) it may be numerically unstable for ReLU/non-ReLU activations and general convolution operators, and (ii) the proposed optimize-then-discretize approach may lead to divergent training due to inconsistent gradients for small time step sizes.” The error is O(1) in the gradient. Not a rounding nuisance. A wrong direction.

The controlled example is as small as an example gets. For dz/dt = -100z over a unit time horizon, resolving the forward and backward flows to within 1% takes about 200,000 time steps. At coefficient -1e4 the flow is “impossible to reverse numerically in double precision”. Scale up to a model anyone would recognise and the same thing happens: testing Neural ODE with an RK45 solver on ResNet-18/Cifar-10 “leads to divergent training in the first epoch”. A second group reached the same place independently. Six authors at ICML 2020 attribute neural ODEs' weaker benchmark performance to this reverse-mode gradient error, and report that their checkpoint-based alternative “achieves half the error rate in half the training time”. Two of those numbers came out of exactly the small controlled comparison step 5 asks for: 200,000 steps for 1% accuracy, and divergence within one epoch.

A neural ODE still uses discrete numerical steps on hardware. An equilibrium layer returns an approximate fixed point under a stopping criterion.

Gradient methods add their own approximation and stability choices. Report solver, tolerances, iteration caps, and failure handling so the architecture can be reproduced and audited.

The equilibrium side carries a measured memory claim rather than an unbounded one. Deep equilibrium models were presented in 2019. They find the fixed point “via root-finding”. They backpropagate through it “analytically”, using implicit differentiation. So “training and prediction in these networks require only constant memory, regardless of the effective ‘depth’ of the network”. On WikiText-103 they report “an up-to 88% memory reduction”. What they do not report is a fixed number of iterations. There is not one.

The bill for that memory arrives in solver work, and the same authors itemised it two years later. Their Jacobian regularization paper, in 2021, says it plainly: “Second, the number of iterations required to solve for the equilibrium quickly grows over the course of training, indicating a trend for approaching instability.” In figures: a DEQ at the end of training can take more than 3x the solver iterations it took earlier. A DEQ-Transformer of 98M parameters reaching perplexity 24.0 on WikiText-103 needs 30 function evaluations at inference and 3.1x the training time of an 18-layer Transformer-XL. And “a multiscale DEQ (Bai et al., 2020) is over 4x slower than ResNet-101 on ImageNet”.

The fix is itself a trade rather than a repair. Jacobian regularization brings the DEQ-Transformer down to 12 function evaluations and 1.5x training time, at perplexity 24.9. Slightly worse language modelling in exchange for well under half the inference work. Constant memory is a real property, quoted correctly at up to 88%. It is sold at a price in iterations that is not constant, not published with the architecture, and measurable only on your own workload.