Taylor Expansions, Remainders, and Curvature

Near zero, the logistic function can be approximated without evaluating the full exponential expression.

A derivative describes the first-order effect of a small change. Taylor expansion organizes higher-order effects and states how the approximation error behaves near a reference point. For a scalar function of one variable, the first-order model is a tangent line. Adding the second derivative produces a local parabola that can represent curvature. In several dimensions, the gradient supplies the linear term and the Hessian supplies the quadratic term. This is the mathematical bridge from calculus to local optimization.

The name is younger than the mathematics, and by a checkable margin. Brook Taylor stated the theorem that carries his name in Methodus Incrementorum Directa & Inversa, published in 1715. James Gregory had written to John Collins on 15 February 1671 — forty-four years earlier. That letter carries power series, each of five or six terms, for seven functions: arctan x, tan x, sec x, log sec x, log tan(x/2 + π/4), arcsec(√2 e^x) and 2 arctan(tanh(x/2)). J J O'Connor and E F Robertson, who wrote the MacTutor biography of Gregory, state it without hedging: “In February 1671 he discovered Taylor series (not published by Taylor until 1715), and the theorem is contained in a letter sent to Collins on 15 February 1671.” The same biography records that “he discovered Taylor series more than 40 years before Taylor”, and the archive's transcription page for the Gregory–Collins correspondence notes that the letter “contains six examples of functions expanded using Taylor series many years before Brook Taylor was born”.

One distinction survives all of that, and it is the one that matters for everything below. Gregory possessed a method for finding the Taylor expansion of any particular function. What the surviving material does not establish is that he had Taylor's general formula for an unspecified function. That is Dehn and Hellinger's conclusion. Expanding the function in front of you is one act; owning a statement that covers functions you have not seen is another. A truncated expansion is always the first kind of object. The rest of this lesson is about not treating it as the second.

Writing “approximately equal” without a remainder condition hides the radius of trust. Taylor's theorem relates the error to higher derivatives and the distance from the expansion point. A smoothness assumption converts that relation into an inequality you can compute with.

Here is the worked case. Assume the Hessian is Lipschitz continuous with constant L. Then for every step s, f(x_k+s) <= f(x_k) + s'g(x_k) + (1/2)s'H(x_k)s + (L/6)||s||^3. The first three terms are the ordinary second-order Taylor model. The fourth is the remainder — bounded, not dropped. The inequality holds for all s rather than only for small s, so the whole expression is an upper bound on the function everywhere, and a step that decreases the model cannot increase f. Nesterov and Polyak published this cubic regularization of Newton's method in 2006. A 2017 paper building a stochastic version describes the mechanism in one line: “In each iteration, it first forms a local upper bound on the function using a third-order Taylor expansion around the current iterate”.

The payoff is a number, not a sentiment. Minimising the cubic upper model drives the gradient norm below eps in at most O(eps^-3/2) iterations. Steepest-descent-like methods carry an O(eps^-2) bound. Cartis and two co-authors, working on adaptive cubic regularisation, state the same overestimation property under a globally Lipschitz Hessian and the iteration count that follows from it. Tripuraneni and co-authors give the same count independently: “The work of Nesterov and Polyak [2006] first proposed the cubic-regularized Newton method, which requires O(eps^-1.5) gradient and Hessian oracle calls to find an eps-second-order stationary point”.

That gap is what remainder control buys. The habit of bounding the error is not bookkeeping added after the derivation; it is the thing that changes the convergence rate. Optimization line searches and trust-region methods rely on exactly this ability to tell a local model apart from the true objective.

Each added derivative captures another kind of local behavior: the value at the point, the slope through it, the bending around it, and then everything the truncation left out. The fourth layer is not decorative. Under a Lipschitz Hessian with constant L it is exactly the (L/6)||s||^3 term that Nesterov and Polyak add to turn the quadratic model into an upper bound on the function. That is the difference between a picture of the neighborhood and a guarantee about it.

Apply these checks when a derivation replaces a nonlinear function with a local approximation: name the expansion point, choose the lowest order that answers the question, check that the required derivatives exist in a relevant neighborhood, bound the remainder, confirm the approximation does not leave the valid domain, and build a new model once the iterate has left the trusted region.

Two of those steps have a measured quantity behind them. The largest Hessian eigenvalue does not stay where it starts. Under full-batch gradient descent on neural networks it rises until it sits just above the value at which a quadratic Taylor model of the loss stops being a contraction for the chosen step size, and then it stays there. Cohen and four co-authors tracked it and reported that in 2021. Their abstract reads: “In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value 2 / (step size), and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales.” Arora and two co-authors, a separate group, report the same stabilisation in 2022: “Sharpness stabilizes around 2/LR and loss goes up and down across iterations, yet still with an overall downward trend”.

That converts two pieces of advice into two things to compute. Bounding the remainder has a target to compare against — the top eigenvalue against 2/(step size) — rather than a vague instruction to be careful. Re-evaluating after moving has a reason too. In these runs the curvature is actively drifting toward the boundary of what the step size tolerates, so the bound you established at the old iterate is not the bound at the new one. A loss that rises for a few iterations and falls over many is not automatically a defect. It is what training at that threshold looks like.

The two approximations offer different information at different computational costs. A first-order model uses the function value and the gradient. It captures local direction, is cheap in high dimensions, and supports gradient methods, but on its own it cannot distinguish a flat minimum from a saddle. A second-order model adds the Hessian: it predicts local bending and supports Newton-like steps, at the price of solving a linear system, and it can mislead far from the reference point. A finite-difference fit estimates local behavior from evaluations. That is useful for derivative checks, but it is sensitive to step size and noise, blind to exact symbolic structure, and expensive in many dimensions.

The first-order limitation is the one with measurements behind it. A Newton-based search can locate the critical points of MNIST and CIFAR-10 networks, and at each one you can measure the index — the fraction of the Hessian's eigenvalues that are negative. Dauphin and five co-authors did exactly that in 2014. The index rose tightly and monotonically with the training error at that point. The MNIST critical points are reported at 0.32%, 23.49% and 28.23% training error, with indices spanning roughly 0 to 0.25. Their conclusion states that the experiments “confirmed the main qualitative prediction of theory that the index of a critical point tightly and positively correlates with its error level”. Read backwards, that is a diagnosis. A point where the gradient has vanished and the error is still high has negative curvature in a quarter of its directions, and the gradient by itself reports none of it.

The abstract states the general case: “Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest.” Pennington and Bahri later derived the same relation from random matrix theory: “Our analysis predicts and numerical simulations support that for critical points of small index, the number of negative eigenvalues scales like the 3/2 power of the energy”. The second-order term is what tells the two situations apart, and its cost is the price of the distinction.