Derivatives and Local Linearization

The derivative f′(x) of a scalar function f(x) describes how the output would change under an infinitesimal change in x. It is a local sensitivity, not a global summary. The linear approximation f(x + Δ) ≈ f(x) + f′(x)Δ turns that sensitivity into a nearby prediction. It improves as Δ becomes small enough for curvature to matter less. ML uses the idea to update parameters, analyze feature sensitivity, and propagate influence through composed functions.

What follows treats that approximation as something with measured consequences rather than a formality. What one linear step can do to a trained classifier. What a gradient costs to compute by each available method. And where the arithmetic of the machine stops the mathematics of the limit.

The first-order Taylor model uses slope: f(x+Δ) ≈ f(x)+f′(x)Δ. The second-order model adds curvature: ½f″(x)Δ². A small derivative does not always mean little change. If curvature is large or Δ is not small, the second-order term can dominate. Optimization methods differ partly in which local model they trust. Gradient descent uses first-order information; Newton-style reasoning uses curvature.

How much a first-order model buys is measurable, and it was measured on the input rather than on the parameters. In 2015 Goodfellow and two colleagues built a perturbation out of nothing but the linear term: “We can linearize the cost function around the current value of θ, obtaining an optimal max-norm constrained pertubation”. One step in the sign of that gradient, at epsilon = 0.25, drove a shallow softmax classifier to a 99.9% error rate on the MNIST test set, with 79.3% average confidence in its wrong answers. At epsilon = 0.1 the same construction produced an 87.15% error rate on CIFAR-10, with 96.6% average probability on the wrong labels.

NIST catalogues the technique in exactly those terms: “Goodfellow et al. [144] introduced an efficient method for generating adversarial examples for deep learning: the Fast Gradient Sign Method (FGSM), which performs a single iteration of gradient descent for solving the optimization.”

Read the numbers as a statement about the local model and not only about security. The linearization is valid in a neighborhood. The attack works because it stays inside one and spends its whole budget on the slope.

Finite changes reveal average behavior. The derivative emerges as the interval shrinks. The four steps are the definition. They are also an instruction no machine can carry out to its end, which is the subject of the callout above and of the closing section below.

All three estimate derivatives. They rely on different mechanisms, and they do not cost the same. Baydin and colleagues quantify the gap in their 2018 survey: “The O(n) complexity of numerical differentiation for a gradient in n dimensions is the main obstacle to its usefulness in machine learning, where n can be as large as millions or billions in state-of-the-art deep learning models”. Reverse-mode automatic differentiation instead returns the whole gradient of a scalar function in a single pass. Its operation count is bounded by a constant factor c that is guaranteed to be c < 6 and is typically in the range 2 to 3.

The same contrast appears in an independent 2019 review by Margossian. Finite differentiation “requires at least D evaluations, where D is the number of partial derivatives required”, while AD is “Exact, speed is comparable to hand-coding derivatives”. That is the whole argument for the middle column at scale: n evaluations against one pass at under six times the cost of evaluating f.

Use numerical checks as diagnostics, not as the primary training method. And read the sweep over h for what it is: a measurement of the differencing scheme rather than of the derivative. There is a published way to see the difference.

The complex-step approximation was introduced in 1998 by Squire and Trapp, whose abstract reads: “A method to approximate derivatives of real functions using complex variables which avoids the subtractive cancellation errors inherent in the classical derivative approximations is described.” Martins and two colleagues generalised it in 2003, reporting that “The complex-step method is tested in two large multidisciplinary solvers and the resulting sensitivities are compared to results given by finite differences. The resulting sensitivities are shown to be as accurate as the analyses.”

A third group states the property independently: “the Jacobian approximation is not subject to subtractive cancellations inherent in roundoff errors”, and “since an arbitrarily small step-size can be chosen, the complex-step method can achieve near analytical accuracy”.

If a different way of forming the estimate removes the cancellation term altogether, then the floor your sweep runs into is a property of subtraction, not of differentiation. That is why the five steps below vary h and compare relative error instead of trusting any single value.

Let p(z)=1/(1+e⁻ᶻ). At z=0, p=0.5 and p′=0.25.