Kernel Ridge and Gaussian Process Regression

These are not two loosely similar methods. Under linked assumptions the curve is the same object twice. Kanagawa and colleagues state the identity outright in their 2018 review of how Gaussian processes and kernel methods connect: “It is widely known in machine learning that these two formalisms are closely related; for instance, the estimator of kernel ridge regression is identical to the posterior mean of Gaussian process regression.”

So the difference is not the fit. It is everything around the fit. scikit-learn's side-by-side example says what the probabilistic formulation adds: “However, the Gaussian process regressor also provide an uncertainty information that is not available with a kernel ridge.” Read the comparison below that way. Not two competing fits, but one fit plus a second moment, an explicit noise model, and hyperparameters that can be learned or selected.

Ridge regression penalizes coefficient magnitude in an explicit feature space. Kernel ridge does the same job through similarities to training examples, which buys nonlinear functions without ever constructing the features. Rasmussen and Williams gave the structural picture in 2006: “Another way to look at this equation is to see it as a linear combination of n kernel functions, each one centered on a training point”. Every training row keeps a seat at prediction time.

Unlike a sparse SVM solution, a standard kernel-ridge predictor commonly uses a contribution from every training row. That cost has a documented direction and a measured size. scikit-learn's user guide reports it in the comparison with support vector regression: “With regard to prediction time, SVR is faster than KernelRidge for all sizes of the training set because of the learned sparse solution.” In the library's own worked comparison, SVR keeps roughly a third of the 100 training points as support vectors. It predicts 100,000 targets more than three times faster. The dense solution is not a footnote about serving cost. It is a factor you can measure before you deploy.

The posterior conditions the prior on observed targets, through the kernel and the noise assumptions. The mean it returns is the kernel ridge estimate under the linked assumptions. The variance is the part only the probabilistic formulation supplies, and it is computed from the same chosen covariance and noise model.

A Gaussian process can express higher uncertainty away from observed inputs. That behaviour has been measured inside a national metrology institute's own instrument calibration. In 2022 Fowler and colleagues, at NIST, recast cryogenic microcalorimeter energy calibration as a GPR-justified approximating spline through x-ray lines of known energy. Across the 5.4-9 keV range populated by anchor points, the calibration uncertainty stays well below 0.5 eV. The shape of the uncertainty is exactly what the theory promises: “As expected, the uncertainty is smallest at the well-measured anchor points and grows with distance from the nearest one.”

The same paper shows what the variance is conditional on. Its Table 1 compares candidate calibration spaces. The median linear-interpolation error runs from 0.5 eV to 22.5 eV depending only on which space is assumed linear. That spread was produced by a modelling choice, not by the data. Variance is still computed under a chosen kernel, mean, noise model and fitted hyperparameters. Misspecification, unobserved regimes, selection bias and shift can each leave the reported interval confidently wrong.

These methods are most attractive when data is moderate, smoothness structure is meaningful, and uncertainty or interpolation matters.

Kernel choice, noise and scaling are model design, not defaults. Two of the steps below have numbers attached to them from the evidence in this lesson. Inspecting support and residuals means measuring empirical coverage against the nominal 95% level — the axis on which the large-spatial-data competition separated methods that MAE and RMSE could not tell apart. Budgeting matrix cost means knowing where the wall is. Storage and linear solves for n > 10,000 were prohibitive on a 2006 workstation, and lifting that took accessing the kernel matrix only through matrix multiplication and distributing those multiplies across GPUs.

Random Fourier features, Nyström methods, inducing points and local approximations all reduce cost the same way: they replace the full kernel calculation with a lower-rank or sparse structure. They trade exactness for speed. Both mean predictions and uncertainty can change, so audit the approximation error across dense, sparse and boundary regions.

The first of those substitutions was proposed at NIPS in 2007. Rahimi and Recht “propose to map the input data to a randomized low-dimensional feature space and then apply existing fast linear methods”, with the random features built “so that the inner products of the transformed data are approximately equal to those in the feature space of a user specified shift-invariant kernel”. scikit-learn ships it as RBFSampler, cites that paper, and states the trade-off in the open: “A higher n_components will result in a better approximation of the kernel”, and “For a given value of n_components RBFSampler is often less accurate as Nystroem”. The same page prices the alternative: “the computational complexity of the exact method is O(n_samples^3), the complexity of the approximation is O(n_components^2 · n_samples)”.

The approximation is a dial. The dial has a reading, and you can report it.

A linear, additive or tree model may perform comparably. It would then serve more cheaply and diagnose more clearly — and serving is not a rounding error here, since the dense kernel-ridge solution is slower to predict than a sparse one at every training-set size. Kernel regression earns its place when its similarity prior and uncertainty structure solve a measured problem.

The kernel ridge estimator is the GP posterior mean, so the real question is whether you need the second moment at all. If you do, the Heaton competition sets the standard to hold it to: report coverage, not just error. Model selection should compare the complete pipeline and the operational burden. That includes asking whether a modern alternative has since moved the number, as DTNN did on QM7 at 8.75 against kernel ridge's 10.22.