Ridge, Lasso, and Elastic Net

Unregularized least squares asks which coefficients fit the training outcomes best. Regularized regression asks for a fit that also keeps the coefficient vector within a preferred shape or size.

This preference can reduce variance, stabilize correlated features, or produce sparse solutions. It also introduces bias deliberately. The penalty must therefore be evaluated rather than treated as free protection.

Both penalties have a date and a stated purpose. Ridge regression arrived in Technometrics in February 1970, under a title that announced the bias: Ridge Regression: Biased Estimation for Nonorthogonal Problems. Hoerl and Kennard accepted that bias in exchange for stability when the predictor matrix is close to singular. The lasso came twenty-six years later, in 1996. Tibshirani returned to it in 2011 and gave the reason for the exponent. Penalties can be indexed by an exponent q. Ridge regression is q = 2, and subset selection is the limit as q → 0. “The lasso uses the smallest value of q (i.e. closest to subset selection) that yields a convex problem. Convexity is very attractive for computational purposes.” The shape of the constraint was chosen for tractability, and for a preference over coefficient vectors. It was not chosen as a remedy for the data.

All three use the same features and the same prediction form. Their coefficient geometry differs. Ridge penalizes the squared magnitude of the coefficients: it shrinks correlated effects together and rarely produces exact zeros. Lasso penalizes absolute magnitude, can set coefficients exactly to zero, and therefore performs selection as a side effect of fitting. Elastic net combines the two penalties and adds a mixing hyperparameter. Sparsity survives, and so do correlated blocks.

The loss contours meet different constraint shapes, and that creates distinct coefficient behavior. The data-fit contours collect coefficient vectors with similar training error. The L2 constraint is a smooth round region, so it encourages shared shrinkage. The L1 constraint is a diamond, and its corners sit on the coordinate axes. That is why solutions land on axes and coefficients become exactly zero. The elastic-net constraint blends corners with curvature.

A penalty acts on coefficient magnitude. That magnitude depends on feature units. A one-unit change in euros is not comparable with a one-unit change in millions of euros.

That is not a piece of advice floating free of any tool. It is a fork in the tooling, and the two reference implementations take opposite branches. glmnet does the standardising for you. Its reference manual for version 5.0, dated 8 May 2026, documents the `standardize` argument of glmnet() as “Logical flag for x variable standardization, prior to fitting the model sequence. The coefficients are always returned on the original scale. Default is standardize=TRUE.” Centring is not even offered there as a choice. The 2010 coordinate-descent paper behind the package says why in one line: “Irrespective of whether the variables are standardized to have variance 1, we always center each predictor variable.”

scikit-learn 1.9.0 hands the job back to the analyst. Ridge, Lasso and ElasticNet expose no standardize argument at all. The user guide states the requirement as an instruction rather than a default: “The feature matrix X should be standardized before fitting. This ensures that the penalty treats features equally.” The consequence is concrete. The same model, fitted to the same table, returns different coefficients under glmnet and under scikit-learn. That holds unless the person fitting it knows which library standardised and which did not.

Standardize numeric features inside the training pipeline when equal penalty treatment is intended. Binary indicators, ordered variables, and engineered features may require more deliberate scaling choices.

This has been measured on real correlated data, twice, in the same year. In 2005 Ein-Dor and colleagues re-ran van 't Veer's own gene-selection method on van 't Veer's own breast-cancer data. Their report in Bioinformatics is blunt: “We showed that, in fact, the resulting set of genes is not unique; it is strongly influenced by the subset of patients used for gene selection. Many equally predictive lists could have been produced from the same analysis.” Which genes came out was decided by which patients went in.

A second reanalysis went wider. In The Lancet, in the same year, Michiels and colleagues reanalysed the seven largest studies of this kind, using repeated random training sets. They found that “The list of genes identified as predictors of prognosis was highly unstable; molecular signatures strongly depended on the selection of patients in the training sets.” Their second finding is the harder one: “Five of the seven studies did not classify patients better than chance.” Correlated predictors carrying nearly the same signal do not produce one true short list. The estimator will not tell you so.

Regularization strength belongs inside the model-selection process. Each step has a published number or a published default behind it, not only a recommendation.

Step 1, the fold-aware pipeline. Selection performed once, on all the data, produces cross-validated errors near zero on data that carries no signal. Ambroise and McLachlan showed this on real published data in PNAS on 30 April 2002: “Using two published data sets, we demonstrate that when correction is made for the selection bias, the cross-validated error is no longer zero for a subset of only a few genes.” scikit-learn 1.9.0 ships the same demonstration on data built to contain nothing: 200 samples, 10,000 randomly generated features, labels assigned at random. Selection that sees the test data scores an accuracy of 0.76. The same selection done inside the folds scores 0.5. “Using all the data to perform feature selection results in an accuracy score much higher than chance, even though our targets are completely random.”

Step 2, the penalty path. “Search over orders of magnitude” has an actual shape, and two independent libraries ship it as their default. The coordinate-descent paper states the recipe: “Our strategy is to select a minimum value λmin = ϵλmax, and construct a sequence of K values of λ decreasing from λmax to λmin on the log scale. Typical values are ϵ = 0.001 and K = 100.” The glmnet manual for v5.0 gives nlambda = 100, with lambda.min.ratio 0.0001 when nobs > nvars and 0.01 when nobs < nvars. scikit-learn's LassoCV defaults to eps = 0.001 with 100 alphas: “Length of the path. eps=1e-3 means that alpha_min / alpha_max = 1e-3.” A hundred values on a log scale, not a narrow linear grid.

Step 4, inspecting coefficient paths, is affordable, because the whole path costs about what one fit costs. The Least Angle Regression paper made the claim in its own abstract in 2004: “LARS and its variants are computationally efficient: the paper describes a publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates.” scikit-learn documents both halves for its own implementation — “It is computationally just as fast as forward selection and has the same order of complexity as ordinary least squares.” and “It produces a full piecewise linear solution path, which is useful in cross-validation or similar attempts to tune the model.” On the leukaemia data the full 100-value path “took under a second in total”. The path, not a single fit, is the object to look at.

Steps 3 and 5, the deployment metric and refitting only after selection. The score at the chosen penalty cannot double as the reported performance. Varma and Simon generated data sets with no difference between the classes at all. They reported in BMC Bioinformatics in 2006: “Even though there is no real difference between the two classes for the "null" datasets, the CV error estimate for the Shrunken Centroid with the optimal parameters was less than 30% on 18.5% of simulated training data-sets.” For SVMs the same thing happened on 38% of them. Nested cross-validation gave an almost unbiased estimate. Cawley and Talbot report the same over-fitting of the selection criterion independently, in 2010: “we demonstrate that a low variance is at least as important, as a non-negligible variance introduces the potential for over-fitting in model selection as well as in training the model”.

Step 6 remains what it always was: compare coefficients and predictions across folds, seeds, or bootstrap samples, and report the spread.

A ridge model may generalize better while making individual coefficients harder to translate. That happens when features are transformed or correlated. Conversely, a sparse lasso can look interpretable and still change its selected set across folds. The seven studies reanalysed in The Lancet in 2005 produced highly unstable gene lists, and five of the seven did not classify patients better than chance.

Evaluate predictive performance and interpretive stability as separate objectives.