Advanced techniques
Stacking, Blending, and Weight Averaging
Build leak-free second-level ensembles and understand when simple averaging is safer than a learned meta-model.
By the end you can
- Construct out-of-fold features for a stacking meta-learner
- Distinguish stacking, blending, weighted averaging, and weight-space averaging
- Recognize leakage and overfitting in second-level models
- Choose a combination method that fits the diversity and data volume available
A combiner can learn model strengths—but only from honest predictions
Stacking treats model outputs as a new feature space. Several base learners make predictions. A meta-learner discovers how those predictions should be combined. A linear meta-model might learn that one classifier is reliable for common cases while another adds value on a narrow slice. A nonlinear combiner can express richer interactions. But it also overfits more easily.
The meta-learner has to train on honest predictions. If a base learner predicts its own training examples, those predictions are usually too optimistic. The meta-model then learns from a distorted view of model competence. Out-of-fold prediction solves this. Each training row receives predictions from a base model fitted on other folds. After the meta-learner is trained, the base learners are typically refit on all available development data for inference.
Stacking succeeds or fails on the integrity of the meta-training predictions.
Case
The stacking design that worked only with probabilities
What the base learners hand up to the combiner matters as much as which combiner you pick. Both halves of that design were tested early. In 1999, in the Journal of Artificial Intelligence Research, Kai Ming Ting and Ian Witten stacked C4.5, naive Bayes and IB1 over “two artificial datasets and eight real-world datasets from the UCI Repository”.
Passing class probabilities to the combiner beat passing hard class predictions. The abstract states that “best results are obtained when the higher-level model combines the confidence (and not just the predictions) of the lower-level ones”. The choice of combiner was even less forgiving. Of four candidate level-1 learners, one survived — “only the MLR algorithm is suitable for the level-1 generalizer, among the four algorithms used”. The paper states the design rule the rest of this lesson assumes: “for successful stacked generalization it is necessary to use output class probabilities rather than class predictions.”
The combiner needs to know how sure each model was. A verdict alone throws that away.
Visual
The out-of-fold stacking pipeline
The level-one dataset must be built from predictions that are out of sample for each row. That loop is not a teaching device. It is the entire design of the strongest tabular AutoML system yet measured.
Amazon Web Services released AutoGluon-Tabular in March 2020 and put the difference in the first line of the abstract: “Unlike existing AutoML frameworks that primarily focus on model/hyperparameter selection, AutoGluon-Tabular succeeds by ensembling multiple models and stacking them in multiple layers.” The same abstract reports what that choice bought: “In two popular Kaggle competitions, AutoGluon beat 99% of the participating data scientists after merely 4h of training on the raw data.”
The verdict from outside the team is what makes it usable evidence. The AutoML Benchmark, published in the Journal of Machine Learning Research in 2024, compared “9 well-known AutoML frameworks across 71 classification and 33 regression tasks” and found that AutoGluon consistently has the highest average rank. It also prices that rank: “The most accurate frameworks achieve higher model accuracy at a large cost to performance in terms of inference speed.” Bagged base models stacked in layers is what buys the accuracy. Serving all of them is what it costs.
- 1
Split development data
Create folds that respect time, groups, and dependencies.
- 2
Train base models per fold
Fit on all folds except the held-out fold.
- 3
Generate held-out predictions
Store probabilities or scores for the omitted rows.
- 4
Assemble meta-features
Join out-of-fold predictions, and optionally carefully selected original features.
- 5
Fit and validate the combiner
Evaluate the full stack using an outer validation protocol.
Every meta-feature must reflect how the base model behaves on unseen data.
Comparison
Four ways to combine trained models
Reach for a more complex combiner only when the simpler one leaves repeatable value on the table. The safe default here was measured rather than chosen by taste.
The original recipe constrains the weights. Leo Breiman built the level-one data by cross-validation in 1996 and then, as his abstract puts it, used “cross-validation data and least squares under non-negativity constraints to determine the coefficients in the combination”. The constraint is the whole point. An unconstrained least-squares combiner is exactly the flexible second-level model this lesson keeps warning about.
The numbers behind it are small and consistent. Breiman stacked the roughly 50 nested subtrees produced by a single CART run; only about 6.5 of them received a nonzero weight. Test-set prediction error fell from 20.9 to 19.0 on the 506-case Boston Housing data, and from 23.9 to 21.6 on the 330-case Ozone data. Level-one data generated by 10-fold cross-validation beat leave-one-out. His summary of the whole exercise is one sentence: “Stacking never does worse than selecting the single best predictor.”
In 2023 that claim was proved rather than measured. Three researchers at Princeton and the National University of Singapore restated the recipe — “The conventional approach uses cross-validation data to generate predictions from the constituent estimators, and least-squares with nonnegativity constraints to learn the combination weights” — and showed that the stacked estimator “has strictly smaller population risk than best single estimator among them”. Four numbers on two small datasets, and later a theorem. That is the whole warrant for preferring constrained weights to an unconstrained meta-model.
Unweighted average
Give every model the same contribution.
- Hard to overfit
- Strong default when models are similarly good
- Ignores slice-specific strengths
- Excellent sanity-check baseline
Weighted average
Tune fixed nonnegative or constrained weights.
- More flexible while remaining interpretable
- Weights can be optimized on validation data
- Selection bias grows with repeated tuning
- Works well with calibrated probabilities
Stacking
Learn a meta-model from out-of-fold base predictions.
- Can learn context-dependent combinations
- Requires leak-free cross-fitting
- Meta-model capacity must be controlled
- Useful for heterogeneous model libraries
Blending
Train the combiner on a dedicated holdout split.
- Simpler than full cross-fitting
- Sacrifices training data to the blend set
- Can be unstable on small datasets
- Operationally easy to explain
Example
When a learned combiner earns its complexity
Stacking is most defensible when the base models are complementary and the dataset supports a second learning stage. Two of the rows below are not hypotheticals. Forecasting and competition settings are the two largest public tests a learned combiner has been put through. Both are worth reading for what they cost as well as for what they won.
- Fraud scoring: A graph model captures account relationships while a tabular booster captures transaction attributes.
- Medical triage: An image model and a structured clinical model contribute different evidence, with strict patient-level folds.
- Forecasting: The M4 competition ran 61 methods over 100,000 series, and combinations took the field. Its organisers reported in 2018 that “Out Of the 17 most accurate methods, 12 were "combinations" of mostly statistical approaches.” The runner-up was itself a learned combiner. FFORMA, built at Monash University, explains in its abstract that “we use a collection of time series to train a meta-model for assigning weights to various possible forecasting methods”, and that “The approach achieved second position in the M4 competition.” The organisers describe the same entry as “a combination of seven statistical methods and one ML one, with the weights for the averaging being calculated by a ML algorithm”. The meta-model chose the weights from extracted time-series features, not from the leaderboard.
- Search ranking: Lexical, embedding, and behavioral rankers feed a constrained reranker trained on held-out queries.
- Competition settings: The Netflix Prize ran on 480,189 users, 17,770 movies and over 100 million ratings, and its winning solutions were blends no production system would accept. The BellKor team wrote in IEEE Spectrum in 2009 that “the solution we had just submitted on 1 October 2007 was a linear combination of 107 separate sets of predictions”, and the Grand Prize documentation records that “The whole ensemble of predictors from BellKor's Pragmatic Chaos is blended linearly in the end.” The same report prices the tail of that blend: “With 18 results we breach the 10% (RMSE 0.8563) barrier. Within these results there are 11 nonlinear probe blends and 7 unblended predictors.” Eighteen components crossed the prize threshold. The other eighty-nine bought the margin. Stack in a competition if you like, provided the leaderboard is never the meta-training set — and be clear which part of the blend you would actually deploy.
Key idea
The meta-model must not see polished fiction
Leakage can enter at several levels: base predictions made on training rows, preprocessing fitted across folds, target encoding computed globally, or model selection performed repeatedly on the same outer holdout. A technically correct out-of-fold loop does not rescue upstream leakage.
The last of those has been measured on data containing no signal at all. In 2006 Varma and Simon simulated “at least 1000 sets of 40 samples, of which 20 belonged to class 1 and the remaining 20 to class 2”, each sample “a vector of 6000 features”, with no true difference between the classes. They tuned classifiers by cross-validation and then reported that same cross-validation error. On data where no classifier can beat chance, the reported error looked respectable: “For SVM with optimal parameters the estimated error rate was less than 30% on 38% of "null" data-sets.” For shrunken centroids the same sub-30% error appeared on 18.5% of the null datasets. True error on independent test data was no better than chance in every case.
Their remedy is the nested loop: an inner loop for tuning, an outer loop for estimating error. In their words, “Proper use of CV for estimating true error of a classifier developed using a well defined algorithm requires that all steps of the algorithm, including classifier parameter tuning, be repeated in each CV loop.” Cawley and Talbot reached the same conclusion outside bioinformatics in 2010: “the degradation in performance due to over-fitting the model selection criterion can be surprisingly large”.
A stack multiplies the opportunities for exactly that error. The base models, the meta-model and the model library are each selected against something. So the meta-learner should also be judged against the best base model and a plain average. A tiny gain that disappears across seeds or time windows rarely justifies a second training and serving layer.
Cross-fitting protects only the boundaries you actually implement.
Key idea
The meta-model can overfit a small second-level dataset
Out-of-fold predictions may contain only one row per original training example. The second-level dataset is therefore small. The failure that invites has been measured twice, eleven years apart.
The extreme case was built at Cornell in 2004, by Rich Caruana and three colleagues: “We train about 2000 models for each problem.” They then hill-climbed a combination on a 1,000-point validation split, with 4,000 points for training and 20,000 held back for test. The greedy combiner “sometimes overfits to the hillclimbing set, reducing ensemble performance”, and the mechanism scales the wrong way — “As the number of models in a library increases, the chances of finding combinations of models that overfit the hillclimbing set increases.” Their fix constrains the selection instead of enriching it. Bagging the selection over 20 bags: “On average, bagged ensemble selection reduces loss an additional 2.5% at p = 0.5”, which “is about a third of the total 8.7% benefit we see with ensemble selection”. A third of the entire gain came from restraining the combiner rather than from adding models.
Eleven years later the auto-sklearn team at Freiburg tried three combiners on held-out predictions — stacking, gradient-free numerical optimization, and Caruana's ensemble selection — and shipped the simplest, with an ensemble of size 50. “While we found both numerical optimization and stacking to overfit to the validation set and to be computationally costly, ensemble selection was fast and robust.” They also found that “simply building a uniformly weighted ensemble of the models found by Bayesian optimization does not work well”. So the lesson is not that the plain average always wins. It is that the learned second level is the first thing to overfit.
Start with averaging, constrained weights, or a regularized linear combiner. Add nonlinear capacity only when repeated folds show stable complementary structure and the final test remains untouched.
One combination skips the second level entirely. Mitchell Wortsman and colleagues averaged the weights of models fine-tuned with different hyperparameters, a recipe they named model soups in 2022, and note that “we may average many models without incurring any additional inference or memory costs”. Their greedy soup of CLIP ViT-B/32 runs reached 81.03% on ImageNet against 80.38% for the best single model, and they report a ViT-G model “which attains 90.94% top-1 accuracy on ImageNet”, a state of the art at the time. No meta-training set exists to overfit, because there is no second level.
The second level often has less trustworthy data than the base models, not more.
Steps
A release checklist for a production stack
Before deployment, verify both statistical and systems properties. Accuracy and serving cost are separate facts about the same stack. The AutoML Benchmark measured both at once: the highest average rank over 71 classification and 33 regression tasks, bought at a large cost in inference speed.
1. Reproduce the fold map
Store exactly which rows produced each out-of-fold prediction.
2. Freeze base output schemas
Define class order, calibration, missing values, and version identifiers.
3. Compare three baselines
Best single model, simple average, and learned stack.
4. Stress missing members
Decide what happens when one base model times out or changes version.
5. Monitor contribution drift
Track whether the meta-model still uses members as expected over time.
Steps
Audit the meta-model’s evidence trail
A stack is only as honest as the predictions used to train its second level. The nested discipline is the one Varma and Simon prescribe: an inner loop for every tuning decision, an outer loop for the number you report. Here it applies to a combiner instead of a classifier.
1. Freeze the outer split
Keep the final test outside base-model and meta-model selection.
2. Generate out-of-fold features
Every training row receives predictions from models that did not train on that row.
3. Repeat preprocessing inside folds
Encoders, scalers, selectors, and calibrators must obey the same boundary.
4. Compare simple combiners
Test averaging, constrained weights, and a regularized meta-model before adding flexibility.
5. Refit with a documented recipe
Train deployable base models only after the stacking design is locked.
6. Test missing-member behavior
Specify what happens when one model is unavailable or stale.
Key takeaways
- Stacking learns a second-level model from base-model predictions. Breiman's original recipe fitted the combiner by least squares under non-negativity constraints, and cut test error from 20.9 to 19.0 on the 506-case Boston Housing data.
- Meta-training predictions must be out of fold for every training row. AutoGluon-Tabular's multi-layer version of that loop beat 99% of the participating data scientists in two Kaggle competitions after 4h of training.
- A simple average is a mandatory baseline because it is robust and difficult to overfit. auto-sklearn kept ensemble selection over stacking precisely because stacking overfit the validation set.
- Blending spends a dedicated holdout on the combiner, and that holdout overfits as the library grows. With about 2000 models per problem, bagging the greedy selection recovered about a third of the total 8.7% benefit.
- Leakage in preprocessing or target encoding survives an otherwise correct stacking loop. Varma and Simon's signal-free null datasets returned an SVM error under 30% on 38% of them.
- Production stacks need contracts for base-model versions, missing outputs, latency, and drift. The most accurate frameworks in the AutoML Benchmark pay for their accuracy at a large cost in inference speed.