Ensemble Diversity and Error Correlation

When several regression models have similar error variance and their errors are weakly correlated, averaging can reduce the variance of the final prediction, because positive and negative deviations partly cancel. Krogh and Vedelsby made that cancellation exact rather than suggestive. Their ambiguity decomposition, published in 1994, splits the ensemble's generalisation error into two terms, E = Ebar - Abar: the weighted average of the members' own errors, minus the ensemble ambiguity — the amount by which the members deviate from the ensemble's own answer. The minus sign carries two consequences. The ensemble can never be worse than its average member. And what it gains is exactly the amount its members disagree. The authors state the corollary without hedging: “A combination of the output of several networks (or other predictors) is only useful if they disagree on some inputs. Clearly, there is no more information to be gained from a million identical networks than there is from just one of them”.

The equivalent bias-variance-covariance form covers the squared error of the ensemble mean, and it is the one to read before ordering more hardware: E{(fbar-t)^2} = bias^2 + (1/M)var + (1 - 1/M)covar. Brown and two colleagues published it in 2005. Adding members divides the variance term by M. It does nothing to bias, and nothing to covariance, whose factor (1 - 1/M) only grows towards 1 as M grows. Member count is a lever on one of three terms. Architecture diversity, feature diversity, resampled data, different random seeds and different inductive biases all aim at the term the count cannot touch. They create alternative paths to the prediction, which is the only way the covariance term moves. If every model learned the same spurious relationship, aggregation leaves that term intact and can make the mistake look more confident.

For classification, averaging calibrated probabilities usually preserves more information than majority voting. Voting discards confidence and can behave oddly when models are differently calibrated. The combination rule should match the output semantics and the decision threshold used downstream.

This layered view separates properties that are often collapsed into the single word “diversity.” Each layer has a citation behind it rather than an intuition.

The first two layers are the two words of Dietterich's condition: the classifiers must be “accurate and diverse”, a requirement he credits to Hansen & Salamon (1990). Drop either word and the benefit is gone. The third layer is where Krogh and Vedelsby's ambiguity term is either preserved or thrown away. An aggregator that reduces each member to a hard class discards the very deviations the decomposition is counting.

Different diversity mechanisms change both the expected benefit and the operational cost, and Leo Breiman turned that trade into a bound. His random forests paper, in 2001, proves an upper bound on forest generalisation error, PE* <= rho-bar(1 - s^2)/s^2, where rho-bar is the mean correlation between member raw margins and s is member strength. Read as an instruction rather than a result, it is the design rule for every mechanism in the table below: “To improve accuracy, the randomness injected has to minimize the correlation ρ̄ while maintaining strength.” Both halves are load-bearing. Correlation sits in the numerator, so decorrelation improves the bound. Strength sits in the denominator, so a mechanism that decorrelates by making members weaker moves the bound the wrong way faster than it helps.

The library most readers will actually run states the same mechanism in its own documentation. Scikit-learn says the randomness it injects yields trees with “somewhat decoupled prediction errors”, so that on averaging “some errors can cancel out”. Bootstrap sampling and feature subsampling are not folklore attached to trees. They are correlation-reduction devices aimed at rho-bar. ECMWF's 50 altered members and the 30 perturbed members of GEFS version 12 are the same device at national scale: perturb the inputs and the physics, and keep every member strong enough to stand alone.

A useful ensemble audit looks beyond the aggregate score. It must also not stop at a diversity number, because the study that measured those numbers most carefully found they do not predict what they are used to predict. Kuncheva and Whitaker took ten diversity statistics in 2003 — four pairwise (Q, correlation, disagreement, double fault) and six non-pairwise — and computed them on 14,700 three-classifier teams enumerated from partitions of the 10 breast-cancer features. The rank correlations between those statistics and the improvement over the single best classifier ran from -21% to 38%. Their conclusion is blunt: “The absolute value of the correlation coefficients in Table 7 are so low that the diversity measures considered here have no useful predictive value.” So measure these things to understand your ensemble. Measure gain itself to decide anything.

Use this small experiment to learn whether combination is helping for the right reason. Two published results should set your expectations before you start, so that the lab confirms or contradicts something rather than merely producing numbers.

First, the diversity statistic is not the objective. Across 14,700 teams, Kuncheva and Whitaker found rank correlations with actual improvement running only from -21% to 38%. Step 3 exists to explain step 5, not to replace it. Second, the size curve flattens early. Ovadia's size sweep reports “diminishing returns beyond size 5”, the main experiments use M = 10, and the largest ensembles it plots are 8-16 members. If your fourth model is still buying obvious gains, that is a finding about your members' correlation, and worth understanding before you buy a fifth.