Cross-Validation, Nested Resampling, and Variance

Cross-validation reuses observations across training folds. The resulting scores reveal sensitivity to partitioning, but they are correlated because models and datasets overlap. The method is valuable when a single holdout would waste scarce data or produce a fragile estimate. It does not manufacture new independent evidence from one dataset.

Ask it to, and it answers anyway. In 2006 Varma and Simon generated at least 1,000 simulated "null" datasets — 40 samples, 20 per class, 6,000 features, and no real difference between the classes at all. There was nothing in that data to learn. They tuned a shrunken-centroid classifier by 10-fold cross-validation and an SVM by leave-one-out cross-validation, then read off the cross-validation error of the tuned model. The tuned shrunken centroid reported a mean error of 37.8%. The tuned SVM reported 41.7%. The true error, measured on 20,000 independently generated test samples, was 50.0% — mean biases of -12.2 and -8.3 percentage points on pure noise.

The tail of that distribution is what a single project would actually see. Their abstract reports it: “Even though there is no real difference between the two classes for the "null" datasets, the CV error estimate for the Shrunken Centroid with the optimal parameters was less than 30% on 18.5% of simulated training data-sets. For SVM with optimal parameters the estimated error rate was less than 30% on 38% of "null" data-sets.” That 38% belongs to the abstract alone. The paper's own Table 1, in the column “Percentage of times training error<30%”, gives 22.2% for the SVM row. The shrunken-centroid figure does match: 18.5%. The remedy is in the same paper — “A nested CV procedure provides an almost unbiased estimate of the true error”.

Cross-validation does not estimate the error of the model you fitted. It estimates the average error of models fitted to other training sets you never saw. Bates and colleagues proved that in 2024 for the linear model fit by ordinary least squares, and showed empirically that the same phenomenon holds for most popular error estimates, including data splitting, bootstrapping and Mallow's Cp. The five scores are not five worlds. They are not even five looks at your model.

Everything the pipeline learns must stay inside the training portion.

Different iterators protect different assumptions. The choice of iterator can move the headline number further than the choice of model.

Split a wearable-sensor dataset by record and the error reads 2%. Split the same data by subject and it reads 27%. The dataset is public: 30 subjects, 10,299 records, 561 features each. In 2017 Saeb and colleagues split it both ways. Record-wise, the cross-validation error was already 2% with only two subjects and barely moved as subjects were added. Subject-wise, with no subject appearing on both sides of a fold, the error started at 27% with two subjects and settled at about 7-9% as subjects and folds grew. Same data, same algorithms, an order of magnitude between the reported errors: “Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms.” Their systematic review found the record-wise method in almost half of the retrieved wearable and smartphone clinical-prediction studies.

The effect reproduces. In 2021 Tougui and colleagues ran the comparison independently on Parkinson's-disease smartphone audio. One SVM pipeline scored 73.54%, 73.75% and 73.53% accuracy under stratified k-fold, leave-one-out and repeated stratified k-fold cross-validation. Under the group-wise versions of the same three procedures it scored 62.26%, 62.68% and 63.13%. Their conclusion: “The record-wise division and the record-wise cross-validation techniques overestimated the performance of the classifiers and underestimated the classification error.”

Blocking generalises beyond time to any dependence that travels with position. A random forest was trained to predict central-African aboveground forest biomass from 9 MODIS and 27 environmental variables, on inventory data covering more than 190,000 plots and about 11.8 million measured trees. Classical random 10-fold cross-validation reported R2 = 0.53 and RMSPE 56.5 Mg/ha. A spatial 44-fold cross-validation built on clusters of 150 km gave R2 = 0.14 and RMSPE 77.5 Mg/ha — barely better than a no-predictor null model at 82 Mg/ha. Ploton and co-authors published that in Nature Communications in 2020. The model did not change between those two numbers. The fold design did. Their abstract puts it plainly: “A standard nonspatial validation method suggests that the model predicts more than half of the forest biomass variation, while spatial validation methods accounting for SAC reveal quasi-null predictive power.” Stock found the same direction independently in a marine remote-sensing case study in 2025: “Depending on the model and error measure, 10-fold cross-validation underestimated prediction errors by 5% (RMSE of MBR) to 54% (APD of RFXY).”

Hyperparameter search consumes validation evidence.

Choose the hyperparameters once on the whole design set, then report the cross-validation error, and the number comes back too good. In 2010 Cawley and Talbot measured how much. They estimated the error of kernel ridge regression with a Gaussian RBF kernel by 10-fold cross-validation on thirteen benchmark datasets, under two protocols. In the external protocol the hyperparameters were chosen once on the whole design set. In the internal protocol the selection was repeated inside every fold. Their verdict: “The external protocol clearly exhibits a consistently optimistic bias with respect to the more rigorous internal cross-validation protocol, over all thirteen benchmarks.”

Preserve out-of-fold predictions so you can inspect thresholds, calibration, subgroup behavior, and error concentration without averaging away structure.

When the final model is trained on all development data, its independent test result remains the cleanest release estimate, and cross-validation supports that decision without abolishing the need for protected evidence.

In regulated work that is not a preference but a published expectation. Ten Good Machine Learning Practice guiding principles for medical device development were published on 27 October 2021 by the US Food and Drug Administration, Health Canada and the UK Medicines and Healthcare products Regulatory Agency together. Principle 4 is “Training data sets are independent of test sets”, and it reads: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition and site factors are considered and addressed to assure independence.” Principle 8 requires performance evidence generated independently of the training data. The list in principle 4 — patient, data acquisition, site — is the grouped-and-blocked fold argument written out by three regulators. Independence has to be engineered into the split, one source of dependence at a time, and then evidenced on data the development loop never touched.