Pipelines, Baselines, and Preprocessing Boundaries

One baseline rarely answers every question.

The top rung of that ladder has been measured. In 2014 Fernández-Delgado and colleagues ran “179 classifiers arising from 17 families”, implemented in Weka, R, C and Matlab. The test bed was “121 data sets, which represent the whole UCI data base (excluding the large-scale problems) and other own real problems”. The winner was a random forest, reached through R’s caret package. It “achieves 94.1% of the maximum accuracy overcoming 90% in the 84.3% of the data sets”. The runner-up was a Gaussian-kernel support vector machine in C, using LibSVM. That one “achieves 92.3% of the maximum accuracy”. The difference between the two was not statistically significant.

A strong classical baseline is not a courtesy paid to the reader. It is the number a new method has to beat.

Every data-dependent step belongs inside resampling and must be refit on each training fold.

That boundary is not a workshop convention. It is written into medical-device guidance. On 27 October 2021 three regulators — the US FDA, Health Canada and the UK MHRA — jointly issued ten Good Machine Learning Practice guiding principles for medical device development. The fourth is titled “Training Data Sets Are Independent of Test Sets”, and it reads as a requirement rather than a suggestion: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition and site factors are considered and addressed to assure independence.”

The second sentence is the one that costs work. Independence is not obtained by drawing a random split. It is obtained by naming every channel through which a held-out row could already be represented in the training data — the same patient, the same acquisition, the same site — and closing each one before a single transformation is fitted. A regulator deciding whether a device is safe does not accept a pipeline whose boundary was assumed. Neither should the person who built it.

Distance and margin methods are sensitive to feature scale. Trees usually do not need standardization, but they still depend on missing-value handling, category representation and valid splits.

Linear models can use one-hot, ordinal, spline or target-derived features. Each encoding changes the function the model can express.

Leakage enters through respectable preprocessing, not through an obvious target column. The scale of it has been counted rather than guessed at. Kapoor and Narayanan surveyed the fields that had taken up machine learning and published the audit in Patterns in 2023: “Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions.” They sort the failures into eight distinct types. This is not a beginner’s mistake being described. It is 294 published papers across 17 fields, each of which passed review.

One of those channels has been measured on data that contained nothing at all. Vandewiele and colleagues drew 10,000 five-dimensional points from a uniform distribution and labelled them at random. Then they applied over-sampling before partitioning. Their 2021 paper reports what came out: “Nevertheless, due to applying the Synthetic Minority over-sampling Technique (SMOTE) before data partitioning, an Area Under the receiver operating characteristic Curve (AUC) of 0.95 on a ‘held-out’ testing set can be achieved. This in contrast to an AUC of 0.49 and 0.48 when applying no SMOTE or SMOTE after data partitioning respectively, which closely resembles random guessing.”

On real data the same misplacement carries a price you can quote. Demircioğlu tested four over-sampling methods on 15 public radiomic datasets in 2024. The biases reached 0.34 in AUC, 0.33 in sensitivity, 0.31 in specificity and 0.37 in balanced accuracy.

0.95 against 0.49 in the same coin-flip data is the number to carry out of this section. Every item below is a different route to it.

The aim is not maximum accuracy. It is a result you can believe.

Step 2 is the one usually settled by convenience, and its cost has been measured. Deep-learning models can identify the tissue-submitting site of a slide in The Cancer Genome Atlas. Across more than 3,000 patients and six cancer subtypes, they do it with AUROC between 0.964 and 0.998. Standard colour normalisation and augmentation do not remove the signal. Howard and colleagues showed that in Nature Communications in 2021, and priced it: enforcing site-preserved cross-validation lowered AUROC by 0.069 on average, with 91.1% of predictable features declining. Their abstract states the consequence: “These site-specific signatures can lead to overoptimistic estimates of model performance, and we propose a quadratic programming method that abrogates this bias by ensuring models are not trained and validated on samples from the same site.”

The same thing had turned up three years earlier in a different modality. Zech and colleagues reported in PLOS Medicine in 2018 that convolutional networks identified the hospital system of a chest radiograph for 99.95% of NIH images and 99.98% of Mount Sinai images. A model with an internal AUC of 0.931 fell to 0.815 externally. A split chosen at random hands the model a shortcut and then congratulates it for taking one.

A control can carry a p-value instead of an impression. Ojala and Garriga set out two permutation tests in 2010. The first asks “whether the classifier has found a real class structure in the data”. It estimates its null distribution “by permuting the labels in the data”. The second asks “whether the classifier is exploiting the dependency between the features in classification”. It permutes the features within classes instead, which leaves each class’s marginal distributions intact. The two tests name two different failures. One is no signal at all. The other is a model that never used the structure you believed it used. A single accuracy figure cannot tell them apart.

Shuffle the labels and confirm that performance collapses. Remove a suspicious feature and measure the change. Train on a tiny subset and see whether the pipeline can overfit it when the model has enough capacity.

A workflow that cannot detect deliberately broken data is not ready to certify an improvement.