Skip to content
AI.info

Evaluation

Grouped, Temporal, Spatial, and Hierarchical Splits

Choose evaluation partitions that respect entities, time, geography, hierarchy, and deployment transfer rather than relying on random rows.

By the end you can

Key idea

The split is a simulation of deployment

A random split asks whether the model can predict another row drawn from the same mixture. That can be useful. Many deployments ask a harder question: a new patient, a later month, an unseen store, a different sensor generation.

Epic's proprietary sepsis prediction model reached hospitals with a vendor-reported area under the curve of 0.76–0.83. Then somebody else measured it. Wong and colleagues validated the model externally at Michigan Medicine, across 27,697 patients and 38,455 hospitalisations, and reported in JAMA Internal Medicine in 2021 “a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64)”. Of the 2,552 patients who had sepsis, 1,709 were never identified. That is 67% of them.

Ostermayer and colleagues then ran the same model across two county emergency departments and 145,885 encounters. In 2024 they reported “sensitivity of 14.7%, specificity of 95.3%, positive predictive value of 7.6%, and negative predictive value of 97.7%”.

The number a model is sold on and the number it earns at a site it has never seen are estimates of two different quantities. The split is what decides which one you compute. It should reproduce the intended transfer. If it does not, an excellent score may answer a scenario that never occurs.

A split is a model of the future.

Case

Forest biomass: R2 = 0.53 under random folds, R2 = 0.14 under spatial ones

A forest-biomass map of central Africa was built from an inventory of 11.8 million trees. Ploton and 12 co-authors validated the random forest behind it twice, and published both results in Nature Communications in 2020.

A classical random k-fold cross-validation returned R2 = 0.53, with a root mean squared prediction error of 56.5 Mg ha-1. They then grouped the same data into 44 spatially homogeneous clusters, each no more than 150 km across — slightly wider than the autocorrelation range of forest biomass. Those clusters took turns as training and test sets, so that no validation point sat inside the autocorrelation range of its training data. The same model on the same data collapsed to R2 = 0.14. Its error rose to 77.5 Mg ha-1, against the 82 Mg ha-1 of a model carrying no predictors at all.

Their own summary of the pair: “A standard nonspatial validation method suggests that the model predicts more than half of the forest biomass variation, while spatial validation methods accounting for SAC reveal quasi-null predictive power.”

Same model, same data, two verdicts. Only the choice of fold boundary separates them.

Visual

Four transfer questions

Each boundary removes a different source of familiarity, and the cases in this lesson sit on this map. Splitting wearable and smartphone data subject-wise rather than record-wise, as Saeb and colleagues did in 2017, probes grouped transfer. Ploton's 44 spatial folds probe spatial transfer. The three hospital systems in Zech's radiograph study probe domain transfer.

FigureHierarchy · 4 levels

  • Grouped transfer

    Can the model handle unseen entities such as users, patients, or devices?

    • Temporal transfer

      Can it work on later periods after behavior and prevalence change?

      • Spatial transfer

        Can it generalize to new locations with geographic correlation?

        • Domain transfer

          Can it operate across sites, vendors, languages, or acquisition protocols?

Comparison

Random rows versus deployment-aligned partitions

Neither is universally correct. They estimate different quantities. Ploton's random forest is a single object carrying two honest scores: R2 = 0.53 under random folds, R2 = 0.14 under 44 spatial ones. Neither figure is an error in arithmetic. They answer different questions about the same model. A report that gives one without naming which question it answers is the actual failure.

FigureComparison · 2 columns

Random row split

Samples rows from a common pooled distribution.

  • Efficient use of data
  • Often lower variance
  • Can leak entity signatures
  • Represents interpolation within the pool

Structured split

Holds out a meaningful group, period, area, or domain.

  • Tests a stated transfer
  • Often more variable
  • May expose coverage gaps
  • Better matches cold-start or future use

Example

Where random splitting breaks

The failure is often subtle, because no target column is copied. It is also measurable, and in at least one field it has been measured twice by different teams.

  • Wearables: windows from the same person appear in train and test, letting identity and device habits carry over. Saeb and colleagues put a size on it in 2017 — “Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms.” In their systematic review the median classification error was 13.00% for subject-wise papers against 5.60% for record-wise ones, and the optimistic method “was used by almost half of the retrieved studies”. Tougui and colleagues reproduced the effect independently in 2021. Their SVM pipeline scored 73.54%/73.75%/73.53% under record-wise cross-validation and 62.26%/62.68%/63.13% under subject-wise cross-validation. They summarised Saeb's finding this way: “the reported classification error of the subject-wise CV was more than twice that of the record-wise CV”.
  • Retail: Random transactions mix the same store promotions across partitions, hiding the challenge of launching at a new site.
  • Remote sensing: Adjacent image tiles share terrain, weather, and labels, so spatially close splits inflate accuracy.
  • Forecasting: Future values influence preprocessing or training when rows are shuffled across time.
  • Industrial inspection: Images from the same production batch share lighting and material conditions.

Analogy

Practicing on different pages from the same answer sheet

The second page carries different questions. The worked examples, the notation, and the hidden clues are the same ones. The source of familiarity has not moved.

Where the groups really are exchangeable, and deployment draws from the same mixture, a fresh row is a fresh situation and nothing is leaking. Where they are not, an unseen row can be a very familiar one: a new window from a subject already in training, a new plot inside a forest cluster already sampled, a new radiograph from a hospital whose equipment the network can name.

Novel rows can preserve familiar entities and environments.

Cold start deserves its own estimate

A recommender may perform well for established users and poorly for newcomers. Averaging those regimes hides what happens at launch.

Report warm-start and cold-start performance separately. Decide whether deployment will include new users, new items, or both. The split should preserve the information that would actually exist at first contact.

Cold-start evaluation is a data-availability test as much as a ranking test.

Steps

Design the boundary from the deployment sentence

Begin with the transfer claim, not with a library default. In one regulated field the first two steps below are not merely advisable. Guiding principle 4 of Good Machine Learning Practice for Medical Device Development, issued jointly by the US FDA, Health Canada and the UK MHRA on 27 October 2021, reads: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition and site factors are considered and addressed to assure independence.”

That sentence hands you the grouping keys rather than leaving you to invent them: the patient, the acquisition, the site. Wong's sepsis validation crossed the site boundary. Saeb's subject-wise folds cross the patient boundary. Zech's external comparison crossed both site and acquisition at once.

FigureProcess · 5 steps

  1. 1. Name the unfamiliarity

    New entity, later time, new location, new device, or new institution.

  2. 2. Identify the grouping key

    Choose patient, account, site, campaign, batch, region, or another independent unit.

  3. 3. Preserve chronology

    Fit transformations and labels using information available before the cutoff.

  4. 4. Check coverage

    Measure whether held-out groups contain enough positive, rare, and severe cases.

  5. 5. Report multiple regimes

    Separate interpolation, cold start, future transfer, and external-domain results when each matters.

Harder splits are not automatically more honest

A split can be so extreme that it tests a different product, or produces estimates too unstable to use. The right boundary is the one that mirrors the intended deployment and supports a decision.

The spatial holdout that demolished the biomass map is itself disputed in print. Wadoux and colleagues ran a numerical experiment on large-scale above-ground biomass mapping in 2021. For the spatial strategies, they write, “we applied the same computational methodology as in Ploton et al. (2020)”. What they found was that “In our experiment, standard cross-validation (i.e., ignoring autocorrelation) led to smaller bias than spatial cross-validation”. Their conclusion is blunt: “We conclude that spatial cross-validation methods have no theoretical underpinning and should not be used for assessing map accuracy, while standard cross-validation is deficient in case of clustered data.”

An independent group reached the opposite practical conclusion. Mahoney and colleagues simulated five cross-validation methods in 2023 and found that “spatial CV approaches generally improved upon resubstitution and V-fold CV estimates”. They also say explicitly that they do not expect their results necessarily to transfer to map accuracy assessments.

The two teams are not returning two verdicts on one experiment. One is grading map accuracy, the other model accuracy. The estimator that is right for one question is deficient for the other. That is the whole argument for writing the question down. Document why the split was chosen, what transfer it simulates, and which deployment conditions remain untested.

Evaluation difficulty should come from realism, not theatrical severity.

Case

The pneumonia model that had learned the hospital

Pneumonia CNNs were compared across three hospital systems by Zech and colleagues, in PLOS Medicine in 2018. Internal performance beat external performance in 3 out of 5 natural comparisons. The best pooled model scored AUC 0.931 (95% CI 0.927–0.936) inside its training sites, against 0.815 (95% CI 0.745–0.885) at Indiana University.

Then they trained a network to do nothing but name the source of an image. From the Results: “A CNN trained to identify hospital systems accurately identified 22,050 / 22,062 (99.95%, 95% CI 0.9991–0.9997) of NIH, 8,386 / 8,388 (99.98%, 95% CI 0.9991–1.0000) of MSH, and 737 / 771 (95.59%, 95% CI 0.9389–0.9693) of IU test radiographs.”

Pooling more sites into training did not repair the transfer. The models “performed better on new pooled data from these sites but not on external data”. Geirhos and colleagues later used the case as their example of shortcut learning, describing a classifier that “had unexpectedly learned to identify particular hospital systems with near-perfect accuracy” through a hospital-specific metal token. The hospital site had become a feature. Only a split along the site boundary could see it.

Key takeaways