Accountable and uncertainty-aware evaluation of sensor-based AI under distribution shift: devices, subjects, and nearly three years underground

AI summary

Overview

Research area: Machine learning evaluation methodology, applied to sensor-based indoor localisation (geomagnetic fingerprinting) in underground environments.

Technical level: Intermediate. The core argument is accessible, but the paper leans heavily on measurement-science concepts (uncertainty budgets, guard bands, reproducibility conditions) and on statistical procedures that assume some familiarity with distribution shift and bootstrap inference.

Scope: The paper proposes a staged, uncertainty-aware evaluation protocol for deployed sensor-based AI and demonstrates it on smartphone-based LSTM classifiers operating across two underground mines, including a 34-month re-evaluation of unchanged models.

What This Paper Is About

Machine-learned localisation systems are almost always evaluated by splitting a single recording campaign into training and test data, which silently assumes the deployment will use the same devices, the same people, and the same time period as training. In reality, devices are replaced, personnel change, and years pass. The authors argue that performance of a deployed model should be treated as a measurement — with declared reference levels and a quantified uncertainty — and build a four-stage protocol that holds out devices, subjects, and time one axis at a time.

Key Contributions

1. A staged evaluation protocol with declared reference levels (chance lines using the correct class count, an out-of-present-scope rate, and the random split demoted to a declared upper bound) and a quantile-based decision rule that ties roll-out decisions to the 5% quantile of repeated trainings rather than the mean.

2. Quantification of the random split's optimism on 112 matched configurations: +0.14 macro precision under device change and +0.41 under subject change — together with the mechanism (99.86% window overlap at the operating point) and a reporting rule requiring every random-split figure to state its overlap.

3. A cross-time evaluation over 34 months, on a device generation released after training and with a surveyor absent from training, decomposed into class loss, annotation-boundary ambiguity, and a bounded residual of genuine signature change, guarded by an out-of-present-scope (OOPS) rate.

4. A three-step one-shot risk assessment over repeated trainings that produces deployment-relevant lower bounds and shows, on a bimodal configuration, that mean-based reporting can invert the deployment verdict.

5. Representation findings with protocol relevance: augmenting the model input with device attitude (roll, pitch, yaw) hurts performance under shift, and the ranking of representations obtained from a random split does not predict their ranking under escalated generalisation demands.

Main Findings

• The random split massively overstates performance. On 112 configuration-matched triples in the historic mine, median macro precision was 0.992 under the random split, 0.797 under device change, and 0.532 under subject change. Paired per configuration, the random split exceeded cross-device by a median of +0.143 (95% CI [0.114, 0.163]) and cross-subject by +0.413 (CI [0.336, 0.443]).

• The random split is nearly useless for model selection. Its ranking correlated only weakly with the deployment rankings (ρs = 0.33 cross-device, 0.30 cross-subject), and those weak correlations came entirely from a degenerate negative-control feature set. On the 83 configurations that converge above 0.95 in the random split, the association reverses (ρs = −0.56 and −0.65). The two structural protocols correlated with each other at ρs = 0.89.

• Window overlap is the leakage path. Consecutive windows of length L with step s share (L−s)/L samples. At L = 700, s = 1 the overlap is 99.86%, placing near-duplicates of most test windows into training. Macro precision increased with overlap (ρs = 0.39 in frame A), and matched configurations isolate the effect: raising the step from 2 to 20 cost the random split 0.042 precision, and raising it from 2 to 100 cost 0.256.

• Optima degrade across the stages, and the uranium mine replicates the pattern. Historic mine: 0.9974 (random) → 0.9544 (device) → 0.6942 (subject). Uranium mine: 0.9993 → 0.8258 → 0.6111 with n = 1152 runs per shift stage.

• After 34 months, unchanged models still beat chance on present classes. The 5% quantile of present-conditioned precision was 0.39 over 299 repeated trainings — 16.5 times chance — though it varies by ±0.08 across the composition of the 42 reachable classes, several times the spread between repeated runs.

• Means mislead on risk. A bimodal configuration passed a mean-based acceptance test decisively while its 5% quantile lay more than an order of magnitude below chance — exactly the weak single draw that matters for deployment.

• The task operates near the sensor noise floor. Central European total field is 48–50 µT; the local anomalies carrying position information span −0.72 to 1.21 µT, while datasheet error bounds of the smartphone magnetometers span ±0.1 to ±0.45 µT.

• Attitude-augmented inputs hurt under shift. Representations that append roll, pitch, and yaw performed worse under device, subject, and time change than the un-augmented equivalents.

Methodology in Plain English

The authors treat a deployed classifier the way a metrologist treats an instrument: define what is being measured, state the reference levels, and attach an uncertainty figure.

The setup. Walkers carry ordinary smartphones (attitude-fixed in a bracket reproducing one-handed posture) through two German underground mines — a historic silver adit and a former uranium mine. Each location marker plus walking direction defines a class (95 classes in the silver mine, 40 in the uranium mine). An LSTM classifies windows of the magnetic time series. Ground truth uses a leader–follower paradigm: one designated "primary" device is annotated live, and annotations propagate to co-carried devices by nearest-timestamp matching.

Four cumulative stages. Each stage holds out one more axis of reality:

1. Random split — training and validation windows from the same recordings. Entered only as a declared upper bound.
2. Cross-device — validation data from a phone that contributed no training data.
3. Cross-subject — validation data from a person absent from training, on a held-out device.
4. Cross-time — unchanged models re-run on recordings made 34 months later, on a phone released after the training campaign, by a held-out surveyor.

Reference levels and decision rule. Reference levels are chance with the correct class count (1/95, 1/40, or 1/42), plus an out-of-present-scope (OOPS) rate that exposes predictions silently directed at classes no longer present in deployment. A chance predictor on the 95 trained classes would put 53/95 = 0.558 of its predictions out of scope; the accepted ceiling is half that, 0.279, following a guard-band concept from measurement standards. Roll-out requires the 5% quantile of the repeat ensemble — not the mean — to exceed k times chance.

Uncertainty from repeats. Identical configurations are trained repeatedly, and the resulting spread is summarised by quantiles. In total the study draws on 7003 training runs and 1288 repeat runs across both mines, each logged with its full configuration and per-run timing under a deterministic hash.

Statistical procedures. Bootstrap percentile intervals from 20,000 draws; paired comparisons report both the share of positive pairs and the Wilcoxon signed-rank test, because the difference distributions are markedly asymmetric.

Disclosure. The authors flag a data quirk: two evaluation notebooks had multiplied certain macro-averaged metric columns by constant factors (0.8 and 0.7) before saving. All affected values are descaled in the paper, confirmed independently from stored confusion matrices.

Why This Matters

This paper argues that a single split of a single recording campaign cannot tell a deploying actor what a system will do in the field, and then supplies a practical instrument for producing that evidence. It bridges machine learning evaluation and measurement science, treating accuracy as a measured quantity that owes the reader a defined measurand, declared reference levels, and an uncertainty statement.

Impact on research: It gives the fingerprinting literature a way to report the under-reported factors — device generation, surveyor identity, recording epoch — that current practice leaves out, and it supplies a concrete, replication-ready template for other sensor modalities exposed to fleet ageing and personnel turnover.

Real-world applications:

• Mine rescue and emergency navigation — localising disoriented people underground where GPS and radio infrastructure are unavailable, using only the phone they already carry.
• Cave and tunnel survey operations — position tracking without installed beacons in environments where installation is impractical or forbidden.
• Device-fleet management — deciding when a deployed model must be retrained as phones are replaced, using quantile lower bounds rather than optimistic averages.
• Safety-critical classifier sign-off — any sensor-based AI whose roll-out decision should hinge on a weak single draw rather than an expected value.

Industry relevance: Organisations that field sensor-based AI (logistics, utilities, defence, mining, first response) routinely face the exact staleness problem the protocol models: trained models outliving the data that produced them. The measured optimism gaps show how badly a random split can mislead procurement and roll-out decisions, and the quantile criterion provides a decision rule that is defensible to a review forum.

Future Directions

• Bit-exact reproducibility. The archived training logs omit the optimiser, input normalisation, and a random seed. The repeat ensembles are therefore seed-free repetitions, which suits making training spread visible but blocks exact re-creation of individual runs.
• Longitudinal resources for the geomagnetic modality. Radio-based localisation has multi-month ageing datasets; the authors found no comparable longitudinal geomagnetic resource, and their own 34-month re-evaluation is a first of its kind. More such resources would let ageing behaviour be characterised rather than demonstrated.
• Larger ensembles for non-parametric lower bounds. The paper notes that a non-parametric lower bound for the 5% quantile only exists above a certain ensemble size, and it remains to be seen whether the criterion scales to applications where hundreds of repeat trainings are impractical.
• Extension beyond this case study. Whether the same staged protocol, OOPS rate, and guard-band discipline transfer cleanly to other sensor modalities and to sites where device and person cannot be separated (as happened in the uranium mine) is an open question the authors raise by construction.

Target Audience

The paper benefits researchers and practitioners who evaluate or deploy sensor-based machine learning: applied ML engineers working on localisation and time-series classification, indoor positioning and fingerprinting researchers, and measurement-science readers interested in how uncertainty evaluation extends to learned models. It is also directly useful to reliability and assurance engineers who must justify a roll-out decision to a reviewing body, and to methodologists studying data leakage and the reproducibility of benchmark claims. Readers looking for a purely algorithmic advance will find the contribution is evaluative rather than architectural.

Authors’ abstract

Sensor-based AI systems are rarely operated under the conditions under which they were trained: devices, personnel and recording epochs change, and each change degrades performance in ways a random train-test split cannot reveal. We propose a staged, accountable evaluation protocol that treats the evaluation of a deployed model as a measurement with declared reference levels and a quantified uncertainty. Four cumulative generalisation stages hold out devices, subjects and time. Each stage is judged on quantiles of repeated trainings against chance references with the correct class count, an out-of-present-scope rate exposes silent misdirection towards classes that are no longer present in deployment relative to training, and an explicit decision rule ties roll-out decisions not to means but to 5% quantiles. We demonstrate the protocol on infrastructure-free geomagnetic localisation with smartphone-based recurrent classifiers in two real underground mines, including a replication of the scheme's training stages at the second site. Unchanged models are re-evaluated on data recorded 34 months after the training campaign, on a device generation unknown at training time and with a held-out surveyor. The 5% quantile of their present-conditioned precision there is 0.39 over 299 repeated trainings, 16.5 times the chance level; across the composition of the 42 reachable location classes the figure varies by +/-0.08, several times the spread between repeated runs. Repeated trainings of a single configuration show why means mislead: a bimodal configuration passes a mean-based test decisively while its 5% quantile lies more than an order of magnitude below chance.