ReuScope: Auditing Data Reuse and Evaluation Sensitivity in Capsule Vision 2024

AI summary

Overview

Research area: Machine learning evaluation methodology — specifically data-leakage auditing, benchmark integrity, and medical-imaging dataset forensics, applied to the Capsule Vision 2024 (CV2024) challenge.

Technical level: Intermediate. The paper is readable with basic familiarity with classification metrics (balanced accuracy, AUC), train/validation splits, and confidence intervals, but it is dense with paired statistical contrasts and multi-condition experimental design.

Scope: A single-benchmark audit of the CV2024 capsule-endoscopy dataset that measures how files, videos and near-duplicates are shared across the public train/validation boundary, and demonstrates that the performance effect of removing a data source reverses depending on which validation population is used — from which the authors derive ReuScope, a four-step audit protocol.

What This Paper Is About

Public ML benchmarks assume that training and validation data are independent, but in medical video datasets the same frames, the same source videos, and even byte-identical files can appear on both sides of the split, inflating scores in ways that are hard to attribute. The authors take Capsule Vision 2024 as a case study and ask a sharper question than "is there leakage?": when you remove a suspected source from training, does the resulting score change keep its direction across different evaluation populations? Combining a file-level census, matched training controls, fixed-prediction re-scoring of submitted entries, and a local-removal experiment, they show that observed data reuse and measured score sensitivity are distinct findings — and they package the resulting comparison and reporting rules into a protocol called ReuScope.

Key Contributions

1. A public-file census of cross-split data relationships in CV2024, reporting byte-identical file groups (216 crossing groups), shared source videos (all 41 contributing Kvasir-Capsule prefixes appear in both partitions), and perceptual-similarity screening, each with its own counting unit and denominator.

2. A demonstration that a training contrast reverses across evaluation populations. Removing all Kvasir-Capsule training rows plus five flagged rows from other sources lowers public-validation balanced accuracy by 21.32 percentage points but raises balanced accuracy by 1.38 points on the non-Kvasir validation rows — a sign reversal the authors attribute to evaluation-population sensitivity rather than to a quantified amount of leakage-induced inflation.

3. Matched control pools that isolate alternative explanations: Source-only vs. Random (equal row count and nominal update count), Comp and Class-match-R (equal class counts), and pool A (66 Kvasir Ulcer images swapped for 66 AIIMS Ulcer images), showing that neither sample size nor class counts alone explains the observed score differences.

4. ReuScope, a four-step audit protocol (define boundary and evidence unit; measure each data relationship separately; separate training changes from evaluation changes; report each conclusion with its scope and uncertainty), together with an exploratory local-versus-distant removal experiment and released code, input lists, saved predictions, and numerical records.

Main Findings

• Byte-identical files cross the public training–validation boundary in 216 groups: 210 SEE-AI groups involving 215 validation files and six AIIMS groups involving six files. A filename-only check would miss 45 of them (39 SEE-AI groups and all six AIIMS groups) because basenames differ across the boundary.

• Identical bytes do not imply identical labels. 177 SEE-AI groups have disjoint released label sets across the boundary, while 33 SEE-AI groups and all six AIIMS groups share a released label.

• Video overlap exists even where no files are identical. No byte-identical files cross the boundary within KVASIR, yet all 41 contributing Kvasir-Capsule video prefixes occur in both partitions; the nearest same-video training frame to each validation row has a median frame-index gap of one (interquartile range 1–1).

• The matching rule changes the measured relationship. Exact-pHash matching links 1,381 of 11,581 KVASIR validation rows (11.92%) to same-source training rows in 6,285 pairs; 540 rows reach NCC ≥ 0.99 when each row is paired with its lexicographically first matching training row, but 799 do when any matching training row is accepted — equivalent to 4.66% vs. 6.90% of KVASIR validation rows.

• Preprocessing choices move similarity counts. On the same 1,381 pairs, 256×256 bilinear grayscale processing yields 387 NCC-positive pairs versus 540 at 96×96 bicubic.

• Similarity screens are not calibrated duplicate detectors. The 1,381 selected pairs have median SSIM 0.880 versus 0.647 for unflagged adjacent same-video pairs (95% interval [0.186, 0.285] on the difference of medians), yet 68 adjacent comparison pairs also reach NCC 0.99 with unknown duplicate status. Screening CV2024 KVASIR against upstream Kvasir-Capsule, 92.31% of 38,592 selected pairs reach NCC 0.99 and mean DINOv2 cosine similarity is 0.9658 versus 0.5376 for random pairs, but only 36.17% pass at NCC 0.99 when the larger pHash/dHash distance of the pair is 3–6.

• The headline sign reversal: Removal-minus-Full balanced accuracy is −0.2132 (95% paired-t interval [−0.2200, −0.2065]) on public validation, +0.0138 ([+0.0043, +0.0234]) on non-KVASIR validation, and +0.0051 on internal-filtered validation with an interval that includes zero. In percentage points: 82.53% → 61.20% on public validation, 74.48% → 75.86% on non-KVASIR, 77.28% → 77.79% on internal-filtered.

• Sample size alone does not explain the drop. Random and Source-only both contain 10,596 rows with equal nominal update counts, yet reach public balanced accuracy 0.779 vs. 0.611; the Source-only-minus-Random contrast is −0.1682 on public validation and +0.0624 on non-KVASIR.

• Class counts alone do not explain it either. Comp and Class-match-R preserve class counts but are 63% KVASIR rows; they reach public balanced accuracy 0.734 and 0.738 but only 0.388 and 0.391 on non-KVASIR, versus 0.745 for Full. Pool A's 66-image Ulcer swap changes public balanced accuracy by −0.069 ([−0.0752, −0.0628]) while non-KVASIR rises by 0.098 ([+0.0898, +0.1057]).

• The reversal generalizes beyond one architecture. Frozen ResNet-50 and ConvNeXt-Tiny with nonlinear heads show Source-only-minus-Random contrasts of −0.200 and −0.210 on public validation and positive values on non-KVASIR.

• Effects are source- and class-specific. Removal-minus-Full balanced accuracy is −0.473 for KVASIR, +0.013 for SEE-AI, −0.044 for KID, and +0.021 for AIIMS, covering 9, 8, 5 and 2 classes respectively (not additive components of pooled balanced accuracy). Public-validation Ulcer recall falls from 0.976 to 0.096.

• Fixed-prediction diagnostics corroborate population sensitivity. Of 27 released workbooks, 25 are scoreable and 24 agree with reported full-list scores (the primary cohort). Their mean organizer-score change on non-KVASIR is −0.050 with 19 entries changing rank; equal-source averaging gives −0.020. Spearman correlation with official-test combined scores falls from 0.557 on public validation to 0.459 on non-KVASIR.

• Score components can diverge on the official test. Source-only training raises mean official-test AUC by 0.0356 (95% interval [+0.0309, +0.0403], all ten seeds) relative to Full, while the balanced-accuracy difference of −0.0117 ([−0.0259, +0.0026]) includes zero. Official-test balanced accuracy is 0.2804 for Full, 0.2786 for Random, 0.2688 for Source-only.

• Local removal lowers confidence more than distant removal, but only in an exploratory sense. With 588 exposed and 588 matched-unflagged rows, the difference-in-differences in true-class probability is −0.355 percentage points (95% interval [−0.486, −0.224]), negative in all ten seeds; the NLL contrast is +0.00495 and the probability-margin contrast is −0.00472. Balanced-accuracy and accuracy intervals span zero, and the authors state the design was chosen with outcome access and that intervals are unadjusted for selection.

• Missing metadata is not a measured zero. Cross-split video overlap is established, but cross-recording patient identifiers are unavailable, so unique shared patients cannot be counted. The organizers confirmed the 70–30 split was made within abnormality-wise source folders without video or case grouping, that duplicate control removed one of two visually similar sequential frames without cryptographic checks, and that they hold no patient- or examination-level labels for Kvasir-Capsule frames.

Methodology in Plain English

The authors treat the benchmark as a scene to be investigated rather than a model to be improved.

Step one — fingerprint the files. They compute SHA-256 hashes of every released public image so that identical bytes can be identified regardless of filename, then confirm matches byte for byte. They read video-prefix identifiers from filenames to see whether the same source recording appears on both sides of the split. They add perceptual hashes (64-bit pHash and dHash) and pixel-level measures (normalized cross-correlation and SSIM) to screen for near-duplicates that are not byte-identical, explicitly varying the rule for which training image counts as a validation row's "counterpart."

Step two — change the training data while holding evaluation fixed. They train one pipeline (DINOv2-L/14 with a nonlinear head and rank-eight LoRA adapters on the final four blocks) across ten paired seeds and ten epochs, on six deliberately constructed training pools: everything (37,607 rows), non-KVASIR only (10,596), non-KVASIR minus five flagged rows (10,591), a size-matched random sample from the full pool (10,596), a class-count-matched pool, and a class-count-matched pool aligned to the removal pool. Because the reduced pools have fewer minibatches per epoch (82 vs. 293), the authors are careful to note that equal epochs do not mean equal update counts.

Step three — change the evaluation population while holding the fitted models fixed. Each model is scored on three views — public validation (16,132 rows), non-KVASIR validation (4,551), and an internal-filtered view (4,326) that drops rows whose basename or exact pHash matches a same-source training row. Separately, they re-score the participants' own submitted prediction files on a narrower population, which is the analogue of changing the view with predictions frozen.

Step four — probe local versus distant removal. For validation frames that have a close training match, they remove that frame's training neighbours (or an equal number of distant, matched training rows in three control arms) and compare the resulting true-class probability on exposed versus matched-unflagged frames, using a difference-in-differences form averaged over seeds.

Throughout, they report intervals that describe seed-to-seed variation conditional on fixed lists — not patient sampling — and they distinguish a census from a sample estimate.

Why This Matters

Impact on research. The paper's central methodological point is that detecting reuse and measuring its effect on a score are separate tasks that require separate comparisons. A benchmark score drop after source removal is routinely read as "leakage inflation"; this work shows that the same training contrast can go negative on one evaluation population and positive on another, so the drop measures sensitivity to source removal under a specified population, not an amount of inflation. It also shows that a filename-only duplicate check misses most byte-identical crossings in this release (45 of 216 groups), and that perceptual-hash screening rules with fixed thresholds still produce 387 versus 540 positives depending on resolution and interpolation — arguments for reporting predicates, counterpart rules and denominators alongside counts.

Real-world applications:

• Clinical AI regulation and device evaluation — auditors reviewing submissions can require that reported generalization metrics name the evaluation population and class support, since source composition alone can move balanced accuracy by tens of points.
• Benchmark and challenge organization — organisers can adopt split construction that groups by video and, where available, by patient, and can publish the grouping keys used.
• Hospital or vendor dataset curation — dataset maintainers can use the identifier and screening components to check for cross-split file identity before releasing derivative datasets.
• Systematic review and meta-analysis of medical imaging models — reviewers can ask which of the audited comparisons (fixed prediction vs. retrained pool) supports a claimed generalization result.

Industry relevance. Any team reporting a held-out score on a public medical imaging corpus faces the same issue: near-duplicate frames from the same video, shared acquisition sources, and mismatched class support across populations. ReuScope's requirement to state sample count, class/source support, update budget, endpoint and seed pairing is directly usable as an internal model-reporting checklist, and the released code and input lists make the comparisons reproducible.

Future Directions

• Patient-level quantification. Shared video recordings are established, but cross-recording patient identifiers are unavailable in the public metadata, so unique shared patients cannot be counted. Obtaining patient or examination linkage would let the overlap be expressed at the clinical unit rather than the file or video unit.

• Broader protocol validation. The audit covers one benchmark and one primary training pipeline; the frozen ResNet-50 and ConvNeXt-Tiny comparisons and the ChestX-ray14, ISIC 2019 and HyperKvasir checks each exercise only part of the protocol. Applying the full sequence of training and evaluation comparisons to other releases remains open.

• Calibrated duplicate detectors. The hash and joint screening rules are described as screening rules, not calibrated duplicate classifiers — the joint rule missed all tested 90% center crops in a 50-frame check. Better-validated screening predicates would tighten the link between measured data relationships and training interventions.

• Pre-specified removal experiments. The local-versus-distant comparison, its seed set, controls and endpoint were selected with access to results, and its intervals are unadjusted for selection. A pre-registered version, and one that can support a fixed-class interval under prefix resampling (only 914 of 10,000 prefix-bootstrap draws retained all six classes), would strengthen the probability-level finding.

Target Audience

Benchmark and challenge organisers, dataset curators, and machine-learning evaluation researchers — particularly those working with medical imaging and video-derived data, where near-duplicate frames and shared acquisition sources are common. It is also directly useful to regulatory or internal reviewers who need to interpret a reported held-out score, to practitioners of data-leakage auditing who want a concrete reporting template, and to competition participants who want to understand why a public-validation leaderboard may not track an official test. Readers need basic familiarity with classification metrics and confidence intervals; the paper's statistical contrasts are more demanding than its conceptual argument, which is stated plainly.

Authors’ abstract

We develop ReuScope, a protocol for auditing data reuse and evaluation sensitivity, using Capsule Vision 2024 as a case study. It combines measurements of data relationships with two comparisons: changing the evaluation population while holding predictions fixed, and changing training data while holding evaluation fixed. Byte-identical files cross the public training-validation boundary, and all 41 Kvasir-Capsule recordings that contribute frames occur in both partitions. Shared source recordings establish cross-split overlap; missing cross-recording patient identifiers prevent counting unique shared patients. Across ten paired seeds, removing all training rows attributed to Kvasir-Capsule, plus five rows from other sources flagged as near-matches to it, lowers public-validation balanced accuracy from 82.5% to 61.2%. Evaluating the same fitted models on the validation rows of the other sources reverses the comparison: removal improves balanced accuracy by 1.38 percentage points. The public-validation drop measures sensitivity to source removal, not an amount of leakage-induced inflation. Matched controls show that sample size and class counts alone do not explain the observed score differences. An exploratory experiment removes the training neighbors of validation frames that have close training matches, or equally many distant training rows, and yields a difference-in-differences estimate, relative to matched unflagged frames, of -0.355 percentage points in true-class probability; the paired-t interval for accuracy spans zero. The resulting procedure distinguishes observed reuse from conditional performance sensitivity and specifies the population, class support and uncertainty unit for each conclusion.