meval: A Statistical Toolbox for Fine-Grained Model Performance Analysis

AI summary

Overview

Research area: Machine learning model evaluation methodology, with a specific focus on medical imaging and fairness/bias auditing.

Technical level: Intermediate — the paper assumes familiarity with metrics such as AUROC, precision-recall curves, calibration error, confidence intervals and hypothesis testing, but explains the statistical reasoning clearly.

Scope: The paper introduces meval, an open-source Python toolbox for statistically rigorous, intersectional subgroup performance analysis of machine learning models, illustrated with two medical imaging case studies.

What This Paper Is About

Developers are increasingly expected to report model performance not just overall, but broken down by patient or recording attributes (gender, age, race, device, protocol, and combinations thereof). Doing this rigorously is hard: different subgroups have different sample sizes and base rates, many subgroups means a high risk of false-positive findings, and metrics like precision-recall curves and expected calibration error are not comparable across groups with different base rates or sample sizes. The paper presents a toolbox that bundles valid metrics, uncertainty quantification, and multiple-comparison-corrected statistical testing so that practitioners can answer the question: "Does this model work for every patient?"

Key Contributions

1. A modular, open-source Python toolbox (meval) that takes a pandas dataframe of predictions, ground truth and metadata — no model access required — and produces an interactive HTML report on subgroup performance with a single function call, while also returning raw results for custom analysis.

2. Implementation of base-rate-independent and debiased metrics, including the (partial) area under the precision-recall-gain curve (pAUPRG) based on Flach et al., and the debiased root mean squared calibration error (DRMSCE) from Petersen et al., alongside standard metrics such as (balanced) accuracy, AUROC, (balanced) Brier score, sensitivity and specificity, plus a blanket AverageMetric for per-recording averages such as the average Dice score.

3. Identification of a previously undescribed problem in the original AUPRG computation — when no decision threshold yields exactly rec = br, the point needed for integration may be ill-defined, especially with small sample sizes and strong class imbalance (br << 0.5) — addressed by providing a partial AUPRG integrated over [recG_min, 1].

4. A pragmatic statistical framework for intersectional analysis, combining a complementary-group testing strategy (rather than combinatorially many pairwise tests), permutation-based testing with studentization following DiCiccio et al., Holm–Bonferroni correction for multiple comparisons, analytical and bootstrap confidence intervals, and volcano-plot-inspired selection of the most "interesting" subgroups for visualization.

Main Findings

• Accuracy gaps can be base-rate artifacts: In the ISIC2020 skin lesion malignancy case study, classification accuracy differs significantly between subgroups, but this appears primarily to be a function of the respective subgroups' base rates — no statistically significant AUROC differences were found. The model was also very poorly calibrated overall.

• Racial specificity gap reproduced, but not an AUROC gap: In the MIMIC-CXR case study, the authors reproduce Seyyed-Kalantari et al.'s finding of a significant gap in 'No Finding' specificity between racial groups, yet observe no significant difference between these groups in terms of AUROC.

• Same threshold, different operating points: Comparison of ROC curves for white and black patients shows the overall curves (and the areas under them) are similar, but the racial groups sit at different ROC operating points (TPR/FPR) for the same decision threshold — a disparity invisible to aggregate AUROC.

• Best-checkpoint model performance (MIMIC-CXR): The retained DenseNet121 checkpoint achieved a macro-averaged validation AUROC of 0.84 and macro-averaged test AUROC of 0.79; the authors note the test set was deliberately constructed to be highly diverse and challenging, so a drop is expected.

• Existing toolboxes fall short: AIF360 and fairlearn do not provide many standard performance metrics, nor statistical testing methodology, nor stratified analyses of common performance curves (ROC, PR, calibration), so they do not meet the needs of comprehensive intersectional evaluation.

• Significance reporting convention: Throughout the figures, ns denotes not significant (p > 0.01), * denotes p ≤ 0.01, and ** denotes p ≤ 0.001.

Methodology in Plain English

The authors build a Python library around a single input requirement: a table of model predictions, ground truth labels, and metadata columns. From that, the toolbox computes performance metrics separately for each subgroup and for intersections of attributes.

To make comparisons fair, the toolbox prefers metrics that do not change simply because a subgroup has a different base rate — AUROC rather than AUPR or F1, and the debiased DRMSCE rather than the standard expected calibration error, which is known to depend on sample size. For each metric it also produces confidence intervals, either through analytical formulas (Fast DeLong's method for AUROC, with a custom Newcombe's method for groups of 50 samples or fewer and for perfectly separated AUROC values of 1.0, since DeLong is known to have poor coverage there; the Wilson score interval via statsmodels for ratio-based metrics) or through a percentile bootstrap, with stratified bootstrapping to avoid undefined metric values when both classes are needed.

For statistical testing, the authors avoid comparing every subgroup pair, which would require correcting for a huge number of tests. Instead, each subgroup is tested against its complementary group — for example, "gender = female and age < 25" versus "gender not female and age ≥ 25" — which sharply reduces the number of tests while still allowing exploratory analysis. The test itself is permutation-based with studentization, following DiCiccio et al., using analytical variance expressions wherever available instead of expensive repeated bootstrapping, and p-values are corrected with the Holm–Bonferroni method.

For intersectional analyses, the user sets a minimum group membership threshold and a maximum interaction level (how many attributes may be combined). The most "interesting" subgroups for display are selected by summing each group's rank on the p-value of its disparity and its rank on the magnitude of that disparity, an approach inspired by volcano plots.

Two case studies demonstrate the workflow. In the first, a ResNet50-V2 pretrained on ImageNet is fine-tuned for 25 epochs on the ISIC2020 skin lesion dataset (80% training / 20% evaluation split, with lesion leakage prevented using the lesion_id field); images are resized to 256×256, center-cropped to 224×224, and normalized with torchvision ImageNet parameters, using stochastic gradient descent with binary cross-entropy, a learning rate of 5 × 10⁻⁴, momentum 0.9, batch size 64, random flips and random color jitter. In the second, a DenseNet121 is fine-tuned for multilabel classification on MIMIC-CXR-JPG frontal (AP/PA) recordings; after discarding recordings so that only one per patient is kept (following Weng et al.) and dropping the 'support device', 'fracture' and 'pleural other' labels, 41,168 recordings remain. The test set is built by randomly sampling 35 positive instances for each of the 11 labels for each of the top-5 race groups, yielding 1,757 samples, with the remaining data split into 37,439 training (95%) and 1,972 validation (5%) samples with no patient overlap.

Why This Matters

Impact on research: The paper argues that stratified and intersectional performance analysis is becoming the accepted norm, but that existing evaluation methodology literature is largely aimed at aggregate evaluation, and that comparative model evaluation work assumes paired samples — a fundamentally different statistical setting. By packaging the methodology into a reusable, reproducible toolbox with published case-study code, the authors lower the barrier to doing this analysis correctly and reduce the risk of p-hacking and HARKing in fairness auditing.

Real-world applications:

• Clinical AI auditing: Hospitals and regulators could use the toolbox to check whether a diagnostic model performs acceptably for every patient cohort defined by demographics, scanner type, or protocol, not just on average.
• Medical device and software certification: The two case studies use models fine-tuned on radiology, which points to the tool's use case for post-market surveillance of imaging software at the point where conditions and thresholds are chosen.
• Fairness and compliance reporting: Organizations required to document performance disparities across protected groups could generate the interactive report directly from prediction tables, without needing access to the model internals.
• General model debugging: Because any metric expressed as an average over per-recording values is supported via AverageMetric, the approach extends to segmentation and other domains where Dice-like scores are the natural metric.

Industry relevance: The requirement that no model access is needed — only a dataframe of predictions and metadata — makes the toolbox practical for third-party auditors, external validation studies and teams that receive predictions from a vendor. The complementary-group testing strategy also directly addresses the practical cost problem of combinatorially many subgroup comparisons.

Future Directions

1. Deconfounding: The authors state that future work may include developing a deconfounding approach for disentangling the effects of different causal factors on model performance, citing prior work in that direction.

2. Broadening analytical scope: The current version supports analytical confidence intervals only for AUROC (DeLong/Newcombe) and ratio-based metrics (Wilson); extending analytical variance and interval expressions to more metrics would reduce reliance on bootstrapping and permutation testing.

3. Beyond the AUPRG fix: The paper flags a previously undescribed problem with the original AUPRG calculation and offers a partial AUPRG as a workaround; whether the full AUPRG can be made well-defined for small samples with strong class imbalance remains open.

4. Encouraging case studies in other fields: The authors express the hope that publishing the toolbox will inspire researchers to run case studies in different application areas, aiding the identification of model blind spots and unfair biases beyond the two medical imaging datasets examined here.

Target Audience

The paper is most useful to machine learning engineers, clinical AI researchers, regulatory and fairness auditors, and data scientists working with medical imaging or other domains where performance must be reported across patient subgroups. Practitioners who already have a predictions table and want statistically defensible subgroup analyses will benefit most, as will methodologists interested in the metric refinements (pAUPRG, DRMSCE) and the complementary-group testing strategy. Readers without prior exposure to confidence interval construction or multiple-comparison correction may need supplementary statistical background, as the paper assumes familiarity with those concepts.

Authors’ abstract

Analyzing machine learning model performance stratified by patient and recording properties is becoming the accepted norm and often yields crucial insights about important model failure modes. Performing such analyses in a statistically rigorous manner is non-trivial, however. Appropriate performance metrics must be selected that allow for valid comparisons between groups of different sample sizes and base rates; metric uncertainty must be determined and multiple comparisons be corrected for, in order to assess whether any observed differences may be purely due to chance; and in the case of intersectional analyses, mechanisms must be implemented to find the most `interesting' subgroups within combinatorially many subgroup combinations. We here present a statistical toolbox that addresses these challenges and enables practitioners to easily yet rigorously assess their models for potential subgroup performance disparities. While broadly applicable, the toolbox is specifically designed for medical imaging applications. The analyses provided by the toolbox are illustrated in two case studies, one in skin lesion malignancy classification on the ISIC2020 dataset and one in chest X-ray-based disease classification on the MIMIC-CXR dataset.