Different Models, Shared Failures: Auditing Frozen-Encoder Pools for Medical Segmentation

AI summary

Overview

Research area: Medical image segmentation, with a focus on model evaluation, ensemble diversity, and reliability auditing of pretrained vision encoders.

Technical level: Intermediate. The abstract assumes familiarity with segmentation metrics (Dice), transfer learning with frozen encoders, and linear/lightweight readouts, but the core argument is conceptual rather than mathematical.

Scope (one sentence): The paper audits whether a pool of pretrained medical-segmentation models fails on the same cases rather than on different ones, and argues that pooled accuracy and model count hide this correlated failure.

What This Paper Is About

When researchers combine several segmentation models, they usually assume that more models means more robustness: if one model gets a case wrong, another will catch it. That assumption only holds if the models make different mistakes. This paper measures how often pretrained encoders fail on the same patients or images, and shows that a pool of nominally independent models can behave like a much smaller pool. The goal is to establish correlated failure — not just average accuracy — as something that must be measured and reported.

Key Contributions

1. An audit protocol for correlated failure. The authors introduce CoFailMed, a study design that evaluates pretrained encoders with frozen features and separately trained, deliberately simple (1×1) readouts, using multiple random seeds per encoder so that per-case behavior can be compared fairly across models.

2. An empirical audit across tasks and encoders. The study covers five 2D medical segmentation tasks and 11 public pretrained encoders, explicitly described as a controlled sample rather than a complete census, plus an expanded 17-encoder analysis of within-family versus cross-family dependence.

3. A threshold-specific "effective size" measure. Instead of reporting only how many models are in a pool, the paper estimates how many independent models the pool actually behaves like, using failure correlations at a pooled 33rd-percentile Dice threshold, restricted to pools with enough failures and passes per member to support the estimate.

4. A counterintuitive result about improving the readout. The paper tests whether richer readouts and fine-tuning reduce dependence, and finds that closing the correlation gap between closely related model pairs does not guarantee greater independence. It pairs this with a concrete reporting recommendation and a public release of aligned per-case traces, analysis scripts, and study code.

Main Findings

• Shared checkpoints are the most correlated pairs. Pairs of models that share a checkpoint show the highest mean per-case Dice correlation, meaning they tend to succeed and fail on the same cases.

• Encoder families behave as families. In the expanded 17-encoder analysis, distinct checkpoints from the same encoder family remain more correlated with each other than cross-family pairs, so simply using different checkpoints of the same architecture does not deliver independence.

• Pools are smaller than they appear. At the pooled 33rd-percentile Dice threshold, four-member pools drawn from distinct encoders have mean effective sizes of 1.56 to 2.92 — a pool of four models acts like somewhere between roughly one and a half and three independent models. The abstract reports these as mean values and does not break them down per task.

• Richer readouts and fine-tuning narrow the gap, but not the dependence. In the primary pool, richer readouts and fine-tuning reduce the Dice-correlation gap between pairs sharing a checkpoint (or the DINOv2-B/S release) and all other pairs. However, dependence between different encoders can increase at the same time.

• Fine-tuning does not reliably buy diversity. Effective pool size at 33rd-percentile thresholds, recomputed separately for each training recipe, does not consistently increase with fine-tuning.

• A smaller gap can hide greater dependence. The authors state directly that a smaller between-group correlation gap can coexist with greater overall dependence, which is why gap-based reasoning is an unsafe proxy for diversity.

• Reporting recommendation. Pool evaluation should report absolute dependence and threshold-specific effective size alongside average accuracy and nominal pool size.

Methodology in Plain English

The authors separate representation from training so that comparisons between encoders are fair. They take publicly available pretrained encoders and freeze them, so the features never change. On top of each frozen encoder they train a very small, separately trained readout (a 1×1 layer) rather than a full segmentation network. Because that readout is simple and randomly initialized, they train four copies per encoder with different seeds to see how much variation comes from training alone.

For every test case, they record the Dice score each model achieves. These aligned per-case traces are the raw material: the analysis looks at whether two models' per-case scores rise and fall together. High correlation means the models fail on the same images. The authors then compare correlation levels across pair types — pairs sharing a checkpoint, pairs within the same encoder family, and pairs from different families.

To translate correlation into an intuitive quantity, they define failure at a pooled 33rd-percentile Dice threshold (the worst third of cases across the pool) and estimate an effective pool size for four-member pools drawn from distinct encoders. Because this estimate needs both failures and passes per member, it is only computed for pools that meet that condition — a caveat the abstract states explicitly. Finally, they repeat the analysis with richer readouts and with fine-tuning, recomputing the 33rd-percentile threshold for each training recipe so the comparison is not confounded by a shifting definition of failure.

The abstract does not report per-task numbers, dataset sizes, confidence intervals, or comparisons to other diversity metrics, so those details are not available here.

Why This Matters

Impact on research. Standard practice in medical imaging reports average accuracy and the number of models in an ensemble. This paper argues that these two numbers are insufficient, because a pool of many models can behave like a pool of far fewer. It gives the field a concrete alternative: report dependence and threshold-specific effective size. The finding that fine-tuning can reduce the measured gap between model groups while increasing overall dependence is a methodological warning that applies well beyond this specific benchmark.

Real-world applications (implied by the framing):

• Clinical deployment planning. A hospital selecting a segmentation tool benefits from knowing whether two "independent" models would catch each other's errors or repeat them on the same difficult cases.
• Ensemble and second-reader design. Systems that route difficult cases to a second model gain little if both models fail on the same anatomy or the same image quality artifact.
• Regulatory and procurement review. Assessment of multi-model or multi-vendor systems could include effective size rather than nominal model count.
• Quality assurance sampling. If failures are correlated, a QA process that audits a random subset of cases across models will miss the shared blind spots it was designed to find.

Industry relevance. Model-selection decisions in medical imaging products are often made from benchmark leaderboards and ensemble counts. Effective-size reporting offers a defensible, auditable alternative metric, and the released per-case traces and scripts lower the cost of adopting it.

Future Directions

• Move from a sample to a census, and beyond 2D. The authors describe 11 encoders as a controlled sample, not a census. Extending the audit to a wider encoder population and to 3D segmentation is the natural next step.

• Explain why fine-tuning fails to increase effective size. The reported result — that fine-tuning does not consistently raise effective pool size at per-recipe thresholds — raises the question of which interventions do produce genuinely independent failures.

• Improve estimation when failures are rare. The effective-size estimate is restricted to pools with enough failures and passes per member. Better estimators for highly accurate or small pools would broaden applicability.

• Turn the recommendation into a standard. The paper calls for routine reporting of absolute dependence and threshold-specific effective size; developing shared reporting templates, benchmarks, and threshold-sensitivity guidance would operationalize that.

Target Audience

Medical imaging and computer vision researchers who build or compare segmentation models; practitioners assembling ensembles or multi-model clinical pipelines; evaluation and reliability researchers interested in diversity metrics; and reviewers, regulators, or procurement teams who need to judge whether a multi-model system's robustness claims are supported. Graduate students will find the frozen-encoder-plus-linear-readout design a clear template for controlled auditing experiments.

Authors’ abstract

Average segmentation accuracy and model count do not reveal whether models fail on the same cases. We present CoFailMed, an audit of correlated failures in pretrained-encoder pools for medical segmentation. Across five 2D tasks, we evaluate 11 public pretrained encoders (a controlled sample, not a census) using frozen features and separately trained $1\times 1$ readouts with four seeds per encoder. Shared-checkpoint pairs have the highest mean per-case Dice correlation. An expanded 17-encoder analysis shows that distinct checkpoints within the same encoder family remain more correlated than cross-family pairs. At the pooled 33rd-percentile Dice threshold, four-member pools drawn from distinct encoders have mean effective sizes of 1.56-2.92, estimated from failure correlations in pools with enough failures and passes per member. In the primary pool, richer readouts and fine-tuning narrow the Dice-correlation gap between pairs sharing a checkpoint (or the DINOv2-B/S release) and all other pairs. Yet dependence between different encoders can increase, and fine-tuning does not consistently increase effective pool size at 33rd-percentile thresholds recomputed for each training recipe. A smaller between-group gap can coexist with greater dependence. Pool evaluation should therefore report absolute dependence and threshold-specific effective size alongside accuracy and nominal pool size. The CoFailMed release provides aligned per-case traces, analysis scripts and study code to reproduce the central audit results.