Cleaning the Pool: Progressive Filtering of Unlabeled Pools in Deep Active Learning

AI summary

Overview

Research area: Deep active learning (AL) for adapting pretrained vision foundation models, specifically ensemble methods that combine multiple instance-selection strategies.

Technical level: Advanced. The two-stage selection mechanism is intuitive, but the paper includes formal definitions and three theorems with proofs (Appendix A), plus AULC-based benchmarking conventions that assume familiarity with the AL literature.

Scope: The paper introduces and evaluates Refine, an ensemble AL method that progressively filters an unlabeled pool with many AL strategies and then applies coverage-based selection, benchmarked on 6 image classification datasets with 3 vision foundation models.

What This Paper Is About

Active learning tries to cut annotation costs by selecting only the most informative unlabeled instances to be labeled, but no single selection strategy works best across all datasets, models, and AL cycles, and a poorly chosen strategy can even do worse than random sampling. The paper's goal is an ensemble method that combines many strategies without needing to know in advance which one will perform best. Refine addresses this in two stages per AL cycle: progressive filtering, which iteratively shrinks the unlabeled pool to candidates that at least one strategy considers valuable, followed by coverage-based selection of the final annotation batch from that refined pool.

Key Contributions

1. Progressive filtering. A mechanism that iteratively refines the unlabeled pool over multiple rounds by taking the union of batches selected by every strategy in the ensemble, retaining candidates that at least one strategy ranks highly while discarding instances no strategy selects. The authors emphasize it is training-free and can also be used as a standalone preprocessing step.

2. Refine, an ensemble AL method. Progressive filtering combined with a final coverage-based selection (implemented with UHerding, which extends MaxHerding with an uncertainty factor) that selects the batch maximizing coverage over the refined pool together with the already labeled pool.

3. Theoretical analysis of progressive filtering. Three results with proofs in Appendix A: a value-preservation lower bound on survival probability (Theorem 1), an exponential reduction bound for uninformative instances (Theorem 2), and a monotonic non-decreasing expected value of the candidate pool across rounds (Theorem 3).

4. Empirical validation. Experiments across 6 image classification datasets and 3 foundation models showing Refine outperforms individual strategies and existing ensemble AL methods, plus a demonstration that progressive filtering improves any individual AL strategy applied to the refined pool, validated on an audio spectrogram classification use case (Appendix B).

Main Findings

• Refine is the most competitive individual-strategy baseline. Across the pairwise comparison matrix averaged over 3 backbones, 5 datasets, and 10 trials, Refine achieves the highest win rates, beating BAIT in 85% and UHerding in 80% of all cases.

• Refine outperforms existing ensemble AL methods. In the ensemble comparison matrix (also averaged across 3 backbones, 5 datasets, and 10 trials), Refine achieves a 100% win rate against both SelectAL and TAILOR, 98% against TCM, and 97% against AutoAL.

• Consistent gains on the complex datasets. Refine consistently outperforms all strategies on CIFAR-100, Food101, and Tiny ImageNet across all backbones, as shown in the relative learning curves for DINOv3 and CLIP.

• Progressive filtering improves every individual strategy as a preprocessing step. On CIFAR-100 with DINOv2 and DINOv3, selecting from the refined pool rather than the raw unlabeled pool improved random sampling and five state-of-the-art strategies by AULC gains ranging from +0.7% to +5.4%. Weaker strategies benefited most, while well-performing strategies such as UHerding showed smaller gains.

• Filtering can rescue a strategy that is worse than random. With DINOv3 on CIFAR-100, BAIT without filtering (Baseline) performed worse than random sampling, and filtering improved its performance (reported at +4.9%, versus +2.6% for AlfaMix).

• Deeper filtering generally helps, with diminishing returns. In Table 2(a), random sampling from the filtered pool improved AULC over random sampling from the unfiltered pool at every tested number of rounds R in {1, 2, 3, 5, 7, 9}: on CIFAR-10 the improvements were 3.02, 3.31, 3.72, 3.71, 3.81, and 3.78 percentage points respectively, and on Snacks 6.91, 6.75, 7.22, 7.79, 8.10, and 8.43. For CIFAR-10 at R = 9 the improvement plateaued.

• The sampling ratio α is robust over a wide range. In Table 2(b), random sampling from the filtered pool (R = 5, J = 5) beat random sampling from the unfiltered pool at every α tested: 3.11 and 7.37 at α = 0.1, 3.33 and 6.32 at α = 0.2, 3.89 and 7.60 at α = 0.3, 3.62 and 8.16 at α = 0.5, 3.97 and 7.53 at α = 0.7, and 3.70 and 8.40 at α = 0.9 (CIFAR-10 and Snacks respectively). Values of α ≤ 0.2 slightly reduced filtering effectiveness, while α in [0.3, 0.9] performed well on both datasets.

• More batches per strategy J generally helps. Table 2(c), at R = 5 and α = 0.4, reports improvements from 3.57 and 7.88 at J = 1, 3.71 and 7.79 at J = 5, 3.79 and 7.22 at J = 10, 3.89 and 8.05 at J = 12, 3.85 and 7.30 at J = 15, to 3.93 and 7.69 at J = 20 (CIFAR-10 and Snacks), with the benefit diminishing at larger values.

• The refined pool captures complementary notions of value. With filtering restricted to Margin (uncertainty) and TypiClust (representativeness) on CIFAR-10, the pool initially profited from TypiClust's representative instances; as TypiClust's effectiveness diminished in later cycles, random sampling from the refined pool benefited from uncertain instances captured by Margin, consistently across both backbones. UHerding on the refined pool further improved performance.

• A reported failure mode. On Dopanim with the CLIP backbone, Refine did not yield the best learning curve, and at around 350 labels its accuracy decreased slightly, which the authors attribute to three AL strategies performing equal to or worse than random sampling. Poorly designed or correlated strategies can degrade the union of the filtered pool.

• Theory supports the mechanism. Theorem 1 shows an instance's survival probability in a round is lower-bounded by 1 − (1 − α · max over strategies of p_{m,r}(x))^J; Theorem 2 bounds the survival of an ε-uninformative instance after R rounds by (1 − (1 − αε)^{MJ})^R, decreasing as O((MJαε)^R) when αε << 1/(MJ); Theorem 3 shows the expected average true value of the pool is monotonically non-decreasing across rounds under the assumption that strategies favor higher-value instances.

Methodology in Plain English

The setting is pool-based batch active learning for classification. A feature extractor from a frozen foundation model (h_φ) feeds a trainable classification head (g_θ); each cycle, the head is fine-tuned for 200 epochs with SGD (batch size 64, learning rate 0.01, weight decay 10⁻⁴, cosine annealing) after new labels are added.

At each cycle, progressive filtering runs R rounds. In each round, every one of the M strategies in the ensemble is applied J times to produce batches of size b, drawn from a uniformly sampled subset of the current candidate pool of size α · |C_{r−1}|. The next round's pool is the union of all those batches. Because an instance must be repeatedly picked to survive, uninformative instances tend to drop out while anything valued by at least one strategy is likely to stay. Sampling subsets rather than the whole pool keeps costs down, lets deterministic strategies produce non-identical batches, and lets the method explore a pool too large for a single strategy to handle in memory. A minimum pool size equal to b is enforced, and the authors fix α = 0.4, R = 5, and J = 10 by grid search on a validation split of CIFAR-10, then reuse these values everywhere.

The second stage selects the final batch from the refined pool by maximizing kernel-based coverage over the refined pool jointly with the labeled pool, applied to foundation model features rather than raw instances. UHerding is used for this, chosen because most strategies in the ensemble emphasize representativeness, so an uncertainty factor balances exploration and exploitation; the authors note any coverage-based strategy, such as TypiClust, could be used instead.

Evaluation uses relative learning curves (accuracy gains in absolute percentage points over random sampling at each cycle) and area under the learning curve (AULC), plus pairwise win rates, averaged over 10 independent trials with standard errors. The protocol runs 20 cycles starting from a randomly sampled initial labeled pool of size b, with dataset-specific batch sizes chosen so that accuracy converges under random sampling. Baselines are random sampling, the uncertainty-based Margin and BADGE, the representativeness-based TypiClust and MaxHerding, the hybrid BAIT, ALFAMix, DropQuery, and UHerding, and the ensemble methods SelectAL, TCM, TAILOR, and AutoAL. The datasets span 10 to 200 classes: CIFAR-10 (50,000 instances, 10 classes, batch size 10), Dopanim (10,484 instances, 15 classes, batch size 50), Snacks (4,838 instances, 20 classes, batch size 20), CIFAR-100 (50,000 instances, 100 classes, batch size 100), Food101 (75,750 instances, 101 classes, batch size 100), and Tiny ImageNet (100,000 instances, 200 classes, batch size 200). Because CIFAR-10 was used for hyperparameter selection, the main results report the remaining datasets. The three backbones are DINOv2-ViT-S/14, DINOv3-ViT-S/16, and CLIP-ViT-B/16, each with a randomly initialized fully connected head.

Why This Matters

Impact on research. The paper reframes ensemble active learning away from learning or planning when to switch between strategies (as in TCM, TAILOR, AutoAL, or AutoAL-style vote weighting) toward a training-free, pool-cleaning mechanism with theoretical guarantees. It also shows that filtering is a reusable preprocessing step, not just one component of a specific method, which suggests a general way to make existing AL strategies better rather than replacing them. The released implementation is available in the dal-toolbox repository.

Real-world applications (the paper itself validates image classification and, as a use case study, audio spectrogram classification; the following are domains where labeling is costly and the method applies):

• Medical imaging, where expert annotation is expensive and each image may contain findings relevant to only some diagnostic criteria.
• Audio and speech annotation, the setting the authors use to demonstrate progressive filtering on an audio spectrogram classification use case.
• Fine-grained or long-tailed visual recognition, where the benchmark datasets with 100 and 200 classes (CIFAR-100, Food101, Tiny ImageNet) approximate the many-class conditions of industrial inspection or product categorization.
• Environmental or remote sensing monitoring, where large unlabeled archives must be triaged before any field verification is possible.

Industry relevance. Companies adapting pretrained foundation models to niche tasks often have a fixed labeling budget and no labeled validation set on which to compare selection strategies in advance, since AL is effectively a one-shot problem. A method that does not require committing to one strategy, and that can reduce the pool that expensive strategies must score, is directly useful for controlling labeling cost and compute. The reported memory benefit matters in practice: many AL strategies do not scale to very large pools without a fixed subsample, whereas filtering explores the whole pool through subsets of size α · |C_{r−1}|, and with M concurrent processes the per-round cost reduces to O(J · C_max(α · |C_{r−1}|)).

Future Directions

• Weighting ensemble components. The authors identify weighting of ensemble components as a promising direction, motivated by the Dopanim/CLIP failure mode where strategies performing at or below random sampling degraded the filtered pool.
• Growing the strategy set. Because Refine accepts any strategy that can produce a batch of candidates, the ensemble can be extended with upcoming state-of-the-art AL strategies offering new notions of value.
• Better ensemble composition rules. The paper recommends building the strategy set from a small set of well-established, complementary strategies rather than arbitrarily including many, but does not derive a principled selection criterion for that set.
• Sensitivity and saturation of filtering depth. Table 2(a) shows diminishing returns and a plateau for CIFAR-10 at R = 9, leaving open how to adapt R, J, and α per dataset or budget rather than fixing them from a CIFAR-10 validation split.

Target Audience

Active learning researchers and graduate students working on data-efficient learning and ensemble selection methods; practitioners adapting vision foundation models to tasks with constrained annotation budgets; and engineers who need pool-based selection to scale to large unlabeled sets where individual strategies are too costly or memory-intensive. Readers seeking the algorithmic idea and empirical evidence can follow the main text, while those interested in the guarantees should read the theorems in Section 4.3 alongside the proofs in Appendix A.

Authors’ abstract

Existing active learning (AL) strategies capture fundamentally different notions of data value, e.g., uncertainty or representativeness. Consequently, the effectiveness of strategies can vary substantially across datasets, models, and even AL cycles. Committing to a single strategy risks suboptimal performance, as no single strategy dominates throughout the entire AL process. We introduce REFINE, an ensemble AL method that combines multiple strategies without knowing in advance which will perform best. In each AL cycle, REFINE operates in two stages: (1) Progressive filtering iteratively refines the unlabeled pool by considering an ensemble of AL strategies, retaining promising candidates capturing different notions of value. (2) Coverage-based selection then chooses a final batch from this refined pool, ensuring all previously identified notions of value are accounted for. Extensive experiments across 6 classification datasets and 3 foundation models show that REFINE consistently outperforms individual strategies and existing ensemble methods. Notably, progressive filtering serves as a powerful preprocessing step that improves the performance of any individual AL strategy applied to the refined pool, which we demonstrate on an audio spectrogram classification use case. Finally, the ensemble of REFINE can be easily extended with upcoming state-of-the-art AL strategies.