DISCO: Diversifying Sample Condensation for Efficient Model Evaluation

AI summary

Overview

• Research area: Efficient machine learning evaluation / benchmarking — specifically, compressing evaluation datasets so that model performance can be estimated cheaply.
• Technical level: Intermediate. The core idea is simple (pick samples where models disagree most), but the paper supports it with information-theoretic propositions (mutual information, Jensen-Shannon Divergence, PDS bounds) and a two-stage evaluation protocol.
• Scope: The paper proposes DISCO (Diversifying Sample Condensation), a method for selecting a small subset of evaluation samples and predicting full-benchmark model performance from model outputs on that subset, validated on four language benchmarks and on ImageNet.

What This Paper Is About

Modern model evaluation is expensive: the paper reports that benchmarks such as LMMs-Eval demand between 30 and 1400 hours on 8 A100 GPUs per model, and HELM requires more than 4000 GPU hours. Existing efficient-evaluation methods typically select an "anchor" subset by clustering samples according to how reference models respond to them, then train a mapping from accuracy on that subset to full test performance — a pipeline the authors describe as complex and sensitive to design choices. DISCO instead argues that what matters is not diversity among samples but diversity in model responses, and it selects the top-k samples on which a pool of source models most disagree.

Key Contributions

1. A simpler selection criterion. The paper proposes selecting samples by model disagreement using sample-wise, greedy statistics (predictive diversity scoring, PDS) rather than global clustering, which the authors argue simplifies the sampling procedure.
2. A theoretical justification. Proposition 1 shows that, under stated assumptions, the mutual information between model performance S(m) and the predicted label on a sample equals the Jensen-Shannon Divergence across the M source models — meaning the sample conveying the most information about model accuracy is the one with greatest JSD. Proposition 2 bounds JSD between two functions of PDS: (2/(M² ln 2))(PDS_i − 1)² ≤ JSD_i ≤ (M/(M−1)) log M · (PDS_i − 1).
3. A direct prediction route. Instead of estimating latent model parameters (as in IRT-based approaches), DISCO defines a model signature — the concatenation of the model's raw outputs on the selected subset — applies PCA for dimensionality reduction, and predicts benchmark performance via kNN or a Random Forest.
4. State-of-the-art empirical results. DISCO reports the best performance prediction across MMLU, HellaSwag, Winogrande, and ARC, and also generalizes to the vision domain on ImageNet-1k.

Main Findings

• Compression with low error on language benchmarks. Reducing each test set to 100 data points yields an inference cost reduction of 99.3% on MMLU and 99.0% on HellaSwag. On MMLU, DISCO (High PDS, signature + Random Forest) achieves 1.07%p MAE with a .987 Spearman rank correlation.
• DISCO results across all four language benchmarks (High PDS + Sig. + RF): MMLU 1.07 MAE / .987 rank; HellaSwag 1.01 / .984; Winogrande 1.00 / .967; ARC 1.47 / .971.
• JSD selection also performs well. High JSD + Sig. + RF gives MMLU 1.30 / .987; HellaSwag 0.86 / .972; Winogrande 1.09 / .973; ARC 1.75 / .938.
• Baselines are weaker. Uniform random sampling with direct evaluation gives 3.45%p MAE / .916 rank on MMLU (2.85/.839 on HellaSwag, 3.60/.827 on Winogrande, 2.61/.898 on ARC). tinyBenchmarks yields 2.79/.922 on MMLU. Anchor-IRT yields 3.25/.922; Anchor-corr 2.08/.927.
• Metabench results are flagged as not directly comparable. Its reported numbers (e.g., 2.08† MAE / .904† rank on MMLU) carry a dagger because it requires more examples to converge: 150 datapoints for MMLU and ARC (up to +50%), 450 for HellaSwag (up to +350%), and 200 for Winogrande (up to +100%).
• Model signatures alone beat the baseline. Even with uniform random sample selection, signature + Random Forest reaches 1.81%p MAE and .933 rank on MMLU before any disagreement-based selection is applied.
• Selection matters on top of signatures. Adding PDS-based selection is what lifts the method to 1.07%p MAE and .987 rank on MMLU.
• Extreme compression favors kNN. Across varying compression rates DISCO is described as consistently the best method, but at 10 samples the non-parametric kNN predictor outperforms the parametric Random Forest, suggesting non-parametric approaches may be more suitable at extreme compression.
• Robustness to the model split. Chronological splitting (source models published before January 13, 2024; target models after) yields a rank correlation of .987, nearly identical to the .986 obtained with uniform splitting.
• Source-model count matters. With only 100 source models DISCO reaches .969 rank correlation, already outperforming TinyBenchmarks, which uses all 382 available source models (.927). Rank correlation rises to a maximum of .987 at |F| = 382.
• Dimensionality reduction helps. Without it (all 3100 dimensions), the correlation is .918; PCA at 256 dimensions improves it to .987. PCA outperforms UMAP across a wide range of dimensions.
• Random Forest is the best predictor tested, reaching .987 rank correlation on MMLU, ahead of all other prediction models considered in the factor analysis.
• Stratification is not helpful under PDS. Stratifying anchors across MMLU's 57 tasks gives .978, which is not effective compared to the .987 obtained when data points are sampled according to PDS.
• Vision domain results. On ImageNet-1k (1.28M images, 400 pretrained timm models), reducing the validation set to 100 data points gives a 99.8% inference cost reduction. DISCO (High PDS + Sig. + RF) achieves 0.63%p MAE and .969 rank correlation, versus 3.03/.652 for the direct-evaluation baseline, 2.06/.838 for Lifelong Bench., and 3.05/.762 for SSEPY.
• Conclusion drawn across domains: instead of selecting anchor points that broadly cover sample difficulty, one should select the points on which models typically disagree.

Methodology in Plain English

The approach has two stages.

Stage one — choose which samples to evaluate. The researchers take a held-out pool of "source" models whose true benchmark scores are already known. For every question in the test set, they look at what all those source models predict. Questions where the models agree closely are considered uninformative; questions where the models disagree strongly are considered informative, because disagreement is what distinguishes one model from another. Disagreement is scored with PDS (predictive diversity scoring), which is a continuous generalization of counting how many distinct argmax predictions the source models produce. The top-k scoring questions become the small evaluation set. The paper also tests JSD (Jensen-Shannon Divergence across the source models) as an alternative score, and proves the two are related by explicit bounds.

Stage two — predict full performance from the small set. Each model being evaluated is represented by its model signature: the concatenation of its raw outputs over the selected samples. Because that vector can be very high-dimensional (potentially up to 50k dimensions for the ImageNet validation set in their illustration), PCA is applied first. Then either a kNN predictor (find the k most similar source models by Euclidean distance in the reduced signature space and average their true performances) or a parametric mapping (a Random Forest, in their case) predicts the target model's full-benchmark score.

The evaluation protocol is held identical across compared methods: same source models, same anchor-selection and prediction stages, same training/test sequence. The reported metrics are Mean Absolute Error in percentage points of accuracy and Spearman rank correlation between the true and estimated model rankings.

Why This Matters

• Research impact: The paper challenges the assumption that diverse or representative samples are what efficient benchmarking needs, and offers an information-theoretic argument that model disagreement is the optimal greedy selection signal. It also argues that skipping latent-parameter estimation (IRT-style psychometric modeling) in favor of direct signature-to-performance mapping is both simpler and more accurate, which could shift how efficient-evaluation methods are designed.
• Real-world applications:
  • Frequent performance tracking of models during training, where full benchmark runs are impractical.
  • Cheap end-user or deployment-time checks of models running on limited compute.
  • Broadening participation in evaluation by lowering the GPU-hour barrier, which the authors link to inclusivity and the pace of innovation.
  • Reducing the environmental footprint of repeated large-scale benchmark evaluations.
• Industry relevance: The reported savings (over 99% cost reduction on MMLU, 99.0% on HellaSwag, 99.8% on ImageNet validation) map directly onto engineering budgets and iteration speed, and the meta-model design (a pool of known models plus a predictor) fits naturally into model-leaderboard and model-selection workflows.

Future Directions

1. Robustness to model-population shifts. The authors name this as the main limitation: new architectures, training methods, or objectives introduce patterns unseen during training and reduce estimator accuracy. Suggested remedies are adaptive sample selection or periodic retraining on newer models.
2. Extending beyond fixed answer choices. DISCO requires predictive probabilities over a predefined set of answer choices, so it is not suitable for open-ended generation tasks such as translation or summarization; applying it there would first require defining sets of correct and incorrect outputs.
3. Understanding the extreme-compression regime. Since kNN beats the parametric Random Forest at 10 samples, the question of when non-parametric prediction should be preferred at very high compression rates remains open.
4. Broader domain generalization. The paper demonstrates language and vision results; the authors frame the method as domain-agnostic, with detailed vision results placed in the appendix.

Target Audience

Researchers and practitioners working on efficient evaluation, benchmarking, and model selection for large language models and vision models; anyone maintaining leaderboards or running repeated evaluations on constrained compute budgets; and readers interested in applying information-theoretic criteria (mutual information, Jensen-Shannon Divergence) to practical data-selection problems. Readers should be comfortable with benchmark evaluation terminology and basic probability concepts; the theoretical propositions and the IRT-based baselines require more background.

Authors’ abstract

Evaluating modern machine learning models has become prohibitively expensive. Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model. Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact. The typical approach follows two steps. First, select an anchor subset of data. Second, train a mapping from the accuracy on this subset to the final test result. The drawback is that anchor selection depends on clustering, which can be complex and sensitive to design choices. We argue that promoting diversity among samples is not essential; what matters is to select samples that $\textit{maximise diversity in model responses}$. Our method, $\textbf{Diversifying Sample Condensation (DISCO)}$, selects the top-k samples with the greatest model disagreements. This uses greedy, sample-wise statistics rather than global clustering. The approach is conceptually simpler. From a theoretical view, inter-model disagreement provides an information-theoretically optimal rule for such greedy selection. $\textbf{DISCO}$ shows empirical gains over prior methods, achieving state-of-the-art results in performance prediction across MMLU, Hellaswag, Winogrande, and ARC. Code is available here: https://github.com/arubique/disco-public.