CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes

AI summary

Overview

Research area: Natural Language Processing — inference-time reasoning improvement for large language models, sitting at the intersection of in-context learning, retrieval-based prompting, and weak-to-strong generalization.

Technical level: Intermediate. The framework itself is conceptually simple (an offline bank of critiques plus a retrieval rule), but evaluating it requires familiarity with Pass@1 accuracy, self-consistency sampling, in-context learning baselines, and distribution-comparison metrics such as Spearman correlation and Jensen–Shannon distance.

Scope (one sentence): The paper introduces CritICL, a two-stage inference-time framework that mines structured failure modes from small language models in a given model family and reuses them as critique-based in-context examples to improve larger models in the same family, with fewer generations and lower token cost than test-time scaling.

What This Paper Is About

Improving LLM reasoning usually means spending more computation at inference time — generating many candidate answers, iterating through self-reflection, or calling a separate judge model. These approaches are expensive, and the paper argues they ignore a cheaper source of signal: the structured ways weaker models fail. The authors observe that small and large models from the same family fail in remarkably similar proportions, and build a framework that turns those shared failure patterns into targeted in-context critiques for the larger model.

Key Contributions

1. CritBank, a failure-aware critique dataset. For each training question and each of several small models, the authors generate five chain-of-thought responses, keep the incorrect ones, use a frontier LLM (gpt-4o-mini) to propose up to five candidate failure mode labels per response, cluster the labels into a taxonomy, and attach a natural-language critique to each (question, response) pair. The dataset is defined as tuples of question, incorrect response, failure mode label, and critique.

2. The empirical finding that failure-mode distributions transfer across scales within a model family. Failure-mode profiles of models as small as 1.5B and 3B parameters line up with those of 70B–72B models in the same family, which is the premise the whole method rests on.

3. Two inference-time variants. CritICL-dynamic predicts likely failure modes for each incoming query and retrieves matching critiques (up to five in the experiments); CritICL-static instead builds a global, model-family-specific failure mode profile from the aggregated weaker models and retrieves critiques for the dominant failure modes, independent of the input.

4. Efficiency evidence. Both variants use fewer generations and fewer total tokens than test-time scaling baselines while matching or exceeding their accuracy.

Main Findings

• CritICL-static leads on Qwen2.5-32B-Instruct. It reaches 49.8% overall Pass@1 (GSM8K 93.6, MATH 59.2, AMC23 26.6, AIME24 19.5, AIME25 17.9), surpassing the strongest test-time scaling baseline, Consistency@7, at 49.5%, by 0.3 points while avoiding repeated inference. CritICL-dynamic reaches 49.1% overall.

• The same pattern holds at 72B scale. On Qwen2.5-72B-Instruct, CritICL-static reaches 59.2% overall, above Consistency@5 (59.0%), LLM-as-Judge (58.5%), Self-Reflection (58.2%), Consistency@7 (57.0%), and the best 5-shot fixed-exemplar baseline (56.3%). CritICL-dynamic reaches 58.7%.

• Reported gains over baselines. The Table 1 caption states CritICL-dynamic and CritICL-static achieve improvements of up to 12.9% and 13.4%, respectively.

• Token and generation cost are sharply lower. On MATH with Qwen2.5-32B-Instruct, CritICL-static uses 3768 total tokens and 1 generation, CritICL-dynamic 3897 tokens and 2 generations, versus 4192 (Consistency@3) through 7533 (Self-Reflection) for test-time scaling baselines. CritICL-static produces 296 output tokens against 308 for 5-shot fixed ICL, so added critique context does not lengthen the answer.

• Failure modes are consistent within families, and aggregation helps. Against Qwen2.5-72B as the strong target, the Qwen aggregate weak profile achieves Spearman 0.91, Kendall τ 0.76, Top-10 overlap 9/10, and JS distance 0.041; the Llama aggregate against Llama-3.1-70B achieves 0.88 / 0.72 / 9/10 / 0.047. Aggregates beat any single weak model, and cross-family transfer is much weaker (Spearman 0.46 and 0.43, JS distances 0.132 and 0.146).

• Failure-mode-based selection beats similarity-based selection. Against random, fixed, and semantic-similarity retrieval under a 5-shot setting on Qwen2.5-72B-Instruct, CritICL wins across accuracy, precision, and recall on GSM8K, MATH, AMC23, and AIME25, with gains of 4–6 points in accuracy on AMC23 and AIME.

• The gain comes from failure-mode alignment, not from extra context. In the source-of-gain ablation, CritICL-static averages 59.9, ahead of 5-shot ICL (56.8), dense correct-exemplar retrieval (57.4), generic GPT critique (57.7), weak incorrect response only (57.3), and shuffled failure labels (57.8).

• One generation suffices against weak-to-strong baselines. In Table 8, CritICL-static averages 59.7 (MATH 84.0, AMC23 35.4) with 1 generation, versus 59.6 for W2S-AlignTree (5 generations) and 59.5 for Consistency@5 (5 generations).

• Annotations are reliable. On 300 re-annotated examples, GPT-4o-mini agrees with GPT-4.1 at F1 0.84 / Cohen's κ 0.77 and with Claude-3.5-Sonnet at F1 0.81 / κ 0.73; on 100 examples, agreement with human annotators is F1 0.82 / κ 0.74, close to human–human agreement of F1 0.86 / κ 0.80.

• Taxonomy granularity has an optimum. A fine-grained taxonomy of 20 groups averages 59.9, ahead of coarse-grained with 8 groups (58.7) and very fine-grained with 45 groups (59.3).

• Generalization beyond mathematics. The paper reports in Appendix E.3 that CritICL also performs effectively on chemistry and biology benchmarks, and reports Llama-family results in Appendix E.1.

Methodology in Plain English

The approach has two offline steps and one online step.

First, the authors take questions from the training splits of GSM8K (7.4k training, 1.3k test) and MATH (7.5k training, 5k test), for a combined 15k questions, and ask each small model to answer with chain-of-thought prompting five times. They check which responses are wrong.

Second, they hand each wrong response to a frontier model, gpt-4o-mini, which proposes up to five possible failure mode labels and writes a natural-language critique explaining what went wrong. Redundant or noisy labels are merged with a clustering procedure borrowed from prior work, producing a failure-mode taxonomy. The result is CritBank: a lookup table mapping (question, wrong response) pairs to failure mode labels and critiques.

Third, at inference time on a new question, the framework picks in-context examples from CritBank. CritICL-dynamic asks the target model to guess which failure modes it is likely to hit on this question, then retrieves critiques tied to those modes. CritICL-static skips the per-query prediction entirely and instead uses a profile of the failure modes that are most common across the whole weaker-model family, retrieving critiques for those. Both variants cap retrieval at five examples.

The framing question throughout is whether training-time fine-tuning is needed for weak-to-strong transfer. CritICL's answer is no: the target model's parameters are never updated. All experiments use greedy decoding at temperature 0.0, and the small-model pools are Qwen2.5-1.5B/3B/7B-Instruct evaluated against Qwen2.5-32B- and 72B-Instruct, and Llama-3.2-1B/3B-Instruct plus Llama-3.1-8B evaluated against Llama-3.1-70B-Instruct.

Why This Matters

Impact on research. The paper reframes model failures as a reusable resource rather than waste. It also separates weak-to-strong generalization into a training-time and an inference-time setting: past work typically uses weak supervision to change the strong model's weights, whereas CritICL shows a plug-in retrieval mechanism can capture part of the benefit without any parameter updates. The failure-mode consistency analysis gives that claim quantitative grounding with rank correlations and distribution distances.

Real-world applications:

• Tutoring and educational systems that want a model to anticipate the specific mistake a student (or a smaller model) is about to make and pre-empt it with a targeted explanation.
• Cost-sensitive deployment on edge or self-hosted hardware, where a single generation at 3768 tokens is affordable but five-sample majority voting at 4814 tokens is not.
• Model-family upgrade paths, letting an organization that has already catalogued a small model's errors reuse that catalogue to scaffold a newly deployed larger sibling.
• Scientific and technical reasoning assistants in domains such as chemistry and biology, which the paper reports as additional evaluation domains.

Industry relevance. The efficiency argument is the commercially salient part. Test-time scaling methods buy accuracy with proportional inference spend, whereas CritICL moves most of the cost into a one-time offline construction of CritBank and pays a modest input-token premium at query time. The measured range of 3768–3897 total tokens for CritICL against 4192–7533 for test-time scaling is the kind of gap that changes serving economics at scale. The GitHub repository is at https://github.com/umwyf/CRITICL.

Future Directions

• Mechanistic explanation. The paper offers shared architecture, tokenizer, pretraining pipeline, and instruction-tuning procedures as a plausible reason failure modes persist across scales, but explicitly states this is not a complete mechanistic account and calls a causal link an open direction.

• Cross-family and cross-domain transfer. Within-family consistency is strong (Spearman 0.91 and 0.88 for aggregates) but cross-family consistency is much weaker (0.46 and 0.43), leaving open whether a family-agnostic failure taxonomy is achievable. Cross-domain transfer is analyzed in Appendix D.2.

• Taxonomy design. Performance peaks at 20 failure-mode groups (59.9 average) and declines at 45 groups (59.3), suggesting that automatic ways to tune granularity per task could matter.

• Statistical grounding. Because the experiments use greedy decoding, the authors use bootstrap resampling over evaluation examples to produce 95% confidence intervals and paired significance tests, an acknowledgement that run-to-run stochastic variation is not the relevant uncertainty here.

Target Audience

Researchers and engineers working on LLM inference efficiency, in-context learning, and retrieval-augmented reasoning will get the most from this paper, particularly those who care about squeezing accuracy out of a fixed inference budget. It is also relevant to anyone studying weak-to-strong generalization, since it offers an inference-time alternative to post-training, and to practitioners building domain-specific reasoning assistants who could amortize a one-time error-catalogue construction across many queries. Readers need working familiarity with benchmark conventions such as Pass@1 and self-consistency to read the result tables, but the core idea is accessible without a deep background in model internals.

Authors’ abstract

Recent advances in inference-time scaling have significantly improved the reasoning performance of large language models (LLMs). However, these methods typically rely on repeated generation or external verification. To address this limitation, we introduce CritICL, a novel inference-time framework that improves reasoning while maintaining high efficiency. Our key insight is that LLM failure modes exhibit structured patterns across model scales within the same family. Instead of treating failures as undesirable outputs, CritICL leverages them as a source of guidance. Specifically, we utilize failure modes derived from weaker models and incorporate them into inference through critique-based in-context examples. We propose two variants: CritICL-dynamic, which adaptively predicts input-specific failure modes and retrieves critiques, and CritICL-static, which uses a global failure mode profile to provide stable guidance. Experimental results show that CritICL consistently outperforms standard in-context learning and achieves performance competitive with or superior to test-time scaling methods, while requiring significantly fewer generations and lower token cost. Code available at: https://github.com/umwyf/CRITICL