BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs

AI summary

Overview

Research area: Natural Language Processing — zero-shot text classification, model benchmarking, and comparative evaluation of encoder-based versus generative architectures.

Technical level: Intermediate. The paper assumes familiarity with transformer architectures, embedding models, and LLM prompting, but the experimental design and conclusions are explained without heavy mathematical formalism.

Scope: A unified benchmark (BTZSC) of 22 datasets and 38 model checkpoints comparing four model families — NLI cross-encoders, embedding models, rerankers, and instruction-tuned LLMs — under a strictly zero-shot classification protocol.

What This Paper Is About

Zero-shot text classification promises to eliminate expensive task-specific annotation by matching text directly against human-readable label descriptions rather than training on labeled examples. However, existing evaluations are fragmented: some compare only NLI-based encoders, others (like MTEB) measure classification using supervised linear probes trained on labeled data, which tests something other than genuine zero-shot ability. This paper builds a single benchmark that evaluates all four major model families under one consistent, label-free protocol to determine which architectures actually work best when no task-specific training data is available.

Key Contributions

1. A new benchmark (BTZSC) of 22 English datasets spanning four task families — sentiment, topic, intent, and emotion — deliberately chosen for diversity in domain (news, social media, reviews, Wikipedia, political manifestos), class cardinality (2 to 77 labels), and document length (8 to 293 tokens on average).

2. A systematic four-family comparison across 38 checkpoints, including 11 NLI cross-encoders (3 of them custom-trained on a mixture of MNLI, ANLI, WANLI, FEVERNLI, and LingNLI), 11 embedding models, 7 rerankers, and 7 instruction-tuned LLMs plus 3 base transformer encoders.

3. An analysis of scaling behavior and the accuracy-latency trade-off, showing which model families benefit from increased parameter count and which saturate early.

4. A study of whether NLI competence predicts zero-shot classification quality, finding that the relationship holds for cross-encoders and LLMs but breaks down for embedding models.

5. Open release of datasets, evaluation code, trained checkpoints, and a live leaderboard to support reproducible comparison.

Main Findings

• Modern rerankers set a new state of the art. Qwen3-Reranker-8B achieves macro F1 = 0.72 and accuracy = 0.76, beating the best NLI cross-encoder by roughly 12 F1 points and the best embedding model by 10 points. Notably, the much smaller Qwen3-Reranker-0.6B (macro F1 = 0.61) already surpasses every NLI cross-encoder despite being a fraction of the size of some of them.

• Strong embedding models close most of the gap at the lowest cost. GTE-large-en-v1.5 reaches macro F1 = 0.62, outperforming all NLI cross-encoders and all comparably sized rerankers. Because embeddings can be precomputed and compared with cosine similarity, this family occupies the most favorable accuracy-latency trade-off — the majority of models in the Pareto-efficient quadrant of the speed-versus-accuracy plot are embedding models.

• Instruction-tuned LLMs become competitive only above roughly 4B parameters. Small models (Gemma-3-270m, Gemma-3-1b) perform at or below the level of untuned base encoders. In the 3B–4B range, results vary dramatically by model family: Qwen3-4B reaches F1 = 0.65 while Llama-3.2-3B-Instruct and Phi-4-mini-instruct sit in the mid-0.40s. The best LLM tested, Mistral-Nemo-Instruct-2407 (12B), reaches F1 = 0.67, exceeding all embeddings and cross-encoders but still trailing Qwen3-Reranker-8B.

• NLI cross-encoders plateau. Custom multi-dataset NLI training improves over off-the-shelf checkpoints like bart-large-mnli by about 6 F1 points, and large backbones beat base backbones by about 3.5 points. But the family tops out around F1 = 0.60 with DeBERTa-v3-large, and adding scale yields diminishing returns. Loss function choice (binary cross-entropy with neutral collapsed versus three-way cross-entropy) has minimal impact.

• Task difficulty varies sharply. Sentiment classification is relatively easy (median F1 around 0.88–0.90), topic and intent are intermediate (0.4–0.55), and emotion detection is hardest (0.25–0.35). Larger models help most on the harder task families.

• Scaling behaves differently per family. Rerankers improve roughly monotonically with size and dominate at every scale. Embedding models improve rapidly up to a few hundred million parameters, then saturate near F1 = 0.58–0.62 (Qwen3-Embedding-8B at 0.59 barely improves on the 0.6B variant at 0.58). LLMs show the steepest curve, rising slowly below 1B and then sharply between 3B and 8B.

• NLI ability predicts zero-shot quality only for some families. NLI cross-encoders show an almost linear relationship between NLI AUROC and BTZSC F1. Large LLMs follow a similar pattern. Rerankers show a positive but looser trend — several achieve strong classification scores with only moderate NLI performance. Embedding models show no clear monotonic relationship at all, suggesting that once basic entailment competence is reached, the structure of the embedding space matters more than NLI benchmark scores.

• Findings replicate on MTEB. Running the same evaluation on eight English classification tasks from MTEB v2 produces strongly correlated rankings (Kendall's τ = 0.69, p < 10⁻⁸), with consistent family-level conclusions.

Methodology in Plain English

The researchers assembled 22 existing, publicly available English single-label classification datasets chosen to cover four task types, a range of label counts, and varied domains and text lengths. Critically, no labeled examples from these datasets were used for training or model selection — models only ever see a document and a list of written-out label descriptions.

Each label was converted into a short natural-language sentence. For example, in the Amazon Polarity dataset the positive class becomes "The overall sentiment within the Amazon product review is positive." The models then score how well each label description fits the input text, using whatever mechanism their architecture supports:

• NLI cross-encoders treat the text as a premise and each label description as a hypothesis, and pick the label with the highest entailment score.
• Embedding models encode the text and each label into vectors and pick the label whose vector is closest by cosine similarity.
• Rerankers treat the text as a query and the label descriptions as candidate documents, selecting the highest-ranked document.
• Instruction-tuned LLMs receive a multiple-choice prompt and the label whose answer token has the highest conditional probability is selected.

The primary metric is macro F1, which weights each class equally regardless of frequency, making it suitable across datasets with wildly different label counts. Accuracy, precision, and recall are also reported. The authors additionally evaluated every model on standard NLI benchmarks and measured AUROC to test whether entailment ability transfers to classification. Statistical robustness was ensured by averaging training runs over three random seeds, and all inference ran in bfloat16 precision on NVIDIA A100 GPUs.

Why This Matters

Impact on research. Prior zero-shot evaluations either tested only one model family or measured embedding quality through supervised probes, which answers a different question. BTZSC provides a controlled, apples-to-apples comparison under a genuinely label-free protocol. It also supplies a reusable testbed for tracking progress, and its finding that NLI scores are not a reliable proxy for zero-shot quality in embedding models challenges a common assumption in model selection.

Real-world applications:

• Customer feedback triage — routing product reviews or support tickets into sentiment, topic, or intent categories without training a bespoke classifier for each new product line.
• Financial and regulatory document sorting — the author's affiliation with the European Central Bank points to use cases in classifying policy documents, filings, or supervisory text where labeled examples are scarce and expert annotation is costly.
• Content moderation — the benchmark includes toxicity, obscenity, threat, and bias datasets, which map directly to automated flagging systems that must generalize to new categories without retraining.
• Conversational agents and voice assistants — intent detection datasets like Banking77 and MASSIVE reflect the need to recognize user intents that were never present in training data.

Industry relevance. The accuracy-latency analysis has direct deployment implications. Embedding models offer the best return on compute for high-throughput production systems, while rerankers deliver the highest accuracy for latency-tolerant pipelines. The finding that an 8B reranker beats a 12B generative LLM at classification suggests that task-specialized retrieval architectures may be a more efficient path than scaling general-purpose generators.

Future Directions

1. Multilingual extension. BTZSC is limited to English; the authors explicitly flag multilingual evaluation as future work, which matters for globally deployed systems.

2. Better label verbalizations and prompt engineering. Every result depends on how labels are phrased and how prompts are written. Systematically optimizing these could shift relative rankings and is left largely unexplored.

3. Pretraining contamination. The paper acknowledges that some evaluation datasets may appear in model pretraining corpora, and that undocumented overlap cannot be fully ruled out. Establishing contamination-free evaluation sets or rigorous detection methods remains an open problem for the field.

4. Closing the gap between rerankers and cheaper alternatives. The best reranker leads by a wide margin but at 8B parameters. Understanding why rerankers generalize better than embeddings and whether that advantage can be distilled into smaller, faster models is a natural next step.

Target Audience

This paper is most useful for NLP practitioners and applied machine learning engineers selecting a model for a classification task without labeled data, and for researchers studying zero-shot generalization, transfer learning, or the comparative capabilities of encoder versus generative architectures. Benchmark designers and evaluation researchers will also find the methodology relevant, particularly the critique of supervised probes in existing benchmarks. The writing is accessible to graduate students and industry engineers with working knowledge of transformer models, though the density of model names and variants assumes some familiarity with the current landscape of open-weight language models.

Authors’ abstract

Zero-shot text classification (ZSC) offers the promise of eliminating costly task-specific annotation by matching texts directly to human-readable label descriptions. While early approaches have predominantly relied on cross-encoder models fine-tuned for natural language inference (NLI), recent advances in text-embedding models, rerankers, and instruction-tuned large language models (LLMs) have challenged the dominance of NLI-based architectures. Yet, systematically comparing these diverse approaches remains difficult. Existing evaluations, such as MTEB, often incorporate labeled examples through supervised probes or fine-tuning, leaving genuine zero-shot capabilities underexplored. To address this, we introduce BTZSC, a comprehensive benchmark of 22 public datasets spanning sentiment, topic, intent, and emotion classification, capturing diverse domains, class cardinalities, and document lengths. Leveraging BTZSC, we conduct a systematic comparison across four major model families, NLI cross-encoders, embedding models, rerankers and instruction-tuned LLMs, encompassing 38 public and custom checkpoints. Our results show that: (i) modern rerankers, exemplified by Qwen3-Reranker-8B, set a new state-of-the-art with macro F1 = 0.72; (ii) strong embedding models such as GTE-large-en-v1.5 substantially close the accuracy gap while offering the best trade-off between accuracy and latency; (iii) instruction-tuned LLMs at 4--12B parameters achieve competitive performance (macro F1 up to 0.67), excelling particularly on topic classification but trailing specialized rerankers; (iv) NLI cross-encoders plateau even as backbone size increases; and (v) scaling primarily benefits rerankers and LLMs over embedding models. BTZSC and accompanying evaluation code are publicly released to support fair and reproducible progress in zero-shot text understanding.