Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters

AI summary

Overview

Research area: Natural language processing — compact encoder models for closed-set decision tasks (tool selection, intent classification, multiple-choice answering, fact verification), with a focus on efficient alternatives to prompting large language models.

Technical level: Intermediate. The paper is readable with a working knowledge of transformer encoders and standard evaluation metrics; the experimental design (matched-compute controls, contamination audits, paired bootstrap) is explained clearly enough to follow without specialist background.

Scope: A three-seed controlled comparison of joint versus per-option option-reading for closed-set decisions, plus the release and external evaluation of six distilled encoders from 17M to 1B parameters (the Crivo family) against a 435M DeBERTa-v3 zero-shot classifier.

What This Paper Is About

Many language-model pipelines end in a decision from a fixed set of options — which tool to call, which intent a message expresses, which of four answers is right. Prompting a large LLM for this is slow and sensitive to the order in which options are listed, so a small dedicated encoder is attractive; the open question is how small such an encoder can be and, more fundamentally, whether it should read all options together in one pass or score each option separately. The paper answers both questions with a controlled ablation at 32M parameters and a family of six released joint-scoring encoders evaluated on two overlap-audited public benchmarks and a public leaderboard.

Key Contributions

1. A controlled joint-versus-per-option comparison. In a 32M grid with three seeds per arm and everything else fixed, per-option scoring beats the joint reader on a held-out sample of the 37-benchmark Decision Index public suite by +0.058 chance-corrected skill at matched optimizer steps and by +0.028 at approximately matched training compute. A 2×2 crossing of initialisation (reranker versus raw encoder) by reader locates the advantage: it exists only when starting from the cross-encoder reranker, whose (query, candidate) pair input matches per-option reading.

2. Six compact joint encoders with contamination-audited external evaluation. The Crivo models (16.9M, 32.0M, 68.4M, 149.6M, 395.8M and 1.03B parameters) cost about US$80 of rented GPU time in total (US$54 recorded, the rest estimated). On all 209 System1-Arena tasks, Crivo-68M and Crivo-150M exceed a 435M DeBERTa-v3-large zero-shot classifier by +0.049 and +0.059 task-mean accuracy, and are at least on par with it on the 145 tasks unseen in training.

3. Robustness, calibration and cost on stated hardware. Reordering options changes 12–23% of the joint models' answers (2% for DeBERTa, none by construction for per-option scoring); the models are over-confident on unseen tasks; Crivo-17M answers a median request in 14 ms on 16 CPU threads (13.5 ms in one section) against 728 ms for the classifier.

4. Private-test evidence and a parameter-free probe of the benchmark. On the Decision Index's maintainer-run private tests, Crivo-17M, -32M and -68M lead their size range on mainly in-distribution skills but stay near the floor on new domains, and the authors' own development panels overestimated every model's private score. Prumo, a rule engine with no learned parameters, matches Crivo-17M on the same-skill tests while staying near chance externally, exposing format-specific solving in part of the board.

Main Findings

• Per-option beats joint at matched steps. On the pub15 held-out sample, per-option (P) reaches 0.245 skill against 0.180 for the joint arm (J) and 0.187 for a steps-matched joint control (J-steps); the P − J-steps difference is +0.058 [+0.050, +0.065]. Per-option is also better on S1-60 task-mean accuracy (+0.037) and BTZSC-500 macro-F1 (+0.116).

• Matching compute closes only about half the gap. The compute-matched control (J-comp, trained for P's 6,858–6,860 steps) reaches 0.217 pub15 skill, leaving P ahead by +0.028 [+0.015, +0.042]. Compute accounts for +0.029 of the step-matched gap.

• The advantage does not reach significance on unseen data. On the 145 unseen S1-Arena tasks the compute-matched difference is +0.011 [−0.009, +0.031] and on the unseen BTZSC datasets +0.025 [−0.034, +0.084]. The authors describe the external benchmarks as agreeing in sign but not significantly.

• The advantage requires reranker initialisation. From the raw Ettin encoder the two readers are indistinguishable at matched compute: C1 (P-raw − J-comp-raw) is −0.011 [−0.038, +0.015] on pub15 and its interval contains zero on all three evaluation sets. The interaction term C4 is +0.040 [+0.014, +0.066] on pub15. The proposed explanation is a format match — rerankers are trained on exactly the per-option (query, document) pair input — but the 2×2 does not test the mechanism.

• The per-benchmark gain is concentrated. P is better on 22 benchmarks and worse on 10; only 8 gains and one loss (ContractNLI, −0.127) have paired intervals excluding zero, and the median gain is +0.009. Five benchmarks (BPoMP, iSarcasmEval, API-Bank, MuSR, ACOS, +0.12 to +0.33) carry most of the mean; without the top two the mean gain is +0.015. The gain does not grow with the number of options (Spearman ρ = −0.14) or input length (ρ = −0.06).

• Per-option reading costs more passes but not proportionally more wall time. It needs 5–13 times more encoder passes per question, taking at most twice the median latency but more than four times the mean, as reported in the introduction.

• Compact models beat a much larger zero-shot classifier on S1-Arena. Crivo-68M and Crivo-150M exceed DeBERTa-v3-large zero-shot (435M) by +0.049 and +0.059 accuracy over all 209 tasks, including after a Holm correction; on the four unseen BTZSC datasets, Crivo-400M and Crivo-1B match or exceed published zero-shot rows of up to 8B parameters.

• Option order matters for joint reading. Permutation flips are 0.277 for J, 0.262 for J-steps and 0.253 for J-comp in the grid, 0.196 for the released Crivo-32M, and 0.000 for per-option; in the wider robustness sample reordering changes 12–23% of the joint models' answers against 2% for DeBERTa.

• Calibration does not track accuracy. J-comp has the lowest pub15 ECE of all arms (P − J-comp +0.063) but a worse S1-60 Brier score (0.672 versus 0.602 for J-steps), which the authors associate with reading each row about 3.25 times. The released models are over-confident on unseen tasks.

• Latency on stated hardware. Crivo-17M answers a median request in 14 ms on 16 threads of an AMD EPYC 7J13, against 728 ms for DeBERTa; measurements are also reported on one A100-SXM4 40 GB.

• The benchmark rewards format-specific solving. Prumo, with no learned parameters and about 2.7k lines of Python, matches Crivo-17M on the board's same-skill tests while staying near chance on external benchmarks. The authors' development panels overestimated every model's private score.

Methodology in Plain English

The authors built a family of encoders, called Crivo, that take a structured request — a "state" (a conversation, document, tool catalogue or board position) plus one or more typed "questions" with a fixed set of options — and return a probability for every option, with no generated text. Each option gets a special marker token inside a single sequence, and the hidden states at those markers are normalised into a distribution over the options. This "joint" reader costs one encoder pass per question, which is why recent compact decision models use it.

To test whether that choice was right, they ran a tightly controlled grid at 32M parameters: same 150,000 training questions, one epoch, same hyperparameters, teacher logits and evaluation sets for every arm, three random seeds each. They compared the joint reader against a per-option reader that encodes one sequence per option, and added two joint controls: one that matches the per-option arm's number of optimizer steps by shrinking its per-step budget, and one that matches its steps and per-step token budget (a repetition control, since it reads each row about 3.25 times). They also crossed both readers with two starting points — the Ettin reranker and the raw Ettin encoder — to see whether the result depended on the pretrained checkpoint.

Evaluation used sets never trained on: a stratified 15% sample of the public suite, 60 items per S1-Arena task, BTZSC's 500-item harness sample, and a set of option reorderings. Statistics are per-item paired bootstraps (B = 2,000) for benchmarks and Welch intervals over seeds for the grid, with Holm adjustment for the twelve Crivo-versus-DeBERTa comparisons.

For the released family, smaller models were distilled from larger ones with cross-entropy plus a Brier term and a KL term against teacher logits (α = 0.5, τ = 2 for more than two options), trained in three stages on 638,306, 169,437 and 24,635 questions. Because contamination would invalidate the external numbers, the authors applied a training firewall (dropping rows matching test states, distinctive states, or shared passages of at least 120 characters), audited the exact training files against all 563,686 suite questions with no match, and afterwards labelled each external task unseen, "seen: split" or "seen: overlap" — only calling unseen results evidence of generalisation. External benchmarks were run once per model after all models were released, never for selection or tuning.

Finally, they built Prumo, a fixed pipeline of classical components (BM25, code execution, chess search, truth tables, calendar arithmetic and others) with no learned parameters, to show what the leaderboard's tests reward without any learning at all.

Why This Matters

Research impact. The reader study challenges an implicit assumption in recent compact decision models — that joint scoring is the natural choice because it is cheaper — by showing that under matched conditions a per-option reader is better, but only when starting from a checkpoint whose pretraining format matches the per-option input. That is a reusable lesson about how pretraining format interacts with architectural choices in distillation, and it comes with a fully released grid, logs and dated pre-registration of the decision rules.

Real-world applications:

• Tool and function selection in LLM agent pipelines, where a decision must be made per step and latency matters.
• Guardrail and moderation checks (does this retrieved passage support the claim, should this action be blocked), which need calibrated probabilities rather than generated text.
• Intent routing for support tickets and chat messages, where label sets differ per deployment and cannot be covered by a single fine-tuned classifier.
• High-throughput text classification under tight compute or CPU-only serving, where the 14 ms median of the smallest model is the relevant number.

Industry relevance. The six released models span 17M to 1B parameters and were trained for roughly US$80 of rented GPU time, which puts a deployable closed-set decision model within reach of small teams. The paper also supplies a caution for anyone shipping such models: joint encoders change 12–23% of their answers when options are reordered, and their confidence is not trustworthy on inputs unlike their training data.

Future Directions

• Test the format-match explanation directly. The 2×2 shows the per-option advantage requires reranker initialisation but cannot establish why; controlled experiments with other pair-trained checkpoints, or with per-option pretraining on option-set sequences, would separate format match from other differences between the two starting points.
• Resolve the cost asymmetry. Per-option reading needs 5–13 times more passes and more than four times the mean latency, and per-option runs took about four times longer in the grid because batches were padded to the longest sequence. Batching strategies or a hybrid reader could make the accuracy gain affordable.
• Explain why the advantage appears on in-distribution but not unseen tasks. The compute-matched gain is significant overall but not on unseen S1-Arena tasks or unseen BTZSC datasets; the paper's own private-test results also show the small models near the floor on new domains, which needs a training-side answer rather than a reader-side one.
• Harden benchmarks against format-specific solving. Prumo, with no learned parameters, matches Crivo-17M on the board's same-skill tests while staying near chance externally, and the authors' development panels overestimated every private score. Better holdouts, or scoring that isolates new domains, would make leaderboard rankings more informative.

Target Audience

Practitioners building efficient NLP pipelines who need reliable closed-set decisions (routing, tool selection, entailment-style checks) rather than generated text; researchers working on distillation and compact encoders who want a carefully controlled ablation template; and benchmark designers interested in contamination audits, held-out discipline and parameter-free baselines as diagnostics. Readers looking for an apples-to-apples comparison against other authors' compact decision encoders (Bekko, Dinah-1, the tasksource decider, Mola, GLiClass) will not find one here — the paper states that these were not run on the external benchmarks and publish no comparable results.

Authors’ abstract

Many language-model pipelines end in a closed-set decision, such as selecting a tool. Prompting a large language model for it is slow and order-sensitive. We ask how small an encoder for such decisions can be and whether it should read the options jointly or separately. In a three-seed study at 32 million parameters, per-option scoring beats joint scoring on a held-out sample of a 37-benchmark decision suite by 5.8 points of chance-corrected skill at matched optimizer steps and by 2.8 at approximately matched training compute; two external benchmarks agree, though not significantly on their unseen tasks. Crossing reader with initialisation shows that the advantage requires a reranker initialisation, whose query-candidate pair input matches per-option reading; from the raw encoder the two readers are indistinguishable. Per-option scoring needs 5 to 13 times more encoder passes: at most twice the median latency, but more than four times the mean. Of six released joint encoders (17 million to 1 billion parameters), the 68- and 150-million-parameter models exceed a 435-million-parameter DeBERTa-v3 zero-shot classifier by 4.9 and 5.9 accuracy points on 209 overlap-audited System1-Arena tasks and are at least on par with it on the 145 tasks unseen in training. The smallest answers a median request in 14 milliseconds on 16 processor threads, against 728 for the classifier. Reordering options changes 12 to 23% of the joint models' answers (per-option scoring: none). Up to 68 million parameters, they lead their size range on a public leaderboard's private, mainly in-distribution tests.