LEV: Thinking Fast and Deep with Looped Decision Models

AI summary

Overview

Research area: Artificial Intelligence — efficient inference and adaptive computation for discriminative "decision models" (multiple-choice question answering without text generation).

Technical level: Intermediate to Advanced. The paper assumes familiarity with transformer inference, LoRA fine-tuning, calibration/temperature scaling, and early-exit or cascade architectures, though the core idea is explainable without the math.

Scope: The paper introduces LEV, a decision model built on natively looped language-model backbones that spends more computation on hard questions by re-running its layers rather than generating reasoning tokens, and decides per question how many loop steps to use.

What This Paper Is About

Decision models answer multiple-choice questions about an input in a single forward pass, returning one probability per option in tens of milliseconds — like the fast, automatic "System 1" of dual-process theory. Their weakness is that depth is fixed at training time, so an easy and a hard question cost exactly the same computation, while text-generating "System 2" models can think longer but pay for every extra step in decoded tokens. LEV's goal is to give a decision model per-question depth by looping its layers and reading the options after every loop step, so one checkpoint can answer at any depth and a small stopper can stop each question as soon as its answer is unlikely to change.

Key Contributions

1. LEV, a "System 1.5" decision model. LEV replaces the backbone of a standard decision model with a natively looped language model (one whose loop was trained during pretraining, not bolted on afterwards), applies a shared pointer head to the options after every loop step, and trains all steps jointly with an averaged loss — so a single set of weights serves every depth.

2. A label-free stopper for on-demand depth. Instead of the usual early-exit rule (a confidence threshold chosen on labelled data), LEV trains a per-depth logistic regression to predict whether the current answer already agrees with the model's own deepest answer, and sets its threshold on unlabelled requests from the deployment's own traffic.

3. Evidence across three looped backbones and multiple benchmarks. LEV is built on Ouro-1.4B, Ouro-2.6B (each running their stack 4 times) and Nanbeige4.2-3B (stack run twice), and evaluated on JevBench, Kev panels and LevBench — a benchmark the authors built and froze before any model read it.

4. A diagnosis of why existing compute-scaling routes fail for decision models. The paper shows experimentally that retrofitted loops on a standard pretrained backbone stay effectively closed, that cascades re-read questions and need a second model, and that generated reasoning speeds nothing up because every question pays for its reasoning text.

Main Findings

• Depth raises accuracy on the same weights, mostly on hard questions. LEV-2.6B goes from 63.2% at depth one to 76.2% at depth four on JevBench, a 13.0-point gain. On JevBench's tiers, going from depth one to depth four gains LEV-2.6B a net 10 of the 72 standard items and 19 of the 111 hard items, while the easy tier is nearly solved at every depth (of the 231 public items, 48 are easy, 72 standard, 111 hard). Of the hard items, 29 become correct and 10 become wrong.

• Diminishing returns with depth. Most of the gain arrives with the first extra step: the held-out average of LEV-2.6B rises from 51.3% at depth one to 62.0% at depth two, against 65.7% at depth four, and the fourth step of the Ouro models adds at most one point. Gains are smallest on the training sources — LEV-1.4B and LEV-2.6B gain about 7 points on Kev-Seen from depth one to four, versus 9.0 to 19.5 points on the four panels further from the training data.

• On-demand depth saves computation at almost no accuracy cost. The stopper stays within one point of the deepest readout while computing 76% to 85% of its cost, corresponding to 15% to 24% of computation saved. On the held-out panels, LEV-1.4B scores 61.0 versus 61.3 at cost 0.78, LEV-2.6B 64.6 versus 65.3 at cost 0.76, and LEV-3B 61.3 versus 62.1 at cost 0.85.

• It beats a fixed mix of depths, modestly and unevenly. Relative to the fixed-depth frontier at the same cost, on-demand LEV-2.6B is 1.4 points above on the tuning panels but only 0.2 points on the held-out panels, while LEV-3B gains 1.0 point there. The Ouro frontier is concave (first extra step brings most of the gain) whereas LEV-3B's two-step frontier is a straight line.

• A locked one-shot test confirms the result. On test partitions read once after the backbones, recipe and stopper were fixed, on-demand depth loses at most 1.0 point against the deepest readout (LEV-1.4B 59.3 vs 60.1 at cost 0.80; LEV-2.6B 63.7 vs 64.6 at cost 0.78; LEV-3B 59.8 vs 60.7 at cost 0.87) and is above the frontier for LEV-1.4B and LEV-3B, level for LEV-2.6B.

• The stopper tracks difficulty without seeing it. On JevBench, on-demand LEV-2.6B uses 1.2, 2.4 and 3.0 steps on easy, standard and hard items respectively. On LevBench it computes 66% of the deepest pass on the id tier and 75% to 87% on tiers further from the training data.

• Confidence thresholds set on labelled data fail here. The questions LEV-2.6B stops at depth one are 94.8% correct on the calibration partition but only 78.9% on Kev-Transfer at almost the same confidence. Confidence separates right from wrong answers with an AUROC of 0.67 to 0.83, but separates the wrong answers that more steps would fix from the rest with only 0.46 to 0.61. No early-exit rule the authors tried (top-probability, margin, entropy, patience-based exit) beat the frontier.

• The label-free threshold is what carries the gain. Set the same way, even a single top-probability threshold comes within 0.2 points of the full stopper, so the engineered features matter little — it is the target and the unlabelled threshold fitting that matter. A correctness target fitted on labelled calibration data falls below the frontier for both Ouro models. A few hundred unlabelled questions suffice: 500 per fold bring LEV-2.6B within 0.2 points of the full fit.

• Latency advantage below 50 ms. At 39 to 46 ms, on-demand LEV-1.4B and LEV-3B are 7.0 to 8.8 points more accurate than the only cascade that reaches that latency and 5.4 to 7.5 points above the best dense option. Aggregated, the 1.4B and 3B models are 5.4 to 8.8 points more accurate than dense models and cascades of the same latency below 50 ms. Part of this comes from a gap in the dense options, none of which answers between 22 and 79 ms. LEV-2.6B at about 150 ms is no better than Kev-4B alone, and counted in floating-point operations every LEV model is at or below the dense options.

• Retrofitted loops stay closed. Looping layers 8–15 of the 24-layer Qwen3.5-0.8B-Base (the backbone of Kev-0.8B) behind zero-initialised per-channel gates changes accuracy by less than half a point; gate learning rates up to 30×, a third pass, or a wider 16-layer block do not help. After training, gate magnitudes average about 0.006, and removing the gates lowers accuracy on new sources by 2.2 points.

• Thinking in tokens is expensive on every question. On the same DiffusionGemma 26B base, generated reasoning raises hard accuracy by 9 to 13 points but makes the median hard item three to four times slower; djev's thinking mode also got 2 of the 48 easy items wrong and returned no valid answer for 72 of 534 decisions at an 8,192-token budget.

• Joint training is nearly free; more data mainly helps shallow readouts. Jointly trained LEV-2.6B is level with single-depth training at depths two and four and at most 1.3 points behind at depth one. Extending from 31,329 to 60k and 120k records lifts depth-one accuracy from 50.5% to 62.9% and the deepest readout from 65.3% to 73.0%, cutting on-demand cost from 0.76 to 0.63. Full-weight fine-tuning lowers the deepest readout of both Ouro models by 1.3 to 1.9 points, while LoRA does not.

• Calibration is not a loop problem. Hard-tier ECE of every readout lies between 0.18 and 0.26 after temperature scaling with no trend in depth, so the gap to Kev and Jev is attributed to the single-stage recipe rather than to the loop.

Methodology in Plain English

The setup. A decision-model request contains a shared "state" plus several questions, each with a list of options wrapped in delimiter tokens and ending in a <decide> token. A block-causal mask lets each question see the state and only its own tokens, so many questions share one pass. A small "pointer head" compares the <decide> hidden state against each option's closing-delimiter hidden state and produces one logit per option.

Loop instead of generate. Instead of a standard stack of layers applied once, LEV uses a backbone pretrained to run the same block of layers several times (Ouro runs it 4 times, Nanbeige4.2 twice). The same pointer head and the same LoRA adapter read the options after every step, so one checkpoint gives one prediction per depth. Training simply averages the log loss over all steps, which makes every depth a usable decision model. Step k only depends on steps before it, so serving at depth k costs roughly k/R of the full pass.

Calibration per depth. One temperature per depth is fitted on the calibration partition of the training suite, never on a benchmark; since temperature scaling does not change the argmax, accuracy at each depth is unaffected.

Deciding when to stop. After each step, a small logistic-regression stopper looks at features derived from the current and previous readouts (top probability, margin between the top two options, normalised entropy, whether the answer changed, KL divergence between consecutive distributions, log of the number of options, and the step index). It predicts whether the current answer matches what the deepest step would say. A question stops at the first step where the predicted probability crosses a threshold τ; otherwise it runs to the deepest step. A request's whole state must run to the deepest depth any of its questions needs.

How the threshold is chosen without labels. Because the target is the model's own deepest answer, no ground-truth labels are needed. The authors pick the smallest τ (i.e. the cheapest stops) under which at least 98% of final answers agree with the deepest answers on the fitting questions. In experiments they emulate deployment traffic with two folds of each test panel split by record group.

Baselines and controls. Dense controls (Qwen3-1.7B-Base, Qwen3.5-2B-Base, Qwen3.5-4B-Base) are trained on identical data with the same recipe — LoRA rank 16, one epoch, 31,329 records, peak learning rate 1e-4 — so the comparison isolates the loop rather than the data. Released Kev-0.8B, Kev-4B and the closed Jev 1.13.0 are reported as references trained differently. Latency is median per request on one H200 GPU in bf16; confidence intervals are 95% paired bootstrap intervals resampling groups of related questions.

Why This Matters

Impact on research. The paper reframes depth as a serving-time setting rather than a training-time property for decision models, and provides a concrete negative result: loops retrofitted onto a standard pretrained backbone stay closed and unused, so the loop must be pretrained. It also shows that for these models the standard early-exit recipe (labelled confidence threshold) underperforms, and that predicting agreement with the deepest answer on unlabelled traffic is a workable substitute — a finding that transfers to other adaptive-computation settings where labels are scarce.

Real-world applications:

• High-throughput classification and routing services (support ticket triage, intent labelling, content moderation queues) where most items are easy and a small tail is hard, and where 15% to 24% compute savings at under one accuracy point matters at scale.
• Latency-bounded deployments such as interactive search or agent tool-selection, where LEV's sub-50 ms operating points are 5.4 to 8.8 points more accurate than dense models and cascades at the same latency.
• On-premise or edge serving, where a single checkpoint covering several operating points reduces the number of models that must be hosted and maintained.
• Deployments that cannot share labelled data, since the stopper's threshold can be fitted on the operator's own unlabelled request stream — the paper notes a few hundred questions suffice.

Industry relevance. The savings are counted in loop steps, not wall-clock time; the authors state explicitly that a serving path batching requests at different depths would be needed to realise them as latency. Fused kernels, graph capture and state caching are also absent for all systems compared. The practical claims are therefore strongest for accuracy-at-fixed-latency in single-request measurement, which is exactly the regime the paper measures.

Future Directions

• A better stopping target. 12.6% of LevBench questions are answered correctly at some shallower depth but wrongly at the deepest one; a stopper that also recovers these could beat the deepest readout rather than merely matching it.

• Scaling the recipe. The study covers looped backbones of 1.4B to 3B parameters, the sizes publicly available; whether the hard-question gains grow with scale is untested.

• Turning step savings into wall-clock savings. The reported cost is in loop steps; batching requests at mixed depths, plus fused kernels, graph capture and state caching, is left as engineering work.

• Broader evaluation. JevBench results rest on its 231 public items, and the authors call for further decision benchmarks and for comparisons on optimised serving stacks for both looped and dense models.

Target Audience

Researchers and engineers working on efficient inference, adaptive computation, early-exit and cascade architectures, or looped/recurrent-depth transformers will get the most from this paper, as will practitioners building low-latency multiple-choice or classification services who need per-request latency-accuracy trade-offs from a single deployed checkpoint. Readers interested in calibration and label-free threshold fitting on production traffic will also find the stopper analysis directly useful. A basic grasp of transformer inference and fine-tuning is assumed.

Authors’ abstract

Decision models answer typed questions about an input with one probability per option, computed in a single forward pass without generating text. Like the fast System 1 of dual-process theory, they answer at once, but their depth is fixed at training time, so they spend the same computation on easy and hard questions. Models that reason in generated text can think longer, but they pay for every extra step in decoded tokens. We present LEV, a decision model that thinks deeper by rerunning its layers in a loop instead of generating tokens. LEV builds on a language model pretrained as a loop, reads the options after every loop step and trains all steps jointly, so one checkpoint can answer at any depth. On three looped backbones, accuracy rises with depth on the same weights, mostly on hard questions, and a 2.6B LEV goes from 63.2% at depth one to 76.2% at depth four on JevBench. On-demand depth, set per question by a small stopper with a threshold fitted on unlabelled requests, saves 15% to 24% of the computation at a cost of under one accuracy point, level with or slightly above a fixed mix of depths. On a locked test set, the 1.4B and 3B models are 5.4 to 8.8 points more accurate than dense (non-looped) models and cascades of the same latency below 50 ms.