StartLux-Decision: AutoThink Decision Models with Reinforcement Learning for Calibrated Deliberative Decisions

AI summary

Overview

• Research area: Natural language processing / large language model decision-making, with reinforcement learning for reasoning control and probability calibration.
• Technical level: Intermediate (accessible with basic familiarity with LLMs, softmax probabilities, and reinforcement learning).
• Scope: The paper introduces and evaluates StartLux-Decision, a family of decision models from 0.8B to 27B parameters offering a single-pass "Instant" variant and a reasoning-capable "AutoThink" variant trained with a new reinforcement learning method, RLCD².

What This Paper Is About

Decision models answer typed questions about an input (such as a routing or tool-call decision) by outputting probabilities over candidate options. Existing open-weight decision models mostly make a single forward pass with no explicit reasoning, which is efficient but struggles on computation-heavy decisions like arithmetic, scheduling, and procedural rule checking. The paper's goal is to build decision models that reason only when they are uncertain, that keep their probabilities well calibrated for downstream routing and selection, and that do not ramble on for thousands of tokens when reasoning is invoked.

Key Contributions

1. A two-variant model family (0.8B–27B parameters). "Instant" models use bidirectional attention in full-attention layers to improve single-pass decision accuracy; "AutoThink" models add a confidence-gated "Read-or-Think" mechanism that answers confident cases directly and reasons only when the initial maximum probability falls below a threshold of 0.9, combining the initial and post-reasoning distributions with weight 0.75 on the post-reasoning readout.
2. A three-stage training pipeline. Stage 1 is full-parameter supervised fine-tuning on the Lux-Release mixture (approximately 2.7 million examples for 0.8B–9B models; approximately 0.8 million for the 27B model), Stage 2 trains adapters on the Lux-Rounds mixture (approximately 1.7 million examples) with the Instant/AutoThink specialization, and Stage 3 adds reasoning supervision on Lux-Think (approximately 4,000–21,000 reasoning examples per model) followed by reinforcement learning.
3. RLCD² (Reinforcement Learning for Calibrated Deliberative Decisions). A decision-oriented RL method that rewards the change in probabilistic decision quality produced by a reasoning trace, measured with proper scoring rules (Brier, logarithmic, and a decision-weighted score using a mixture of a uniform density and a Beta(8,2) density) computed by a frozen copy of the starting model, plus a soft length penalty (λ = 10⁻⁴) beyond a token budget.
4. DecisionPivot, a programmatically verifiable benchmark of computational decisions built from contrast pairs that differ in a single answer-changing detail, requiring both members of a pair to be correct, together with policy-aware calibration, which calibrates confidence under the deployed selection or routing policy rather than per prediction.

Main Findings

• Instant models lead at their scale on Decision Index 0.3. On the public subset of Decision Index 0.3, the Instant models outperform the strongest evaluated comparable-scale open-weight decision models by 7.4–16.9 points, the 4B Instant model surpasses all evaluated reference decision models, and the 27B AutoThink model raises its score to 73.4.
• Bidirectional attention helps single-pass decisions. In controlled comparisons from the same Stage 1 checkpoint with identical data, bidirectional attention in full-attention layers (8 of 32 layers at 4B and 9B) improved Decision Index 0.3 performance at four of the five evaluated scales, with gains of up to 2.4 points; runs with identical configurations varied by 0.2–0.7 index points across seeds.
• Reasoning helps most on computation-intensive decisions. On DecisionPivot, AutoThink outperforms its Instant counterparts by 11.5–18.4 percentage points in pair accuracy, with the 27B AutoThink model reaching 86.9% and leading all evaluated reference systems.
• Selective reasoning corrects errors without always reasoning. Across 7,800 core DecisionPivot items, the 27B AutoThink model answered 39% of questions directly at 96% accuracy; on the 61% where reasoning was invoked, accuracy rose from 57% before reasoning to 93% after fusion, correcting 1,783 initial errors while introducing 94 new ones.
• RLCD² dramatically improves reasoning termination and length. Termination within the generation limit rose by up to 48 percentage points and average reasoning length fell by up to 65%. At 9B, termination went from 45.9% to 93.9% and mean trace length from 2,494 to 868 tokens. At 4B, termination went from 89.5% to 99.3%, mean length fell 37%, and the 90th-percentile trace length fell from 4,096 to 1,358 tokens. The 27B model already terminated 97.2% of traces at a mean of 313 tokens.
• Decision accuracy is largely preserved through RL. On 2,986 held-out examples, default-answer accuracy changed from 74.35 to 74.82 (4B), 80.38 to 80.48 (9B), and 79.07 to 79.54 (27B); the paper reports uncertainty intervals for these changes that include zero.
• Calibration gains depend on scale. At 4B, raw post-reasoning calibration error fell from 9.0 to 8.1 percentage points (interval [−1.7, −0.1]), though after temperature scaling the errors were comparable at 3.1 and 3.2 points; at 9B and 27B, calibration changes were within statistical uncertainty.
• Benchmark gains appear at some scales. The 9B model's DecisionPivot pair accuracy rose from 53.3% to 59.9% under the 3,072-token budget (interval [+5.0, +8.3]).
• The reasoning budget trades accuracy against efficiency. For the 4B model before RL, traces averaged approximately 2,100 tokens on DecisionPivot with 60.4% pair accuracy; a 512-token budget roughly halved length but dropped accuracy to 52.9%, while 2,048 and 3,072 tokens restored it to 59.2% and 59.9%. Removing the length penalty reached 61.7% but produced 40% longer traces, with approximately one third failing to terminate versus about one tenth under the 3,072-token budget.
• Gains and losses split by computation length. At 4B with a 2,048-token budget, accuracy improved by 2.5 points on examples whose pre-RL traces were under 512 tokens but fell by 5.9 points on those with 2,048–4,095-token traces; a 3,072-token budget reduced that loss to 4.2 points. At 9B, previously unterminated traces accounted for 54% of DecisionPivot examples and improved by 7.3 points, while long-trace examples still lost 3.4 and 5.8 points under the 3,072 and 2,048 budgets.
• The length penalty and the predictive reward do different jobs. Including the length penalty produced nearly universal termination even when the predictive reward was replaced with random noise; calibration improvements required the proper-score reward, proper-score rewards outperformed accuracy-only rewards, and a termination bonus gave no calibration improvement.
• Per-prediction calibration does not transfer to selection policies. Predictions assigned 80–90% confidence by the 9B verifier achieved 92% accuracy when evaluated individually but only 78% after selecting the highest-scoring response from 32 candidates; policy-aware calibration reduced confidence–accuracy gaps to within 2.7 percentage points across 184 settings spanning four decision pipelines.
• Decision-oriented training damages reasoning before it is repaired. The decision-trained 4B model produced little or no reasoning for approximately 40% of questions it was prompted to reason on, while the 9B model kept generating to the length limit on approximately 70% of such questions.

Methodology in Plain English

The model is asked a typed question about an input and must distribute probability across the candidate options. Instead of a separate prediction head, the authors reuse the language model's own output layer: each option is labeled with a letter (A–Z), and the model's logits for those letters at the end of the prompt become the decision probabilities after temperature scaling. Each readout handles up to 26 options, and longer lists are split into groups of at most 25 with the top three candidates from each group advancing to a final round. The models handle contexts up to 262,144 tokens and pass images through a frozen pretrained vision tower.

Training happens in three stages. First, the whole model is fine-tuned on a broad mixture of public datasets recast as decisions, with candidate options deterministically permuted to fight position bias and with test-set overlap removed. Second, lightweight adapters are trained on a wider mixture and merged back in; the Instant variants additionally get bidirectional attention in their full-attention layers so that earlier prompt positions can see the question and options that follow, while linear-attention layers stay causal. Released Instant models average the weights of three independent runs at 0.8B and four runs at 2B–27B. Third, the AutoThink variants are taught to reason using a few thousand supervised reasoning traces and then optimized with RLCD².

The RL step works like this: a frozen copy of the starting model scores the decision probabilities both before and after a sampled reasoning trace, and the improvement in a proper scoring rule becomes the reward. A small per-token penalty applies only after a soft budget is exceeded, so the model is nudged toward shorter traces without being hard-truncated. Training uses group-relative policy gradients with advantages clipped to ±5, a KL term toward the starting policy, and no division by the within-group standard deviation; only questions whose initial maximum probability is below 0.9 are trained on, matching the inference gate. Traces under eight tokens count as empty reasoning and receive zero reward. At inference, AutoThink computes an initial distribution, answers directly if its maximum is at least 0.9, and otherwise reasons up to 4,096 tokens and blends the two distributions with weight 0.75 on the post-reasoning one.

Why This Matters

Decision models sit underneath request routing, retrieval assessment, tool-call validation, and agent outcome verification, where a probability—not just a label—determines whether a system acts, escalates to a stronger model, or asks a human. This paper shows that reasoning can be added to such models without giving up the cheap single-pass path, and that reasoning length can be controlled by optimizing for decision quality rather than answer correctness alone.

Real-world applications implied by the paper:

• Agent pipelines: validating tool calls and verifying agent outcomes, where calibrated confidence decides whether to proceed or defer.
• Request routing and model cascades: using confidence to send easy inputs to a small model and hard ones to a stronger model.
• Retrieval assessment: judging whether retrieved evidence is sufficient before generating an answer.
• Policy and compliance checking: procedural rule interpretation of the kind DecisionPivot tests, such as data-export approval decisions.

Industry relevance is reinforced by deployment support across vLLM, llama.cpp, and MLX, quantized variants for lower memory, open model weights, and published inference code, which makes the models usable on a range of hardware rather than only in a research setting.

Future Directions

• Fix the long-computation regression. RLCD² improves accuracy on previously unterminated traces but reduces accuracy on examples with very long pre-RL traces; finding rewards or curricula that help long computations instead of shortening them is an open problem.
• Make calibration portable. The gap between individually calibrated predictions (92% accuracy at 80–90% confidence) and post-selection accuracy (78% out of 32 candidates) suggests policy-aware calibration needs to be extended and validated across more selection and routing policies.
• Scale up the reasoning budget studies. The reported trade-offs come from budgets between 512 and 3,072 tokens on a 4B model and specific budgets at 9B and 27B; whether the same conclusions hold at other scales, budgets, and reasoning lengths is not established here.
• Broaden the evaluation of deliberative reasoning. The reasoning-specific evidence leans heavily on DecisionPivot's 7,800 core items and a 2,986-example held-out set; independent evaluation on other computation-intensive decision benchmarks would test generality.

Target Audience

Researchers and engineers working on LLM-based decision systems, model routing and cascades, agent tooling, and reinforcement learning for reasoning. It is also useful for practitioners who need calibrated probabilities in production and want a concrete template for confidence-gated reasoning, and for readers interested in how proper scoring rules can replace answer-correctness rewards in RL.

Authors’ abstract

We introduce StartLux-Decision, decision models from 0.8B to 27B parameters that combine efficient single-pass prediction with adaptive, confidence-guided reasoning. The Instant models use bidirectional attention in full-attention layers to improve decision accuracy while keeping single-pass inference. The AutoThink models invoke explicit reasoning for uncertain decisions and answer confident cases directly. A three-stage training pipeline establishes shared decision capabilities, specializes the two variants, and equips AutoThink with reasoning through reasoning supervision and reinforcement learning. We introduce Reinforcement Learning for Calibrated Deliberative Decisions (RLCD$^2$), which uses a frozen model and proper scoring rules to reward improvements in decision probability quality and applies a soft penalty to overlong reasoning traces. RLCD$^2$ raises the share of reasoning traces that terminate within the generation limit by up to 48 percentage points and cuts average reasoning length by up to 65%, while largely preserving decision accuracy. On the public subset of Decision Index 0.3, the Instant models outperform the strongest evaluated comparable-scale open-weight decision models by 7.4-16.9 points, and the 27B AutoThink model reaches 73.4. We also introduce DecisionPivot, a programmatically verifiable benchmark of computational decisions built from contrast pairs that differ in a single answer-changing detail. On DecisionPivot, AutoThink outperforms its Instant counterparts by 11.5-18.4 points in pair accuracy and leads among evaluated systems. The models run on vLLM, llama.cpp and MLX, with quantized variants for lower memory use.