Confidence-Calibrated Inference Expansion for Evaluator-Guided Test-Time Reasoning

AI summary

Overview

Research area: Adaptive test-time compute for large language model reasoning — specifically, sequential decision-making about when to stop, fall back, or buy another unit of inference under evaluator (verifier or judge) guidance.

Technical level: Intermediate. The paper combines a formal finite-horizon control formulation and regret analysis with a lightweight learned controller; readers need comfort with expected-value reasoning and basic calibration concepts, but the empirical story is accessible.

Scope: The paper reframes evaluator-guided reasoning as a three-action, compute-priced control problem (trust, fallback, expand), analyzes when confidence errors flip decisions, and evaluates the resulting calibrated controller on verifier-guided width expansion (MATH, AIME25) and judge-guided revision (a 600-example OpenThoughts-derived benchmark) across three backbones.

What This Paper Is About

Test-time reasoning methods — sampling more candidates, reranking with a verifier, or running extra critique-and-revision rounds — generally improve average accuracy as more inference compute is spent, but they rarely ask whether the next unit of compute is worth its cost for this particular problem instance. This paper turns that question into an explicit sequential decision: at each intermediate state, should the system emit the evaluator-preferred answer, emit a conservative fallback answer, or spend one more inference step and decide again? The goal is a controller that reaches comparable answer quality at substantially lower online compute, by predicting action-specific success probabilities, calibrating them, and subtracting an explicit price for computation.

Key Contributions

1. A compute-aware three-action formulation. Evaluator-guided reasoning is cast as a controller over trust, fallback, and expand, with terminal actions scored by predicted answer success minus cost and expand scored by a Bellman continuation value minus immediate cost. The abstraction covers both width expansion (more candidates) and revision expansion (another critique-and-revision round), and is explicitly distinguished from selective prediction and reject-option classification, since low confidence need not imply abstention.

2. Action-gap and trajectory-regret analysis. Lemma 1 shows that if twice the local score error is smaller than the compute-adjusted action gap (2δ_J(s) < Γ*(s)), the deployed policy selects the ideal action. Theorem 1 bounds finite-horizon regret by 2δ_J(s) plus an expected continuation regret triggered only when the policy expands, yielding a trajectory bound of twice the expected sum of per-state score errors along states the policy actually visits. Theorem 2 gives a 1 − α conditional version under visited-state coverage, and the appendix adds results on certified local optimality, when another revision round helps, mis-selection under sub-Gaussian evaluator noise, noise amplification from taking maxima over candidates (O(σ√log n)), and the limits of a single scalar confidence score.

3. A lightweight calibrated controller. One sigmoid confidence head per action over inference-time-only features (evaluator score, score margin, disagreement summary, agreement/rollback support, recent progress, expansion level, normalized budget), trained with binary cross-entropy, calibrated post hoc per action on a disjoint split, with an optional local uncertainty penalty estimated from calibration residuals over coarse buckets.

4. Empirical evaluation of compute allocation. Held-out comparisons against fixed and maximum budgets, threshold and rollback heuristics, raw and calibrated confidence, budget-conditioned and receding-horizon value models, and a fitted-Q controller, plus ablations on controller components, expansion targets, target corruption, and compute-price retuning.

Main Findings

• Lower compute at comparable utility in both settings. At the nominal price of λ = 0.02, the controller improves observed utility over calibration-selected fixed policies while reducing normalized online compute by about 30% (width) and 16% (revision). In width expansion it reaches utility 0.715 at compute 0.547, versus 0.707 at compute 0.781 for the best fixed policy (majority vote at N = 32), and matches majority vote at N = 48 (utility 0.715) while using 0.453 less normalized compute. In revision it reaches utility 0.673 at compute 0.529, versus 0.668 at compute 0.632 for the best fixed-depth policy.

• Accuracy is not sacrificed for the headline width result. The controller's held-out accuracy is 0.824 on MATH and 0.628 on AIME25, compared with 0.821 / 0.625 for the best fixed policy and 0.824 / 0.646 for majority vote at N = 48. Calibrated confidence alone reaches utility 0.714 at compute 0.586; the uncalibrated variant reaches 0.701 at 0.611.

• Calibration measurably improves every action head. ECE for the trust / fallback / expand heads drops from 0.061 / 0.066 / 0.084 to 0.024 / 0.029 / 0.041 in width, and from 0.069 / 0.073 / 0.096 to 0.031 / 0.035 / 0.052 in revision. The expand head retains the largest calibration error.

• Seed-level compute advantage is consistent, utility advantage is not. Across four Qwen3-8B generation seeds against fitted Q, compute is lower in all eight seed–scenario comparisons: width ΔC = −0.037 with 95% interval [−0.055, −0.020], revision ΔC = −0.027 with [−0.040, −0.014]. Utility intervals include zero in both settings (width ΔU = −0.001, [−0.006, 0.004]; revision ΔU = +0.002, [−0.003, 0.007]), with utility favoring the controller in 2 of 4 width seeds and 3 of 4 revision seeds.

• Both controller components matter. Separate per-action calibration adds 0.004 utility in width and 0.006 in revision over shared predictive weights; the local uncertainty penalty adds a further 0.001 and 0.005 while reducing compute by 0.039 and 0.050. Retaining the fallback action adds 0.006 utility in width and 0.010 in revision over a trust/expand-only controller.

• The continuation target choice matters less than expected, but matters. Policy-induced continuation targets reach utility 0.714 in width and 0.669 in revision, close to oracle-aligned targets (0.715 and 0.673); one-step targets reach 0.709 and 0.665. Corruption of the joint fit-and-calibration split from 10% to 30% lowers utility from 0.709 to 0.689 in width and from 0.664 to 0.642 in revision.

• Behavior tracks difficulty and scale. Width compute falls from 0.644 for Qwen3-4B to 0.434 for DeepSeek-R1-Distill-Qwen-14B; revision compute falls from 0.606 to 0.446 and mean rounds from 1.680 to 0.940. On MATH, initial expansion rises from 0.18 at Level 1 to 0.57 at Level 5, and mean expansions rise from 0.62 to 1.61.

• Decisions are concentrated near brittle boundaries. Evaluator margin correlates with the empirical controller action gap at 0.426 in width and 0.368 in revision; 62% (width) and 59% (revision) of action changes occur in small-gap states. Calibration most often changes expand to trust in width and expand to fallback in revision.

• The operating point is price-dependent. When compute is free, width utility is 0.003 below the best fixed reference, while revision utility is 0.005 above maximum-round revision. At λ = 0.02, one-shot budget-conditioned and receding-horizon value baselines reach 0.711/0.602 and 0.716/0.565 in width, and 0.665/0.581 and 0.671/0.548 in revision, versus the three-action controller's 0.715/0.547 and 0.673/0.529.

• Revision quality is non-monotone in rounds. Full-cohort revision success rises from 0.603 at zero rounds to 0.681 at two, falls to 0.673 at three, and reaches 0.688 at four — supporting the paper's argument that extra inference can corrupt correct drafts.

• Offline overhead amortizes. Estimated break-even points range from roughly 3×10³ to 2×10⁴ queries, depending on the fixed-policy comparator.

Methodology in Plain English

The work starts by refusing to treat reasoning as one-shot. At any moment the system has a current answer (the one an evaluator likes best), a safe backup answer (majority vote across sampled candidates in width expansion, or a rollback draft in revision), and the option to spend more compute. Each of these is scored as a probability of eventual correctness minus a price times the compute it consumes.

Because a raw verifier score is not a probability of being right, the authors train three small logistic-style predictors — one per action — on features available only at inference time, such as the evaluator's score, how far ahead the top candidate is, how much the candidates disagree, and how much budget has been used. Training targets come from offline "ladders" that run the full expansion process to completion: for trust and fallback, whether the emitted answer was actually correct; for expand, whether the continuation eventually succeeded. Each head is then calibrated on a separate split so its output behaves like a real probability. For expand, the authors are explicit that this is a success-based surrogate, not a direct Bellman value, because it does not include the cost of downstream computation — the immediate expansion cost is subtracted separately.

To avoid flipping decisions in states where two actions are nearly tied, they estimate a per-action uncertainty radius from calibration residuals grouped into coarse buckets by expansion level, margin, uncertainty, and budget, and subtract a tuned multiple of it from each action's score. Setting that multiplier to zero recovers the plain calibrated controller, which is one of the ablations.

Evaluation uses the objective U = S − λC with λ = 0.02, where C is total online prompt and generation compute normalized by a max-budget baseline, and utility is recomputed from unrounded success and compute. Success is scored by reference matching independent of the revision judge. Baselines are selected on calibration data under the same objective, and offline ladder generation is excluded from per-query cost.

Why This Matters

Impact on research. The paper argues against the implicit assumption that evaluator-guided reasoning is simply "a uniformly stronger reasoner" and reframes it as resource allocation under uncertainty. The action-gap and regret results give a concrete reliability target for a controller — the post-pricing, post-penalty action score, not a raw verifier margin — and explain why a well-calibrated single correctness score can still be insufficient when different actions induce different success events. The negative-leaning seed results on utility also set a realistic bar: the compute savings are robust, the quality gains are not.

Real-world applications (implications, not claims evaluated in the paper):

• Cost governance for deployed reasoning models, where per-query inference budgets must be enforced without a fixed, hand-tuned stopping rule.
• Mathematical and quantitative answer services, where candidate sampling plus verifier reranking is already standard and majority vote is a natural fallback.
• Iterative drafting pipelines with judge feedback — code repair, document editing, structured extraction — where extra critique rounds can corrupt already-correct drafts.
• Systems that need a conservative answer under uncertainty rather than an abstention, since fallback emits an answer instead of declining.

Industry relevance. The controller is deliberately lightweight: three small heads over observable features plus a bucket lookup, with decision overhead negligible relative to generation, verification, judging, and revision. That matters for latency-sensitive serving, and the explicit compute price λ gives operators a single dial for trading quality against cost. The paper is candid that offline ladder construction is required for fitting and evaluation but not for online execution, and that reported break-even points of roughly 3×10³ to 2×10⁴ queries mean the setup cost amortizes only under a stable model and prompt configuration.

Future Directions

• Broader task and scale coverage. The evaluation covers three backbones, MATH/AIME25 width expansion, and a curated deterministic-answer revision benchmark; open-ended generation and long-horizon interaction remain untested, as does behavior at model scales beyond those studied.

• Robustness to evaluator shift. Generalization to unseen evaluator shifts is stated as unknown, and changes to the model, evaluator, or prompts require refitting and recalibration — a natural target for methods that transfer or adapt without full refits.

• Closing the continuation-target gap. The expand head is trained on a binary continuation-success surrogate that omits downstream compute costs, and the theoretical coverage argument does not by itself establish coverage of the Bellman score error for expand. Directly estimating or bounding that mismatch is an open problem the paper flags.

• Better handling of small-gap states. Since the majority of action changes occur where the compute-adjusted action gap is small and the expand head carries the largest calibration error, sharper local uncertainty estimates or gap-aware decision rules are an obvious next step.

Target Audience

Researchers and practitioners working on test-time compute, inference-time scaling, verifier-guided decoding, and LLM system design will get the most from this paper. It suits readers interested in the theory side (action-gap conditions, regret bounds along realized trajectories) as well as engineers who want a concrete, cheap controller and an honest accounting of where it helps and where it does not. Readers looking for a uniformly stronger reasoning method should note the paper's own framing: the utility gains are small and seed-dependent, while the compute reductions are consistent.

Authors’ abstract

Evaluator-guided reasoning presents a local control problem at every step: trust the current answer, return a fallback, or spend more inference. We formulate this choice as a compute-aware three-action controller, scoring terminal actions by predicted answer success and expansion by a rollout-based continuation-success surrogate, with compute cost priced explicitly. A local action-gap analysis characterizes when score errors can change a decision, while an expected regret bound accounts for errors along the realized expansion path. Guided by this analysis, we learn and calibrate action-specific confidence estimates, with local uncertainty penalties for ambiguous decisions. On held-out width expansion and revision tasks, the controller improves observed utility over calibration-selected fixed policies while reducing normalized online compute by about 30% and 16%, respectively. Across four Qwen3-8B seeds, compute is lower than fitted Q in both settings, while utility differences are small and mixed. The results frame evaluator-guided reasoning as resource allocation under uncertainty, rather than a uniformly stronger reasoner.