Logit-Entropy Adaptive Stopping Heuristic for Efficient Chain-of-Thought Reasoning

AI summary

Overview

• Research area: Natural Language Processing, specifically efficient inference and chain-of-thought (CoT) reasoning in large language models.
• Technical level: Intermediate. The method is conceptually simple but requires familiarity with token logits, softmax probabilities, entropy, and decoding settings such as nucleus sampling.
• Scope: The paper introduces LEASH (Logit–Entropy Adaptive Stopping Heuristic), a training-free decoding rule that halts chain-of-thought rationale generation when two intrinsic signals plateau, and evaluates it against vanilla CoT and direct-answer decoding on four instruction-tuned models over two math reasoning benchmarks.

The paper appears as arXiv:2511.04654v1 [cs.CL], dated 06 Nov 2025, under a workshop title "Efficient Reasoning."

What This Paper Is About

Chain-of-thought prompting improves reasoning accuracy by making models "think out loud," but generating a full fixed-length rationale costs tokens and adds latency. The authors' goal is to decide, per question and during decoding, when a model has reasoned "enough" so that generation can stop early without collapsing accuracy. They propose a rule that watches how the model's own next-token uncertainty and confidence change over time, and stops when those signals flatten out.

Key Contributions

1. A training-free adaptive stopping rule. LEASH monitors the windowed slope of token-level entropy and the improvement in the top-two log-probability margin, and halts rationale generation when both plateau within a sliding window, after a minimum warm-up length.
2. A concrete stopping criterion with safeguards. The criterion combines a per-step plateau test, a majority vote over the last L non-saturated steps, a saturation guard that excludes highly confident steps from trend analysis, and an entropy-drop gate that prevents premature stops.
3. A model-agnostic, drop-in design. LEASH reuses logits already computed by the base model, adds O(1) overhead per token in time and memory using ring buffers, requires no extra training, verifiers, or auxiliary heads, and works with greedy or sampled decoding and quantized or full-precision inference.
4. An empirical characterization of the accuracy/efficiency trade-off. The paper reports accuracy, token reduction, and latency reduction for LEASH versus vanilla CoT and versus direct-answer (No-CoT) decoding across four instruction-tuned models.

Main Findings

• Token and latency savings: On GSM8K, LEASH reduced generated tokens by an average of 35.3% and end-to-end latency by an average of 26.9%; on AQuA-RAT the average reductions were 29.8% in tokens and 27.6% in latency.
• Per-model efficiency (GSM8K): Llama-3.1-8B-Instruct showed 30.97% token reduction and 29.74% latency reduction; Mistral-7B-v0.1 35.12% and 27.80%; Phi-3-Mini-128k-Instruct 41.50% and 25.15%; Qwen2.5-7B-Instruct 33.45% and 24.90%.
• Per-model efficiency (AQuA-RAT): Llama-3.1-8B-Instruct 28.60% token and 26.10% latency reduction; Mistral-7B-v0.1 34.20% and 27.50%; Phi-3-Mini-128k-Instruct 28.30% and 28.75%; Qwen2.5-7B-Instruct 28.15% and 28.10%.
• Largest token saving: Phi-3-Mini-128k-Instruct on GSM8K, at 41.5% token reduction.
• Accuracy cost (GSM8K): LEASH scored 62.32 versus CoT's 74.33 for Llama-3.1-8B-Instruct, 38.67 versus 47.20 for Mistral-7B, 69.87 versus 82.67 for Phi-3-Mini-128k-Instruct, and 54.85 versus 65.33 for Qwen2.5-7B-Instruct.
• Accuracy cost (AQuA-RAT): LEASH scored 54.68 versus CoT's 63.20 for Llama-3.1-8B-Instruct, 19.25 versus 26.38 for Mistral-7B, 50.24 versus 61.67 for Phi-3-Mini-128k-Instruct, and 68.15 versus 77.35 for Qwen2.5-7B-Instruct.
• Average accuracy drop: The paper reports an average drop of approximately 10.9 percentage points on GSM8K and approximately 9.1 percentage points on AQuA-RAT relative to CoT, and describes the abstract-level figure as roughly a 10 p.p. accuracy drop.
• LEASH still beats no reasoning: It substantially outperforms No-CoT in all cases. On GSM8K, Llama-3.1-8B's LEASH accuracy (62.32%) is stated to be 4.4× its No-CoT accuracy (14.00%), and Mistral-7B's (38.67%) is stated to be 6.1× its No-CoT accuracy (6.33%).
• Latency consistency: Latency savings were highly consistent, especially on AQuA-RAT, where all four models clustered in a 26–29% reduction range. The paper gives a concrete example: for Llama-3.1-8B-Instruct, latency fell from 4.04s (CoT) to 2.84s, a 29.7% speed-up with 31.0% fewer tokens.
• Retained accuracy framing: The introduction states that LEASH retains approximately 85% of vanilla CoT accuracy on GSM8K.
• Fixed hyperparameters: LEASH hyperparameters were held fixed across models unless otherwise noted, suggesting it works without per-task retuning; the authors also report robustness across sampling temperatures and decoding settings.

Methodology in Plain English

The authors treat stopping as an online decision made at every generated token. During the rationale phase, they disable end-of-sequence termination so the model cannot stop on its own; instead, their rule decides when to cut generation short. After the cut, the model is prompted a second time to emit only a short final answer.

At each step, they take the model's next-token logits (the raw scores over the vocabulary) and convert them into probabilities and log-probabilities. From these they compute two signals that together describe whether the model is settling on an answer:

• Entropy, which measures how spread out the probability mass is — that is, how uncertain the model is.
• The margin between the top two token log-probabilities, which measures how far ahead the leading candidate is.

They then look at how these signals change over a sliding window of the last k steps: the entropy slope and the margin improvement. Separately, steps where the peak token probability already exceeds a threshold are marked "saturated" and excluded from the trend analysis, so that very confident steps do not distort the picture.

A step passes the plateau test if the entropy slope has flattened (it is at or above a small negative tolerance) and the margin improvement has stalled (at or below a small positive tolerance), and the step is not saturated. The model stops when three conditions hold at once: a minimum warm-up length has passed; a majority of the last L non-saturated steps passed the plateau test; and entropy has dropped by at least a threshold amount relative to a reference value computed as the median entropy over the first k steps. A maximum length caps generation regardless.

Because the signals reuse logits already produced during normal decoding, the approach requires no extra forward passes for auxiliary models, no training, and no gradient updates. The only added cost is maintaining ring buffers of the last k entropy and margin values, which the authors describe as O(1) per token in time and memory.

Experimental setup. The authors evaluated four instruction-tuned models: Llama-3.1-8B-Instruct, Mistral-7B-v0.1, Phi-3-Mini-128k-Instruct, and Qwen2.5-7B-Instruct. They used GSM8K as the primary benchmark, evaluating on a randomly sampled subset of n = 300 test problems with a fixed seed, and additionally reported results on the test split of AQuA-RAT (the paper does not state the number of AQuA-RAT problems used). Baselines were Vanilla-CoT and No-CoT. For the rationale phase they used nucleus sampling with p = 0.95 and temperature 0.7; the final answer was decoded at temperature 0.0. Hyperparameters were fixed across models: window k = 8, consistency L = 5, entropy slack ε_H = 0.005, margin slack δ_M = 0.05, and minimum/maximum rationale lengths m = 64 and M = 320. Metrics were exact-match accuracy after normalization, percent token reduction relative to CoT (counting rationale plus answer tokens), and percent end-to-end latency reduction per example.

Why This Matters

Impact on research. The paper frames adaptive stopping as an alternative to fixed token budgets, brittle surface heuristics (such as stopping on "Therefore" or punctuation), multi-sample reranking, and auxiliary heads or verifiers. Its distinctive claim is that token-level signals already available under standard top-p sampling are sufficient to detect reasoning convergence, without the answer-entropy-threshold approach used in prior work. It offers a lightweight baseline that future efficiency work can compare against, and it motivates theoretical analysis of when stopping is safe.

Real-world applications:

• Interactive assistants and chat systems, where end-to-end latency per response directly affects user experience and a roughly one-quarter latency reduction matters.
• Cost-capped or high-volume LLM deployments, where token usage is billed and reductions of roughly 30–35% translate into direct savings on math and word-problem style workloads.
• On-device or quantized inference, since the method operates purely on logits, needs no extra model, and the paper states it is compatible with quantized or full-precision inference.
• Toolchains already built on standard decoding APIs, where a drop-in stopping rule can be added without architectural changes or retraining.

Industry relevance. The savings reported are per-instance and require no supervision or reward models, which lowers the barrier to deployment. However, the trade-off is explicit: the authors report roughly a 10 percentage-point accuracy drop relative to full CoT, so the method targets settings where compute and latency budgets matter more than peak accuracy.

Future Directions

• Extending beyond short numeric answers. The authors note that LEASH is evaluated on short-answer math tasks and that long-form, non-numeric targets are an open direction.
• Tool-augmented settings. The paper identifies integration with tool use as a promising extension, since stopping decisions would need to interact with external calls.
• Theoretical stopping guarantees. The authors call analysis of theoretical stopping guarantees in chain-of-thought reasoning a crucial research direction.
• Dependence on logit access. A stated limitation is that LEASH assumes access to token-level logits, which excludes black-box APIs that expose only text — an open practical question the paper does not resolve.

Target Audience

This paper is most useful to machine learning engineers and researchers working on LLM inference efficiency, decoding strategies, and chain-of-thought reasoning. Practitioners deploying reasoning models under latency or token budgets will find the efficiency numbers and the drop-in, training-free design directly relevant. Researchers studying adaptive computation or test-time reasoning will value the comparison against fixed budgets, heuristic triggers, and multi-sample reranking, as well as the explicitly stated limitations around short-answer tasks and logit access.

Authors’ abstract

Chain-of-Thought (CoT) prompting is a key technique for enabling complex reasoning in large language models. However, generating full, fixed-length rationales is computationally wasteful, inflating both token usage and latency. We introduce LEASH: Logit-Entropy Adaptive Stopping Heuristic, a training-free decoding algorithm that adaptively halts rationale generation. LEASH monitors two intrinsic signals: the slope of token-level entropy and the improvement in the top-logit margin. It terminates the generation once both signals plateau, indicating the model has reached a stable reasoning state. Across four instruction-tuned models on the GSM8K and AQuA-RAT benchmarks, LEASH reduces average token generation by 30--35% and latency by 27%, while incurring a 10 p.p. accuracy drop relative to CoT. LEASH is model-agnostic and requires no additional training or supervision, offering a simple and efficient alternative to CoT decoding.