Thinking Inertia: LLMs Keep Thinking When Told Not To

AI summary

Overview

• Research area: Natural Language Processing / large language model reasoning control and evaluation.
• Technical level: Intermediate. The paper defines formal response-level metrics, but its core question—whether "stop thinking" instructions actually stop models from visibly reasoning—is accessible without deep technical background.
• Scope: The paper proposes response-centric metrics for measuring "no-thinking" in LLMs and uses them to show that disabling thinking modes or telling models to answer directly does not eliminate visible explicit inference.

What This Paper Is About

LLMs increasingly ship with explicit "thinking" switches and reasoning-effort settings, but nobody has agreed on how to define or measure "no-thinking." Existing approaches rely on surface proxies—a disabled thinking mode, the absence of a <think> block, or a short output—and the authors show each of these can fail: a model with thinking disabled can still emit question-specific inferential text, and a short response can still contain a compact reasoning step. The paper's goal is to define no-thinking as an observable response behavior and then test whether it can actually be controlled across models, tasks, and answer formats.

Key Contributions

1. A measurable definition of no-thinking. The authors decompose any model response into pre-answer text T and a final answer A, then evaluate T at three progressively stricter levels: the Empty-Thinking Rate (ETR) for strict answer-only compliance, Question–Pre-answer Relevance (QRel.) for whether T is semantically related to the question, and the Explicit Inference Rate (EIR) for whether T contains a visible inferential step.

2. A blinded LLM-judge rubric. Table 1 defines five mutually exclusive labels for non-empty pre-answer text—Generic/off-topic, Paraphrase-only, Relevant non-inferential, Explicit reasoning, and Unclear—with the judge receiving only (Q, T) while model identity, intervention mode, final answer, correctness, and QRel. are hidden.

3. A six-mode intervention spectrum. Table 2 organizes interventions along a single axis of no-think intensity: Think-On (M1), Think-Off (M2), CoT (M3), Soft (M4), Strict (M5), and Prefix (M6), evaluated on six LLMs across six benchmarks.

4. Systematic empirical characterization of "thinking inertia." The paper documents that visible explicit inference persists under no-think controls, scales with the openness of the answer space, and can trade off against task accuracy.

Main Findings

• No-think controls are not reliable no-thinking controls. Even DeepSeek-V4-Flash (284B) retains visible explicit inference in 99.9% of open-ended responses under its disabled-thinking setting.

• Thinking inertia follows the answer space, not the switch. Under both native and strict no-thinking settings, visible explicit inference is lowest on Boolean tasks, intermediate on multiple-choice tasks, and highest on open-ended tasks. Under M2, models reach roughly 80–100% ETR on Bool, MCQ ETR ranges from near zero to 95% across models, and open-ended ETR collapses to nearly zero everywhere.

• Step-by-step prompting recovers reasoning even with the thinking switch off. Under M3, QRel. and EIR often move back toward their M1 level, but the Bool < MCQ < Open-ended staircase in six-model mean EIR persists.

• Open-ended tasks expose a no-thinking–performance trade-off. Across the five LLMs with fully disabled thinking, moving from M2 to M5 raises mean ETR from near zero to roughly 40% while reducing mean accuracy by about 15 percentage points on open-ended tasks. Corpus-wide EIR decreases from 99.69% to 62.00% (Appendix Table D.3).

• Bool and MCQ show little corresponding accuracy cost. From M2 to M5, EIR falls from 5.15% to 0.12% on Bool and from 45.57% to 25.39% on MCQ, while mean accuracy remains largely flat—Bool accuracy stays nearly flat at around 85%.

• Prefix forcing is not a stronger constraint. Contrary to the expectation that prefilling "The answer is \boxed{" would enforce answer-only output, mean ETR drops from M5 to M6 on MCQ and open-ended tasks, while mean QRel. and EIR rebound in every answer-space family.

• Higher accuracy does not imply easier no-thinking. Under a fixed four-choice MMLU interface, STEM combines the highest accuracy with the lowest ETR (0.3–36.4%) and the highest QRel. (0.509–0.784) and EIR (59.7–85.5%) in every model block.

• Changing only the answer interface reshapes behavior. Rewriting matched numeric-answer MMLU/MMLU-Pro items for Qwen3-32B under M2 into Boolean verification leaves accuracy nearly unchanged (77.8% to 78.9%) but raises ETR from 10.5% to 62.5%, lowers QRel. from 0.709 to 0.314, and lowers EIR from 86.95% to 36.90%. The open-ended rewrite reduces accuracy to 52.6%, drives ETR to 0%, and raises QRel. to 0.810 and EIR to 90.14%.

• QRel. is not just a proxy for length. Pre-answer length under M1 grows sharply from Bool to open-ended tasks, yet QRel. changes much less across those answer spaces. A full-corpus audit flags only 301 of 572,688 outputs (0.053%) as near-verbatim repetitions, and zeroing their QRel. values changes the global mean by at most 0.00053; a Level 3 audit labels only 355 outputs (0.06%) as paraphrase-only.

• Human validation supports all three metrics. The strict-compliance decision underlying ETR agrees with manual inspection on 565 of 600 outputs (94.2%). QRel. reaches a mean item-level Spearman correlation of ρ = 0.829 with five annotators' relevance judgments, compared with 0.809 for all-MiniLM-L6-v2. Human explicit-inference labels achieve 98.8% mean binary agreement with the primary GPT-5.5 judge and 96.7% with the independent Claude Opus 4.8 judge. Human annotations reproduce the same answer-space and intervention trends.

Methodology in Plain English

The authors start from a simple observation: there is no agreed definition of what a "no-thinking" response looks like, and the shortcuts people use (checking a setting, looking for a <think> tag, or measuring output length) don't describe the actual content of the response. So they reframe the problem around the observable output.

For each response, they split the text into everything visible before the final answer (T) and the final answer itself (A), then score T three ways. ETR simply asks whether T is empty while the answer is still parseable in the required format. QRel. embeds the question and the pre-answer text with an instruction-aware encoder (Qwen3-Embedding-4B) and takes the cosine similarity, so it captures whether the leftover text is related to answering the question rather than just filler. EIR sends (Q, T) to a blinded GPT-5.5 judge at temperature 0, which picks one of five labels from a fixed rubric; only the "Explicit reasoning" label counts as a hit.

They then run a controlled intervention spectrum on six models—Qwen3-4B, Qwen3-32B, Qwen3-235B-A22B, DeepSeek-V4-Flash (284B), GPT-5.4, and Gemini-3-Flash—across six benchmarks spanning three answer spaces: BoolQ and StrategyQA (Boolean), MMLU and MMLU-Pro (multiple-choice), and GSM8K and MATH (open-ended). Interventions range from enabling thinking, through disabling it natively, through increasingly strict natural-language constraints, to structurally prefilling the answer template.

To make sure the automated metrics mean something, they run a blinded human study: 600 sampled outputs (100 per benchmark), preserving each benchmark's empty/non-empty proportion, with five annotators scoring all 402 non-empty pre-answer texts for relevance and reasoning while seeing only the question and the pre-answer text. A second judge from a different model family (Claude Opus 4.8) replicates the Level 3 results. A separate experiment holds the underlying math questions fixed and rewrites only the answer interface, isolating answer space as a causal factor.

Why This Matters

Impact on research. The paper argues that "no-thinking" is a non-trivial capability that deserves systematic evaluation alongside reasoning ability itself, rather than being assumed from a model setting or an instruction. By showing that common surface proxies conflate instruction control with the suppression of visible reasoning, it gives the field a testable behavioral framework that other researchers can adopt, critique, or extend to different models and tasks. The finding that the answer space shapes behavior more than the thinking switch reframes how reasoning-control results should be interpreted.

Real-world applications:

• Inference cost and latency planning. Teams that rely on no-thinking modes to cut token consumption cannot assume those modes will actually shorten visible reasoning, particularly on open-ended generation.
• Agent and pipeline design. Systems that depend on structured, machine-parseable answers from LLMs need to account for question-specific text leaking into the answer channel even when a direct-answer format is requested.
• Regression testing and model selection. The ETR/QRel./EIR metrics provide a concrete way to compare model versions or providers on controllability, not just accuracy.
• Safety and auditability. When an application needs a model to answer without exposing intermediate reasoning—or, conversely, needs to guarantee that reasoning is visible—surface settings are insufficient evidence.

Industry relevance. Every major provider now ships thinking toggles and reasoning-effort settings, and the paper shows these are not equivalent to response-level compliance. The strict answer-only prompting that does improve compliance can cost roughly 15 percentage points of mean accuracy on open-ended tasks, which is a direct product decision: tighter output control versus task performance.

Future Directions

• Defining no-thinking beyond the visible response. The paper explicitly limits its claims to observable response behavior and does not make claims about latent reasoning processes inside the model; extending the framework to internal computation is left open.
• Extending the framework beyond the studied task families. The evaluation covers Boolean, multiple-choice, and open-ended answering across six benchmarks; whether the same thinking-inertia pattern holds for other task structures, answer interfaces, or modalities is not established here.
• Explaining the mechanism behind answer-space dependence. The authors note the observable progression is "consistent with greater computational demands" in open answer spaces, but this is offered as a consistency argument rather than a demonstrated mechanism.
• Designing interventions that suppress visible inference without accuracy cost. Prefix forcing (M6) failed to behave as a stronger constraint, and Strict (M5) produced the clearest accuracy cost on open-ended tasks, so an effective low-cost control remains an open engineering problem.

Target Audience

This paper is most useful to LLM evaluation and benchmarking researchers, practitioners who configure thinking modes and reasoning budgets in production systems, and anyone designing prompts or output formats that assume a model can be told to answer directly. It is also relevant to researchers working on reasoning controllability, chain-of-thought compression, and inference-cost reduction, as well as to readers interested in how answer formatting interacts with model behavior. The formal metric definitions and the human-validation protocol make it accessible to readers with a general machine-learning background, while the experimental detail is technical enough for specialists.

Authors’ abstract

Large Language Models (LLMs) increasingly ship with explicit "thinking modes", yet their counterpart, "no-thinking", has received far less attention. We study LLMs' no-thinking behavior along two axes. a. How to measure no-thinking? Prior work typically defines no-thinking through proxies such as a disabled thinking mode or the absence of long traces. These proxies are unreliable: disabled thinking modes may still emit reasoning, while long traces may contain filler rather than genuine inference. We instead normalize each response into a pre-answer trace and final answer, and evaluate it at three levels: (i) Empty-Thinking Rate for strict answer-only compliance; (ii) instruction-aware Question-Pre-answer Relevance for similarity between the question and pre-answer trace; and (iii) LLM-as-judge Explicit Inference Rate for visible explicit inference. Together, these metrics distinguish answer-only output, relevant but non-inferential text, and explicit inference. b. How does no-thinking vary across tasks and models? We evaluate six prompting interventions on six LLMs across Boolean, multiple-choice, and open-ended questions. We find that explicit no-think controls cannot reliably eliminate visible inference. Models instead exhibit "Thinking Inertia": explicit inference persists even under strict controls and becomes more prevalent as the answer space opens. Accuracy remains stable on Boolean and multiple-choice tasks, whereas open-ended tasks reveal a trade-off between answer-only compliance and task accuracy. Rewriting the same questions across answer spaces shows that supplying candidate answers makes answer-only responses easier to produce. These findings establish no-thinking as a non-trivial capability: stopping explicit reasoning cannot be assumed from model settings or instructions alone and deserves systematic evaluation alongside reasoning ability.