Are Large Reasoning Models Interruptible?

AI summary

Overview

Research area: Evaluation of Large Reasoning Models (LRMs) — specifically their robustness to mid-inference interruptions and dynamic context changes, as opposed to the standard "frozen world" static, turn-based evaluation paradigm.

Technical level: Intermediate. The concepts (chain-of-thought reasoning traces, thinking tokens, benchmarking) assume some familiarity with how modern reasoning models generate outputs, but the paper's framing is largely conceptual and its findings are presented through accessible failure-mode categories.

Scope: The paper defines, benchmarks, and taxonomizes how five open-weight LRM families (Qwen3-8B, GPT-OSS-20B, Magistral-Small-1.2, plus GPT-OSS-120B, DeepSeek-R1, and Nemotron-3-Nano in the appendix) degrade when their reasoning is cut short or their problem specification changes mid-inference, across math (GSM-8K, MATH-500, AIME-24/25) and programming (LiveCodeBench-v6) tasks.

What This Paper Is About

Large Reasoning Models are almost always evaluated as if the world stays frozen while they think: the model gets a fixed problem, generates a long internal reasoning trace, and finally emits an answer. In reality, users interrupt to demand a faster answer, or new information arrives mid-inference that changes the problem. This paper asks whether LRMs can handle those dynamic conditions, and finds that high static accuracy does not transfer: models can fail unpredictably, with accuracy dropping by up to 60% when updates arrive late in the reasoning process.

Key Contributions

1. Problem setting — "interruptible reasoning." The authors formalize a practically motivated evaluation setting for LRMs covering two classes of dynamic intervention: time-constrained interruptions (hard interrupts that force an answer, and soft interrupts that request speedup) and update-driven mid-inference changes to the task specification.

2. Benchmark and evaluation suite. They release a public benchmark with human-verified updates and standardized protocols, built on GSM-8K, MATH-500, AIME-24/25, and LiveCodeBench-v6, with extensive results and ablations across models and interruption positions. Updates were generated with GPT-5 and manually verified by the authors so that the update is required to solve the problem correctly.

3. Error taxonomy. They identify and analyze three recurring failure modes — reasoning leakage, panic, and self-doubt — and quantify how much of the degradation each one accounts for.

4. A training-free baseline. They show that "prompt guidance" — a short postfix written in the model's own voice stating that the update is correct and user-verified — substantially mitigates self-doubt on easier tasks, fully resolving it on GSM8K and MATH-500.

Main Findings

• Static performance overestimates robustness. Even state-of-the-art LRMs that achieve high accuracy in static settings fail unpredictably under interruption or changing context, with performance dropping by up to 60% when updates are introduced late in the reasoning process.

• Reasoning leakage. When hard interrupted too early, models continue reasoning inside their final answer region instead of halting. This can produce answers up to 10x longer than those generated with full thinking. Small models (Qwen3-1.7B) exhibit longer leakage traces even under extreme hard interrupts, sometimes providing the answer and then reasoning after it without emitting an <EOS> token.

• Panic under speedup. Over 90% of new errors under speedup arise from panic, defined concretely as the model closing its thinking after using less than 1% of its left context limit after the soft interrupt. This pathology results in up to 30% accuracy drops, with up to 80% of performance loss attributable to panic behavior.

• Self-doubt under updates. Roughly 80% of update-driven interrupt errors stem from self-doubt, where models question whether the update is correct and continue their original reasoning without incorporating the new information. The pathology is more pronounced on math tasks, and GPT-OSS exhibits minimal doubt behavior compared to the other models.

• General anytime behavior under hard interrupts. Accuracy is generally worse when interrupting early and better when interrupting late. Exceptions include Magistral on AIME (0.3) and coding tasks, where interrupting late led to slightly improved performance relative to not interrupting.

• Timing matters for efficiency. Under soft interrupts, Qwen and GPT-OSS generally comply with speedup instructions and produce shorter reasoning and answer lengths, while Magistral shows little change. When the speedup signal is issued near the end of reasoning (0.9), models sometimes generate more tokens than in the uninterrupted setting because they spend tokens reflecting on and incorporating the update.

• Update cost stays below restart cost. With prompt guidance, overall reasoning effort increases but in most cases remains below the number of tokens required to restart from scratch. For GPT-OSS on coding, accuracy remains static across interruption positions and reasoning cost never exceeds 110% of the original no-update reasoning cost, even for late updates.

• Prompt guidance works on easier tasks only. Prompt guidance fully resolves the self-doubt issue on GSM8K and MATH-500 and reduces the gap on AIME and LiveCodeBench, but AIME tasks still show considerable room for improvement. Models, specifically Qwen3, remain particularly fragile when handling mid-reasoning updates on coding tasks.

• Model scaling offers limited benefit. Under hard interrupts, Qwen3-1.7B, 8B, and 32B perform similarly, with scale only clearly benefiting the harder AIME problems. In the update-driven setting, Qwen3-1.7B struggles to generalize and performs substantially below baseline even on GSM-8K.

• Assistant-turn beats user-turn intervention. Inserting the interruption within the model's ongoing reasoning trace outperforms inserting a new user turn mid-reasoning, with the difference most pronounced for Magistral on GSM8K and MATH-500.

• Prompt guidance is not surface-form sensitive. A paraphrased version of the guidance postfix produced no statistically significant performance difference from the original.

• Compact reasoning does not fix the problems. Chain of Draft with Qwen3-8B often reduces output length, but all three pathologies persist: at hard interruption position 0.3, answer lengths remain inflated by 1.38x on AIME and 6.27x on LiveCodeBench-v6 relative to no interruption; under soft interruption on AIME at 0.3, its panic rate is higher (13.1% versus 3.8% with the 1% threshold); and it still exhibits self-doubt under update-driven interruption.

• Benchmark memorization is unlikely. Running models on the revised problems without any interruptions and evaluating against the original problems' ground truths yielded nearly zero performance for all state-of-the-art open-weight models, indicating they solve the revised problems rather than recalling the originals.

Methodology in Plain English

The authors take standard math and coding benchmarks that require long reasoning traces and turn them into tests of dynamic behavior.

Simulating interruptions. For each problem, they first let the model generate its complete reasoning trace. They then re-run the model but cut the reasoning at relative positions — 0.1, 0.3, 0.5, 0.7, and 0.9 of the full trace length — and insert interruption tokens. Relative positions are used because reasoning length varies substantially across models and samples.

Two interruption types. Time-constrained interruptions either forcibly end thinking (hard interrupt: injecting an <end-thinking> token, or a <force-answer> token that also prescribes the output format) or merely ask the model to speed up (soft interrupt: a directive like "Please answer faster," with reasoning still permitted). Update-driven interruptions inject new information that changes the correct answer.

Building the update dataset. For math, initial conditions are modified (for example, changing variable values) so that the revised problem plus the update is semantically equivalent to the original. For programming, only the textual problem description is given first, then updates alter starter code, adjust variable ranges, or add constraints and sample test cases. All augmentations were generated with GPT-5 and manually verified by the authors.

Measurement. Two quantities are tracked: interruption-conditioned accuracy (whether the post-interruption answer matches ground truth) and the number of tokens generated after interruption (a proxy for computation cost). Following the DeepSeek-R1 evaluation protocol, they run 16 independent trials for AIME-24/25 due to its small size and high variance, and a single run for the other datasets, reporting mean accuracy with bootstrapped 95% confidence intervals.

Models and infrastructure. They evaluate open-weight model families because closed-source APIs do not support mid-reasoning interruption or context updates. As a best-effort proxy, they run GPT-5.4-Mini with explicit answer-side thinking fields. Experiments use the vLLM inference engine on NVIDIA Ampere or newer GPUs.

Error attribution. Doubt behavior is classified across failure cases using an LLM-based classifier (GPT-5) given the update and the truncated reasoning trace as inputs. Panic is detected with the 1%-of-remaining-context heuristic, with sensitivity to the threshold and agreement with human labels reported in the appendix.

Robustness checks. Sentence-level and absolute-token interruption schemes were tested, and the same trends held. The authors also used an assistant-turn vs. user-turn ablation, a paraphrase ablation for the guidance prompt, and a scaling study across Qwen3-1.7B/8B/32B.

Why This Matters

Impact on research. The paper argues that interruptibility is a distinct capability, not a byproduct of reasoning quality, and that the standard "thinking tokens versus accuracy" plots used in prior work do not faithfully reflect real inference cost — because longer answers can hide additional reasoning, quietly inflating compute even when a model appears to stop thinking early. It supplies a benchmark, a formal setup, and an error taxonomy that future work can build on, and it extends the line of work on missing-premise overthinking (where models given unsolvable questions generated 2-4x more tokens rather than abstaining) into the setting where updates actually supply or correct the missing information.

Real-world applications:

• Interactive assistants and chat interfaces where users interrupt a long-running computation to get a quick partial answer, or interject after spotting a flaw in the model's reasoning. Current interruption mechanisms in some software are ad-hoc (injecting a new user turn, or discarding traces and restarting), and the paper shows these can trigger leakage, panic, and self-doubt.

• Collaborative coding environments where multiple models and users modify the same codebase. LiveCodeBench problems take on average more than 6 minutes for GPT-OSS to complete on 80G A100 GPUs, making mid-task specification changes a realistic and costly scenario — and the paper shows coding updates are where Qwen3 remains particularly fragile.

• Time-sensitive deployments such as incident response, live tutoring, or any setting where a user needs a best-effort answer now rather than a perfect answer later, and where a model that silently keeps reasoning inside its "final" answer defeats the purpose of the interrupt.

• Agentic and multi-agent workflows, where a planner or another model revises task specifications mid-execution. The paper notes its findings should generalize to multi-round interruptions even though it studies single-round interactions.

Industry relevance. The failures described are reliability failures, not just accuracy failures: the impact statement notes that LRMs can behave unpredictably under interruption, sometimes producing incorrect or misleading outputs with high confidence, which could contribute to user mistrust and provide potential vectors for LLM attacks in safety-critical applications. The paper offers one cheap mitigation (prompt guidance) that resolves the issue on easier benchmarks but leaves harder ones largely unsolved, meaning production systems cannot currently rely on prompt tweaks alone.

Future Directions

1. Extending beyond math and coding. The authors focus on these two domains because they induce long, reproducible reasoning traces, and view long-horizon QA settings such as deep research or multi-hop QA — and tool-using, multi-turn agentic workloads — as promising extensions where mid-reasoning intervention points are meaningful.

2. Richer interruption dynamics. The current setup models interruptions as single, well-defined events inserted at pre-specified points. Real interruptions may be noisy, adversarial, or repeated multiple times within one episode; the framework naturally extends to multiple and other update types (for example, distracting ones).

3. Interruption-aware training. The paper deliberately stops at problem identification and analysis, proposing no new training or control methods. A natural next step is to post-train with simulated time-constrained and update-driven interruptions, combining correctness rewards with penalties that specifically target reasoning leakage, panic, and self-doubt.

4. Faithful evaluation of proprietary models. Applying the protocol requires access to intermediate reasoning traces and the ability to intervene during generation, which many proprietary APIs do not expose. The GPT-5.4-Mini proxy is not a faithful substitute for intervening on native model reasoning, leaving open how closed models behave. The scaling limit observed in Qwen3-1.7B under update-driven interruption is also flagged as an unexplained phenomenon for future investigation.

Target Audience

Researchers working on reasoning-model evaluation, inference-time control, and efficient reasoning; practitioners building interactive or agentic systems that need to interrupt, redirect, or update a model mid-generation; and safety and reliability engineers concerned with unpredictable model behavior in time-sensitive deployments. Readers interested in LLM failure-mode taxonomies and benchmark design will also find the error taxonomy and evaluation protocols directly useful.

Authors’ abstract

Real-world applications of Large Reasoning Models (LRMs) often require reasoning about changing prompts or environments. In this work, we challenge the frozen world assumption and evaluate LRM robustness under two realistic dynamic scenarios: interruptions, which test the accuracy of model responses under budget-constrained outputs, and dynamic context, which tests model adaptation to in-flight changes. Across mathematics and programming benchmarks that require long-form reasoning, static evaluations consistently overestimate robustness: even state-of-the-art LRMs, which achieve high accuracy in static settings, can fail unpredictably when interrupted or exposed to changing context, with performance dropping by up to 60% when updates are introduced late in the reasoning process. Our analysis further reveals several novel failure modes, including reasoning leakage, where models fold the reasoning into their final answer when interrupted; panic, where under time pressure models abandon reasoning entirely and return incorrect answers; and self-doubt, where performance degrades when trying to incorporate updated information. Project Page: http://dynamic-lm.github.io/