Reasoning While Asking: Transforming Reasoning Large Language Models from Passive Solvers to Proactive Inquirers

AI summary

Overview

Research area: Natural Language Processing — reasoning-oriented large language models, multi-turn interaction, and reinforcement learning from user feedback.

Technical level: Advanced. The paper assumes familiarity with chain-of-thought reasoning, supervised fine-tuning, and group-relative policy optimization used in reinforcement learning for LLMs.

Scope: The paper identifies a failure mode it calls "blind self-thinking" in reasoning LLMs and proposes Proactive Interactive Reasoning (PIR), a two-stage training paradigm (uncertainty-aware SFT followed by US-GRPO) that teaches models to ask the user for missing information instead of reasoning on an ambiguous premise.

What This Paper Is About

Reasoning LLMs such as GPT-o1 and DeepSeek-R1 deliberate through long chains of thought, but they keep reasoning even when the user's request is incomplete or ambiguous, producing overthinking, hallucination, and misaligned answers that the user must then correct. PIR targets this premise- and intent-level uncertainty by training models to detect the moment their confidence drops, ask a clarification question, and fold the user's reply back into the ongoing reasoning chain. Unlike retrieval- or tool-based frameworks, which treat errors as gaps in knowledge, PIR treats them as gaps in understanding the user's intent.

Key Contributions

1. Formalizes the "blind self-thinking" problem in reasoning LLMs and proposes the PIR framework to enable proactive clarification during, not after, reasoning.
2. Introduces an uncertainty-aware data augmentation and SFT procedure that uses Predictive Entropy to locate high-uncertainty reasoning steps and converts linear chain-of-thought traces into interleaved think-ask-respond trajectories for cold-start training.
3. Develops US-GRPO, a Group Relative Policy Optimization method with a dynamic user simulator and a composite reward combining an extrinsic task-success term and intrinsic helpfulness-efficiency terms, so that clarification behavior is optimized rather than templated.
4. Demonstrates gains across interactive tasks and robustness on non-interactive benchmarks, including mathematical reasoning, code generation, document editing, factual knowledge, question answering, missing-premise tests, an ambiguity classification evaluation, and a blinded human user study.

Main Findings

• Interactive task performance: On MATH-Chat, accuracy rises from 22.90 to 32.70 (+9.80) relative to the strongest baseline; on BigCodeBench-Chat, pass rate improves from 19.70 to 22.90 (+3.20); on DocEdit-Chat, BLEU rises from 28.00 to 41.36 (+13.36). The abstract reports these as "up to 32.70% higher accuracy, 22.90% higher pass rate, and 41.36 BLEU improvement," and the results section reports the smaller deltas of 9.8% Accuracy, 3.2% Pass Rate, and 13.36 BLEU.

• Efficiency gains: PIR maintains roughly 1.3–1.7k tokens with the smallest Turns to Resolution (TTR) across benchmarks. The introduction reports an average reduction of approximately 2k tokens per task and roughly half the unnecessary interaction turns.

• Helpfulness of asking: After US-GRPO training, the model's helpfulness scores are 0.44 on MATH-Chat, 0.46 on BigCodeBench-Chat, and 0.66 on DocEdit-Chat, outperforming all baseline systems on this metric.

• RL alignment matters more than structure alone: Active SFT introduces the interactive format but degrades in real interactive settings (MATH-Chat falls from 15.30 to 9.70 for the interactive variant), whereas adding US-GRPO restores and substantially improves accuracy.

• User simulator quality: Upgrading the simulator from Llama-3.1-8B-Instruct to gpt-4o-mini raises final-task accuracy from 32.70 to 34.00, while token usage increases modestly from 1.70k to 1.85k and TTR rises from 1.80 to 2.27.

• Reward ablation: Removing the reasoning-oriented reward drops ACC to 27.5 with TTR rising to 2.04. Using only the helpfulness reward yields higher helpfulness and accuracy but the highest TTR (2.63), a tendency to over-ask. Optimizing solely for efficiency degrades accuracy to 25.1 through premature commitment.

• Non-interactive generalization: On factual knowledge, accuracy improves only from 60.12 to 62.51 on MMLU and from 51.21 to 52.87 on MMLU-Pro, even with a strong user simulator. On question answering the gains are much larger — accuracy rises by 25.74 on TriviaQA and 29.69 on SQuAD with the strong simulator.

• Missing-premise robustness: PIR reaches 17.35 on MIP-GSM8K and 25.00 on MIP-MATH with fewer tokens, while the base model and GPT-4o-mini degrade severely.

• Ambiguity classification (Abg-CoQA): PIR achieves the highest overall accuracy (60.00%), with 49.48% correct clarification requests on ambiguous inputs versus 15.44% for GPT-4o and 16.26% for LLaMA-3.1-8B-Instruct, while answering directly on 70.52% of non-ambiguous inputs.

• Human study: From approximately 500 pairwise annotations by 8 participants on the GSM8K Missing Premise dataset, PIR was preferred 52.45% of the time versus 19.62% for the reasoning-oriented LLM, with accuracy of 59.38% versus 30.93%, and reduced token consumption by about 68% (1.15k versus 3.67k) and turns (1.22 versus 2.63). Against the standard LLM, PIR won 34.52% versus 21.81%, with accuracy of 57.89% versus 47.37%.

Methodology in Plain English

The approach has two stages. In the first stage, the researchers take a frozen teacher model (DeepSeek-R1) and have it produce ordinary reasoning chains. They split each chain into sentence-level steps and compute Predictive Entropy for each step — a measure of how unsure the model was at that point. Steps falling in the top-k% of uncertainty are treated as candidate moments where a human would need to ask something. At those moments, a strong instruction-following model such as GPT-4o-mini inserts a clarification question and simulates the user's reply, turning a single linear chain into a think-ask-respond trajectory. The model is then fine-tuned on these augmented trajectories so it learns the causal pattern: uncertainty leads to clarification, which leads to better reasoning. This produced a Reasoning-while-asking SFT dataset, and an analysis across dataset sizes of 1k to 4k showed the template-correctness of asking-response sequences rising from 0.15 at 1k to 0.64 at 2k and near-perfect at 4k, so 4,000 samples were adopted.

The second stage replaces real users with a simulated one, because training against real people would be expensive and noisy. A user simulator (Llama-3.1-8B-Instruct, or gpt-4o-mini as a stronger variant) is conditioned on a hidden intent and replies to the model's questions. The policy is optimized with GRPO, which avoids training a separate value critic, plus DAPO to speed convergence; gradients apply only to the policy's own reasoning and question tokens, and simulator replies are masked out. The reward has two parts: an output reward that pays only when the final answer matches ground truth, and a reasoning reward that is activated only when the answer is correct and multiplies interaction efficiency E(r) = (N_max − n)/(N_max − 1) by an LLM-judged helpfulness score in [0,1]. This design means that when information is missing, a direct answer earns nothing, so the model learns to ask; when the query is clear, it is rewarded for answering immediately.

Evaluation uses Math-Chat, BigCodeBench-Chat, and DocEdit-Chat from Wu et al. (2025), with a maximum of 5 interaction turns for all methods. Baselines include Qwen2.5-Math-7B-Instruct, a Proactive Prompt variant, STaR-GATE, CollabLLM, DeepSeek-R1-Distill-Qwen-7B, Active SFT, and US-GRPO. Multi-turn baselines use an explicit [TERMINATE CHAT] signal, while PIR decides when to stop on its own.

Why This Matters

Impact on research. The paper reframes a core weakness of reasoning models — overthinking under ambiguity — as an intent-alignment problem rather than a knowledge problem, and shows that clarification can be trained into the reasoning chain instead of bolted on as a prompting template. It also offers evidence that interaction training does not hurt, and in some cases helps, non-interactive performance.

Real-world applications:

• Mathematical tutoring systems that detect underspecified word problems and ask the student for the missing constraint rather than silently assuming one.
• Coding assistants that request the missing API, data schema, or expected behavior before generating a solution.
• Collaborative document editing tools that clarify the user's intended tone, scope, or audience before rewriting.
• Customer-support and question-answering agents that identify information gaps in a user's request and ask a single targeted question instead of guessing.

Industry relevance. The efficiency results matter commercially: PIR roughly halves unnecessary interaction turns and reduces token consumption (in the human study, from 3.67k to 1.15k tokens), which translates directly into lower inference cost and shorter user sessions. The model and code are publicly available at https://github.com/SUAT-AIRI/Proactive-Interactive-R1.

Future Directions

• Richer user simulation. The authors acknowledge that their simulator may not capture the linguistic noise and dynamic intent of real human interaction and likely over-represents majority interaction patterns, leaving minority user behaviors unmodeled.
• Safety alignment. The paper states that no dedicated safety assessment was performed — the PIR model has not been screened for handling of violence, sexual content, self-harm, or hate speech.
• Extending beyond the three evaluated domains. The interactive results cover mathematics, code, and document editing; whether the paradigm transfers to other intent-sensitive domains is not established here.
• Understanding simulator-policy co-evolution. The ablation shows that a stronger simulator raises accuracy to 34.00 but also raises turns to 2.27, an accuracy-versus-interaction-cost trade-off that the paper leaves open for further study.

Target Audience

Researchers and engineers working on reasoning LLMs, reinforcement learning from human or simulated feedback, and multi-turn conversational agents. It is also relevant to practitioners building interactive assistants where ambiguous user input is common, and to readers interested in measuring and reducing inference cost in chain-of-thought models. Parts of the paper, particularly the reward design and GRPO formulation, require prior familiarity with LLM post-training methods.

Authors’ abstract

Reasoning-oriented Large Language Models (LLMs) have achieved remarkable progress with Chain-of-Thought (CoT) prompting, yet they remain fundamentally limited by a \emph{blind self-thinking} paradigm: performing extensive internal reasoning even when critical information is missing or ambiguous. We propose Proactive Interactive Reasoning (PIR), a new reasoning paradigm that transforms LLMs from passive solvers into proactive inquirers that interleave reasoning with clarification. Unlike existing search- or tool-based frameworks that primarily address knowledge uncertainty by querying external environments, PIR targets premise- and intent-level uncertainty through direct interaction with the user. PIR is implemented via two core components: (1) an uncertainty-aware supervised fine-tuning procedure that equips models with interactive reasoning capability, and (2) a user-simulator-based policy optimization framework driven by a composite reward that aligns model behavior with user intent. Extensive experiments on mathematical reasoning, code generation, and document editing demonstrate that PIR consistently outperforms strong baselines, achieving up to 32.70\% higher accuracy, 22.90\% higher pass rate, and 41.36 BLEU improvement, while reducing nearly half of the reasoning computation and unnecessary interaction turns. Further reliability evaluations on factual knowledge, question answering, and missing-premise scenarios confirm the strong generalization and robustness of PIR. Model and code are publicly available at: \href{https://github.com/SUAT-AIRI/Proactive-Interactive-R1}