StreamingThinker: Large Language Models Can Think While Reading

AI summary

Overview

Research area: Large language model reasoning, chain-of-thought (CoT) generation, and streaming/efficient inference.

Technical level: Intermediate. The paper assumes familiarity with transformer attention masks, RoPE position encoding, KV caches, and chain-of-thought reasoning, though the core idea (reasoning while input arrives) is intuitive.

Scope: The paper proposes a "streaming thinking" paradigm in which an LLM begins reasoning as sentences arrive rather than after the full input is read, and instantiates it in a framework called StreamingThinker that is evaluated on Qwen3 models across math, logical, and context-based QA reasoning tasks.

What This Paper Is About

Current LLM reasoning follows a "batch thinking" paradigm: the model waits for the entire input before it starts to think, which introduces latency and dilutes attention to earlier parts of long inputs. The authors take inspiration from how humans read and reason at the same time, and ask whether an LLM can reason incrementally as text streams in, then deepen or adjust its reasoning once the full input is available.

Key Contributions

1. Introduction of the streaming thinking paradigm for LLM reasoning, in which reasoning unfolds in the order of the input stream and its depth is adjusted after reading completes — the authors state this is the first such formulation for LLM reasoning.
2. A streaming CoT generation pipeline that inserts sentence-level boundary tokens (<EOS>, <EOT>) into inputs, prompts the model for order-preserving reasoning per unit, and filters generated traces with quality-control metrics.
3. A training and inference framework for the paradigm: a streaming attention mask plus streaming position encoding (streaming RoPE) for training alignment, and parallel KV caches (source cache and target cache) that decouple input encoding from reasoning generation for concurrent inference.
4. Extensive evaluation on the Qwen3 family over math reasoning, logical reasoning, and context-based QA reasoning, reporting reasoning quality comparable to batch thinking alongside a stated 80% reduction in token-level waiting before reasoning begins and a more than 60% reduction in time-level latency for producing the final answer.

Main Findings

• Reasoning depth trades off against accuracy and cost. In Table 1 (batch-processing setting), accuracy rises consistently with reasoning depth D1 → D2 → D3, where D1 = direct answer, D2 = with global reasoning, and D3 = with self-reflection. For Qwen3-4B on GSM-Symbolic, Batch-S, D3, SPE reaches 0.874 accuracy versus 0.437 at D1, with token usage rising from 199.64 to 493.26. The global reasoning stage (D2) gives the largest marginal improvement.

• Streaming thinking matches batch thinking quality in batch mode. Under batch processing, the streaming thinking model approaches the original batch and distilled-SFT baselines. On ProofWriter with Qwen3-4B, Batch-S, D3, SPE records 0.861 versus 0.620 for Batch, Original and 0.861 for Batch, SFT, SPE.

• Large latency reduction with preserved accuracy in true streaming. Table 2 (Qwen3-4B) shows that on GSM-Symbolic, batch thinking yields 0.855 accuracy, TTFT 94.74, and delay 47.70s, while Streaming, D3 yields 0.856 accuracy, TTFT 20.77, and delay 9.768s. On HotPotQA, Streaming, D3 gives 0.603 accuracy versus 0.575 for Batch, Original, with delay falling from 20.37s to 6.50s.

• Streaming position encoding preserves capacity and sharpens attention. Streaming RoPE (SPE) achieves nearly identical accuracy and token consumption to the original RoPE at the same depth (for example, Qwen3-1.7B on GSM-Symbolic, Batch-S, D2: 0.692 SPE versus 0.688 RoPE). Attention visualizations show the original RoPE has no clear positional preference, while Streaming RoPE shows a pronounced diagonal concentration on the current context.

• Parallel streaming outperforms naive interleaving. In Table 2, the interleaved mode has the same TTFT as streaming but consistently lower accuracy and higher delay. On GSM-Symbolic D3, Interleaved reaches 0.843 accuracy with 15.46s delay, while Streaming reaches 0.856 with 9.768s delay. The authors attribute this to a distributional mismatch between interleaved sequences and the pretraining corpus, plus forced serial synchronization.

• Input ordering matters. In the question-first setting, the model expands logic immediately for identified relevant segments, giving higher D1 token usage but smaller incremental growth at D2/D3. In the context-first setting (Table 3), reasoning is more conservative; on HotPotQA, Streaming, D1 uses 1172.36 tokens versus 165.42 on GSM-Symbolic, and total generation length grows in sparse-information tasks because the model cannot skip irrelevant content.

• Parallel KV overhead is negligible. Efficiency analysis on Qwen3-4B with 100 samples from GSM-Symbolic (Table 4) reports first-token latency dropping from 28.003s to 6.231s (a stated roughly 4.5x speedup), with split_kv and merge_kv taking less than 5ms combined. Peak memory stays consistent with the baseline at approximately 7.99 GB, though bandwidth cost increases due to repeated prefill phases in streaming scenarios.

• Latency bottleneck is input arrival. Because the input rate is set to 150 words per minute to match average human speech, and LLM decoding is much faster, the authors state the effective bottleneck in streaming reasoning is input arrival rather than output generation.

Methodology in Plain English

The researchers first build a dataset that looks like streaming reasoning rather than batch reasoning. They insert boundary tokens into the input to mark sentence-level reasoning units, then prompt an LLM to produce reasoning for each incoming sentence in order and to stop each step with an end-of-thought token. A larger teacher model (Qwen3-235B-A22B-Instruct) reconstructs the generated traces, and Qwen3-32B is used as the initial generation model.

To check quality, they define two automatic scores: a granularity score (the ratio of input boundary tokens to output boundary tokens, where 1 is ideal alignment) and a sequential consistency score (the cosine similarity between the embeddings of a reasoning sentence and its corresponding input sentence, computed with SentenceBERT). Samples that fail are regenerated, and those still failing under the Pass@2 metric are discarded. Passing samples are then expanded into depth-controlled variants (D1/D2/D3) via token-level intervention.

For training, they change two things so the model respects the streaming order. A streaming attention mask blocks each reasoning step from attending to future input sentences, and a streaming position encoding gives input tokens and reasoning tokens independent position IDs both starting at zero, which removes positional contention and keeps each reasoning step closest to its matching input sentence.

For inference, they keep two separate KV caches: a source cache for incoming input tokens and a target cache for generated reasoning tokens. Before decoding, the caches are merged so reasoning can attend to the inputs; after finishing a sentence, they are split again. This allows prefill of new input and decoding of reasoning to happen concurrently, unlike batch or interleaved approaches that rely on a single continuous cache.

Evaluation uses Pass@1 for accuracy, token-to-first-token (TTFT) for token-level latency, and a delay measure for time-level latency computed with input arriving at roughly 150 words per minute. Models tested are Qwen3-1.7B and Qwen3-4B. Datasets are GSM-symbolic and MetamathQA (math), LogicNLI and ProofWriter (logical), and PubMedQA and HotpotQA (context-based QA).

Why This Matters

Impact on research: The paper reframes reasoning efficiency as a scheduling problem — not just fewer tokens or smaller models, but reasoning at the same time as input is consumed. It also identifies positional contention as a concrete obstacle to concurrent input encoding and reasoning generation, and offers streaming RoPE and dual KV caches as a remedy. It adds a streaming dimension alongside existing efficiency directions such as token compression, quantization/pruning, and efficient decoding.

Real-world applications:

• Real-time dialogue and interactive systems, which the paper's Appendix A.1 names as a motivation, where waiting for a full user turn before any reasoning creates noticeable lag.
• Speech and audio interaction, since the paper's related work cites streaming speech recognition and speech translation as domains where input arrives over time.
• Video understanding, cited in the related work as a streaming modality where content unfolds continuously.
• Long-document question answering, where the paper's HotPotQA and PubMedQA experiments show latency reductions on context-based QA reasoning.

Industry relevance: Deployment contexts that bill or measure latency per request — assistants, agents, and interactive pipelines — benefit directly if the reported reductions hold. The efficiency measurements also matter operationally: the authors report that parallel KV cache operations add less than 5ms and that peak memory stays near the baseline at roughly 7.99 GB, suggesting the approach does not require dramatically more memory. The increase in bandwidth cost is a trade-off practitioners would need to weigh.

Future Directions

• Scaling beyond small backbones. The streaming evaluation is limited to Qwen3-1.7B and Qwen3-4B. Whether the paradigm holds for much larger models, and for model families other than Qwen3, is not reported.
• Handling unknown input order. The paper explicitly evaluates question-first and context-first settings and shows that performance characteristics differ. Designing a model that is robust when the order is unknown, without the conservative-reasoning penalty seen on sparse context-based QA tasks, remains open.
• Depth control without instruction signals. Reasoning depth is currently steered by explicit instructions and token intervention. Automatically choosing the appropriate depth per input, or deciding when to skip irrelevant content, is not resolved.
• Reducing bandwidth cost. The authors note that bandwidth cost increases because of multiple prefill phases in streaming scenarios, separate from the parallel KV cache mechanism. The content provided does not report a proposed solution.

Target Audience

Researchers and engineers working on LLM inference efficiency, chain-of-thought reasoning, and streaming or interactive model deployment. It is also useful for practitioners building latency-sensitive applications (dialogue, agents, long-document QA) who want to understand how reasoning can overlap with input arrival, and for readers interested in cognitive-inspired architectures for reasoning.

Authors’ abstract

Large language models (LLMs) have demonstrated remarkable capabilities in chain of thought (CoT) reasoning. However, the current LLM reasoning paradigm initiates thinking only after the entire input is available, which introduces unnecessary latency and weakens attention to earlier information in dynamic scenarios. Inspired by human cognition of thinking while reading, we first design a \textit{\textbf{streaming thinking}} paradigm for LLMs, where reasoning unfolds in the order of input and further adjusts its depth once reading is complete. We instantiate this paradigm with \textit{StreamingThinker}, a framework that enables LLMs to think while reading through the integration of streaming CoT generation, streaming-constraint training, and streaming parallel inference. Specifically, StreamingThinker employs streaming reasoning units with quality control for CoT generation, enforces order-preserving reasoning through streaming attention masks and position encoding, and leverages parallel KV caches that decouple input encoding from reasoning generation, thereby ensuring alignment and enabling true concurrency. We evaluate StreamingThinker on the Qwen3 model family across math reasoning, logical reasoning, and context-based QA reasoning tasks. Experimental results show that the StreamingThinker preserves performance comparable to batch thinking, while yielding an 80\% reduction in token waiting before the onset of reasoning and a more than 60\% reduction in time-level latency for producing the final answer, demonstrating the effectiveness of the streaming paradigm for LLM reasoning. Code is publicly available at https://github.com/EIT-NLP/StreamingLLM/tree/main/StreamingThinker.