InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning

AI summary

Overview

Research area: Large language model reasoning, specifically iterative/long-horizon chain-of-thought and reinforcement learning for LLM post-training.

Technical level: Intermediate. The paper assumes familiarity with chain-of-thought reasoning, supervised fine-tuning, and policy-gradient RL (GRPO), though the core ideas are understandable without deep RL background.

Scope: The paper introduces InftyThink+, an end-to-end reinforcement learning framework that trains a reasoning model to decide when to summarize its own intermediate thinking, what to keep in those summaries, and how to resume reasoning from them — improving accuracy while cutting inference latency and training cost.

What This Paper Is About

State-of-the-art reasoning models solve hard problems by generating extremely long chains of thought, but this "one long trace" approach has three problems: attention cost grows quadratically with length, the model hard-stops at its context window, and performance degrades as critical early information gets lost in the middle. Iterative reasoning — periodically pausing to write a summary, then continuing from a fresh, bounded context — addresses these issues, but prior methods either use fixed chunk boundaries or rely on supervised fine-tuning that only teaches the model to imitate a format, not to make good strategic decisions. This paper's goal is to make those decisions learnable through trajectory-level reinforcement learning.

Key Contributions

1. Reinforcement learning for the iterative reasoning paradigm. InftyThink+ is the first framework to apply end-to-end RL to iterative reasoning, jointly optimizing when to summarize, what to preserve, and how to resume — rather than imitating a fixed format via supervised learning.

2. A complete training recipe for single-trajectory, multi-generation RL. The paper designs trajectory-level rollouts with a maximum iteration budget, an efficiency-aware reward with quadratic decay over iteration count, and a shared-advantage policy gradient scheme (built on GRPO) where every round in a trajectory receives the same advantage signal — so good early summaries are credited for later success.

3. A two-stage training protocol. A supervised cold-start stage transforms existing reasoning data into the InftyThink format (using an external LLM to generate intermediate summaries), teaching syntax and structure; an RL stage then optimizes strategy on top of that format.

4. Empirical evidence across models and task types. Using DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-4B-Base, the method is shown to beat both SFT-only iterative reasoning and standard long-chain-of-thought RL on accuracy, latency, and training speed, and to generalize to out-of-distribution benchmarks including code and scientific reasoning.

Main Findings

• Large accuracy gains over standard long-CoT RL. On AIME24 with the 1.5B model, InftyThink+ improves accuracy by 21% over the cold-start baseline, and by roughly 9 points more than conventional long-CoT RL. On the out-of-distribution GPQA_diamond benchmark it gains 5% over baseline and about 4 points over the vanilla approach.

• Reasoning depth is decoupled from wall-clock time. Because each iteration attends only over a bounded context, InftyThink+ achieves lower latency than vanilla long-CoT even before RL (average 77.6s vs. 111.0s), and after task-only RL it can reason longer (more tokens) with almost no latency increase — the opposite of vanilla, where longer reasoning directly inflates latency.

• The efficiency reward buys a much better trade-off. Adding the efficiency reward reduces average latency from 100.2s to 48.4s and tokens from 20.0K to 10.7K, at a modest accuracy cost (53.96 → 50.58 average), while still beating the cold-start baseline by 6.5 accuracy points.

• Adaptive summarization timing beats fixed or random timing. Ablations show that forcing summaries at a fixed 5K tokens or at random intervals (3000–6000 tokens) consistently hurts accuracy, and the penalty grows after RL — indicating RL sharpens a genuine policy for when to compress rather than memorizing a schedule.

• RL transforms summaries from "format compliance" into a functional policy component. Under SFT only, replacing the model's own summaries with ones generated by an external LLM improves performance (AIME24: 29.48% → 32.40%), showing the model wasn't yet writing useful summaries. After RL, the same substitution hurts (50.94% → 48.42%), meaning the learned summaries are now tightly coupled to downstream reasoning success.

• Continuation capability must also be learned. When an InftyThink summary is handed to a vanilla reasoning model to continue from, that model degrades and its extra gains saturate after roughly the second iteration. InftyThink+ converts summaries from later iterations into monotonic improvements, showing that "how to resume" is not a free capability.

• Training itself gets faster. Per-step RL training time drops from about 300s for vanilla long-context RL to 225s for InftyThink+ (~25% speedup), and to ~175s with the efficiency reward (~40% speedup).

Methodology in Plain English

The approach builds on an existing paradigm (InftyThink) where a model never produces one continuous chain of thought. Instead, it works in rounds: it reads the original question plus a summary written by its previous round, produces some reasoning, then writes a new summary — and it decides on its own when to stop summarizing and emit a final answer. Each round only ever sees a small, fixed-size context, so cost stays bounded no matter how long the overall reasoning gets.

The authors train this behavior in two stages. First, a cold start: they take existing reasoning datasets, chop each long reasoning trace into segments of about 6K tokens, use a general-purpose LLM to write a chain of summaries of at most 1K tokens each (each summary conditioned on the previous one, matching what the model will see at inference), then fine-tune the target model on these transformed examples. This teaches the format but nothing about strategy.

Second, they run reinforcement learning on top. For each training question, the model rolls out a full multi-round trajectory (capped at 5 iterations), and the whole trajectory receives a single scalar reward. That reward is the product of two parts: a task reward (1 if the final answer verifies as correct, 0 otherwise) and an efficiency reward that starts at 1 for a single-iteration solution and decays quadratically toward 0 as iterations approach the cap. Multiplying rather than adding means incorrect answers get zero reward regardless of how fast they were, so the model can't learn to bail out early and be wrong. The policy gradient (GRPO-style, with token-level loss averaging) shares the trajectory's advantage across all its rounds — a good first summary gets credit for enabling correct later reasoning. They also apply token-level gradient masking between the inference and training engines to stabilize training.

They then evaluate with 32 samples per problem at temperature 0.7, using an off-the-shelf verifier model, on MATH500, AIME24, AIME25, and GPQA_diamond.

Why This Matters

Impact on research: The paper reframes iterative reasoning from a format problem to a decision-making problem, and provides concrete evidence for that reframing via the summary-swap and continuation-transfer experiments. It also shows that RL and iterative structure are complementary — the structured intermediate summaries act as a substrate that RL can exploit more effectively than raw long-CoT. In an era where RL is the dominant training paradigm for reasoning models, demonstrating 25–40% training speedups matters for how labs allocate compute.

Real-world applications:

• Cost reduction for deployed reasoning services. A 30–70% latency reduction on math-style reasoning directly lowers per-query serving cost for any product that runs chain-of-thought inference at scale.

• Long-horizon agentic tasks. Web navigation, software engineering, and multi-tool workflows routinely overflow context windows; bounded-context iterative reasoning with learned summaries is a natural fit for keeping agents running indefinitely.

• Scientific and mathematical assistants. The method generalizes to scientific and code reasoning benchmarks, suggesting use in research copilots and automated proof/derivation tools.

• On-device and edge reasoning. Smaller models (1.5B) with bounded per-step context are more feasible to run locally than models requiring an enormous KV cache.

Industry relevance: The work comes from a Zhejiang University / Ant Group collaboration and uses production-grade infrastructure (verl, SGLang, Megatron-Core, FSDP). The results are directly relevant to any team training or serving reasoning models, and the efficiency-reward design offers a general template for controllable trade-offs between accuracy and cost at deployment time.

Future Directions

• Scaling to much larger and more capable base models. The largest model tested is Qwen3-4B; it remains open whether the gains hold or compound at 30B+ scale and on models already trained with heavy RL.

• Beyond math and science. Evaluation is concentrated on mathematical and scientific reasoning; whether learned summarization policies transfer to open-domain, multilingual, or subjective tasks is untested.

• Better summary mechanisms. The current summaries are plain text with rigid length constraints. Learned compression, structured state, or latent summaries might preserve information more faithfully than a fixed 1K-token budget.

• Reward and hyperparameter sensitivity. The multiplicative task × efficiency reward and the quadratic decay shape are design choices; how sensitive the results are to these formulations, to the iteration cap, and to segment length during cold-start data construction deserves more systematic study.

• Interaction with memory and retrieval. The paper notes its conceptual link to human reasoning; combining self-summarization with external memory or retrieval could extend the effective horizon further.

Target Audience

Researchers and engineers working on LLM reasoning, reinforcement learning for language models, and inference efficiency. It is most valuable to those already familiar with chain-of-thought reasoning and policy-gradient methods (e.g., PPO/GRPO) who want to understand how iterative reasoning can be optimized end-to-end rather than hand-designed. Practitioners deploying reasoning models under latency or cost constraints will find the efficiency-reward design and latency results directly actionable, while those purely interested in applications can skim the methods section for the two-stage training idea and the headline numbers.

Authors’ abstract

Large reasoning models achieve strong performance by scaling inference-time chain-of-thought, but this paradigm suffers from quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects. Iterative reasoning mitigates these issues by periodically summarizing intermediate thoughts, yet existing methods rely on supervised learning or fixed heuristics and fail to optimize when to summarize, what to preserve, and how to resume reasoning. We propose InftyThink+, an end-to-end reinforcement learning framework that optimizes the entire iterative reasoning trajectory, building on model-controlled iteration boundaries and explicit summarization. InftyThink+ adopts a two-stage training scheme with supervised cold-start followed by trajectory-level reinforcement learning, enabling the model to learn strategic summarization and continuation decisions. Experiments on DeepSeek-R1-Distill-Qwen-1.5B show that InftyThink+ improves accuracy by 21% on AIME24 and outperforms conventional long chain-of-thought reinforcement learning by a clear margin, while also generalizing better to out-of-distribution benchmarks. Moreover, InftyThink+ significantly reduces inference latency and accelerates reinforcement learning training, demonstrating improved reasoning efficiency alongside stronger performance.