LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards

AI summary

Overview

Research area: Natural Language Processing — reinforcement learning for long-context reasoning in large language models.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for LLMs (policy gradients, KL-regularized objectives, DAPO/GRPO-style group-based training), information theory (pointwise mutual information, negative log-likelihood), and long-context benchmarks.

Scope (one sentence): The paper introduces LongR, a training framework that teaches LLMs to interleave reasoning with explicit quotation of document spans and rewards those quotations with a dense, annotation-free "relative information gain" score, evaluated on LongBench v2, RULER and InfiniteBench with Qwen3 4B and 8B models.

What This Paper Is About

Long-context tasks — long-dialogue understanding, structured data analysis, multi-document synthesis — require not just consuming many tokens but performing deduction over them. Standard reinforcement learning for reasoning relies on sparse, outcome-only rewards (a single signal for whether the final answer is right), which the authors argue is too coarse to teach a model where in a long document to look. The goal is a training recipe that makes a model actively consult the document while it reasons, guided by a dense reward that needs no human evidence annotations and no pre-chunked documents.

Key Contributions

1. Dynamic "Think-and-Read" mechanism. Instead of forcing the model to emit discrete chunk identifiers (as the prior method LongRLVR does), LongR embeds evidence-seeking directly into the chain of thought: the model quotes exact text snippets verbatim inside its reasoning, so no document segmentation is needed and the original information is preserved.

2. Contextual dense reward via relative information gain. For each quoted segment, a frozen verifier model (Qwen3-4B by default) measures how much the segment's token-level negative log-likelihood drops when conditioned on the query and document, normalized by its unconditional negative log-likelihood. This produces a bounded, scale-invariant utility score computed without evidence annotations.

3. Self-learned reasoning pattern via a progressive curriculum. The "consult-then-reason" behavior is acquired purely through RL with a two-stage length curriculum (16K then 32K tokens), starting from generic open-source instruction data rather than long-context-specific demonstrations.

4. Broad empirical validation. Gains on LongBench v2 (over 9%), RULER and InfiniteBench, reproduced across DAPO, CISPO and GSPO, plus ablations on reward design, verifier size, curriculum stages, reasoning-chain length and distractor robustness.

Main Findings

• LongBench v2, 8B: LongR reaches an overall score of 36.23, versus 32.84 for outcome-only DAPO (+3.39) and 26.64 for the SFT baseline (+9.59). The Hard split rises from 25.64 to 33.20 and the Long (>128k) split from 22.97 to 39.12.
• LongBench v2, 4B: LongR reaches 35.44, versus 30.41 for DAPO (+5.03) and 25.65 for SFT (+9.79). Hard split: 28.16 → 35.61.
• RULER, 8B: average of 61.48 at 128K and 71.98 at 64K, beating outcome-only DAPO by +6.61 and +16.96 respectively; the largest gains appear in Variable Tracking (80.20 at 128K) and QA (45.60 at 128K).
• RULER, 4B: 60.71 at 128K and 69.43 at 64K, with the main improvement in QA (35.00 at 128K).
• InfiniteBench: best average scores of 50.20 (8B) and 47.61 (4B); En.MC improves from 54.15 to 58.52 (8B) and En.QA from 12.68 to 21.85 (4B).
• Relative beats absolute and thresholded density. On LongBench v2 (8B), relative density scores 36.23 vs 32.80 for outcome-only, 32.41 for absolute density and 34.10 for threshold density. Absolute density is described as reward-hackable (the policy generates verbose or esoteric segments to inflate prior uncertainty), and threshold density treats all segments above a 0.5 cutoff identically. On 4B the same ordering holds (35.44 vs 30.42 / 31.61 / 33.95).
• Verifier scale saturates. Moving from a 1.7B to a 4B verifier improves 8B from 34.59 to 36.23 and 4B from 35.09 to 35.44; scaling to 32B adds only marginal gains (36.83 for 8B, 36.08 for 4B). A 4B verifier is chosen as the efficiency/performance trade-off.
• Works across RL objectives. With the same recipe, CISPO and GSPO deliver +8.35 and +8.75 overall over SFT for 8B, and +8.30 and +8.25 for 4B. Gains concentrate differently: CISPO is strongest on the 8B Medium split (36.57) and GSPO on the Hard split (36.29 for 8B, 35.93 for 4B). DAPO is the default because it is the strongest and most balanced overall.
• Curriculum matters. For 8B, the overall score goes 26.64 (SFT) → 32.41 after the 16K stage → 36.23 after the 32K stage; for 4B, 25.65 → 31.56 → 35.44.
• No length inflation or distractor blow-up. On the NIAH task from 8K to 128K, LongR's reasoning chains are comparable to, and at some lengths (32K, 128K) marginally shorter than, the outcome-only baseline, and its distractor usage profile is nearly identical to that baseline.
• Reward is position-insensitive in the tested setup. Mean contextual reward is 0.86 for evidence placed at the front (5%), 0.86 at the middle (50%) and 0.85 at the tail (95%), showing no recency advantage. (The appended content is truncated before the query-sensitivity analysis in Appendix E.2 is reported.)

Methodology in Plain English

The pipeline has two phases.

Cold start (supervised fine-tuning). The base Qwen3 model is first trained to imitate expert reasoning traces, using 20,000 open-source instruction-following samples with a cosine schedule, peak learning rate 2e-5, warmup ratio 0.03, maximum sequence length 16,384, sequence packing, and 900 steps. This gives the model the required output structure (reasoning wrapped in <think> tags, answers in <answer> tags) so that later group-based RL is stable.

RL with dense utility rewards. The model is prompted to quote verbatim snippets from the document inside square brackets during its reasoning. Training data consists of 1,000 synthesized samples per length configuration (16K and 32K), plus the DocQA-RL-1.6K dataset in the second stage. The total reward is the standard sparse reward (1.0 for valid format, 2.0 for a correct answer) plus a contextual reward:

• Every bracketed quotation is extracted deterministically from the generated reasoning.
• A frozen Qwen3-4B verifier scores each quotation by its token-averaged negative log-likelihood both with and without the query-and-document in context. The difference is the information gain; dividing by the unconditional negative log-likelihood turns it into relative information gain, a scale-invariant number at most 1 with no finite lower bound.
• The trajectory-level contextual reward is the average over all unique quotations, and is 0 if the model quotes nothing. Irrelevant spans get near-zero reward because context adds no predictive power; genuinely context-dependent evidence (unique entities, dates) gets high reward because the document sharply reduces uncertainty.

Curriculum. Training is split into two 100-step stages with a constant learning rate of 1e-6. Documents are capped at 16K tokens in stage 0 and 32K in stage 1, so the model learns core reasoning in shorter contexts before generalizing to longer ones. Other settings: batch size 8, 16 rollouts, maximum sequence length 32,768, maximum output length 4,096 tokens, Ulysses sequence parallelism of 2, run on 16 NVIDIA H20 141GB GPUs. Evaluation uses vLLM with temperature 1.0 and up to 16,384 generated tokens; LongBench v2 results average four runs over the same 503 examples with decoding seed 0 and the official script's 0.25 credit for unparseable answers enabled.

The distinguishing move versus prior work is that the reward is white-box and information-theoretic, derived from a frozen verifier, rather than either a learned reward model (which the authors say is unstable and needs an expensive co-evolution loop, as in the concurrent EAPO) or a binary chunk-selection signal (as in LongRLVR).

Why This Matters

The paper targets a known failure mode in long-context RL: outcome-only rewards are too sparse to teach a model which parts of a huge input actually matter, so models fall back on superficial query matching instead of reading the context. LongR shows that a cheap, annotation-free dense signal can be extracted from a small frozen model and that it composes with standard RL algorithms without touching the architecture or the data format. It also argues against structural constraints (pre-chunking, special token formats) that diverge from natural language reasoning and from mainstream post-training distributions.

Real-world applications:

• Long-dialogue understanding — tracking and citing specific turns in extended multi-session conversations.
• Structured data and document analysis — synthesizing insights across heterogeneous documents where the supporting passage must be located and grounded.
• Repository-scale code debugging — reasoning over large codebases where the relevant lines must be found before a fix can be justified.
• Autonomous agents on long trajectories — sustaining coherent, evidence-grounded actions across many steps, which the paper cites as a motivation.

Industry relevance: the reward requires only a small frozen verifier (4B is sufficient) and generic instruction data, so it avoids costly long-context annotation and data engineering; it also plugs into existing RL stacks (the authors use verl and DAPO) and existing post-training pipelines, which lowers adoption cost for teams already running RL fine-tuning.

Future Directions

• Scaling beyond 32K. The authors explicitly list this as a limitation: due to computational constraints, RL training was only run at context lengths up to 32K, and they plan to explore longer contexts.
• Verifier choice and cost. Performance saturates between 4B and 32B verifiers, leaving open whether a cheaper or better-calibrated verifier (or a different estimation method for the prior and posterior) could shift that trade-off.
• Reward shaping variants. The relative-information-gain reward has no finite lower bound; a lower-clipped variant is mentioned in Appendix I (beyond the provided content), suggesting open questions about bounding and stabilizing the dense signal — especially given the paper's observation that irrelevant quotations can still receive positive rewards.
• Where gains concentrate. CISPO and GSPO improve different regions of the difficulty/length spectrum than DAPO, raising the question of whether the RL objective should be chosen deliberately for a target workload rather than for overall average.

Target Audience

Researchers and engineers working on RL post-training for LLMs and on long-context capabilities specifically. It is most useful to readers who already understand outcome-reward RL for reasoning (e.g., DAPO/GRPO-style training) and want a concrete, low-annotation recipe for adding dense credit assignment over retrieved evidence; it is also relevant to practitioners building retrieval-grounded or agentic systems who need models to cite and rely on document spans rather than answer from parametric memory. Readers new to RL fine-tuning will find the high-level idea accessible but the reward derivation and experimental comparisons demanding.

Authors’ abstract

Reinforcement Learning (RL) has emerged as a key driver for LLM reasoning. This capability is equally pivotal in long-context scenarios---such as long-dialogue understanding and structured data analysis---where the challenge extends beyond consuming tokens to performing rigorous deduction. While existing efforts focus on data synthesis or architectural changes, recent work points out that relying solely on sparse, outcome-only rewards yields limited gains, as such coarse signals are often insufficient to effectively guide the complex long-context reasoning. To address this, we propose LongR, a unified framework that enhances long-context performance by integrating a dynamic ``Think-and-Read'' mechanism---which interleaves reasoning with document consultation---with a contextual density reward based on relative information gain to quantify the utility of the relevant documents, without evidence annotations. Empirically, LongR achieves a gain of over 9\% on LongBench v2 and consistent improvements on RULER and InfiniteBench, demonstrating robust efficiency in navigating extensive contexts. Furthermore, LongR consistently enhances performance across diverse RL algorithms (e.g., DAPO, GSPO). Finally, we conduct in-depth analyses to investigate the impact of reasoning chain length on efficiency and the model's robustness against distractors.