LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

AI summary

Overview

Research area: Natural Language Processing — reinforcement learning post-training of Large Language Models for long-context reasoning (Reinforcement Learning with Verifiable Rewards, or RLVR).

Technical level: Intermediate. The paper includes formal propositions and gradient derivations, but its central argument — that answer-only rewards are too sparse to teach a model to find evidence — is explained in accessible terms, and the practical method is a reward-function modification on top of GRPO.

One-sentence scope: The paper diagnoses why standard RLVR fails at long-context tasks, proves a "vanishing grounding gradient" result, and proposes LongRLVR, which adds a verifiable reward on retrieved evidence chunks to the usual answer reward.

What This Paper Is About

Standard RLVR improves LLMs by rewarding only the correctness of the final answer, which works well when the model can rely on knowledge it already learned during pretraining. In long-context settings, however, the model must first locate relevant evidence inside a large externally provided document — a process the paper calls contextual grounding — and an outcome-only reward gives almost no useful signal for that search. LongRLVR's goal is to make the grounding step explicitly trainable by rewarding the model for selecting the correct evidence chunks before it produces its final answer.

Key Contributions

1. A formal diagnosis of RLVR's long-context failure. The paper refactors the long-context policy into a Grounding Head (selecting a subset of context chunks) and an Answer Head (producing the answer given those chunks), then proves in Proposition 1 that the answer-only gradient for any essential chunk is bounded by $p_j(1-p_j)\bar{\delta}j \Pr\theta(\mathcal{E}_j)$ — suppressed by the probability that all other prerequisite evidence has already been selected, an event the authors argue is extremely unlikely early in training.

2. LongRLVR: a verifiable context reward. Proposition 2 shows that adding a context reward $r_{\text{ctx}}(Z,G)=\sum_{c_k \in G}\alpha_k \mathbf{1}{c_k \in Z}$ to the answer reward introduces a dense gradient component $\alpha_j \mathrm{Var}(z_j) = \alpha_j p_j(1-p_j)$ that is not multiplied by the rare activation probability, so the grounding signal no longer vanishes.

3. A practical reward instantiation. The theoretical additive form is turned into a normalized reward based on the $F_\beta$ score, combined with the answer reward as $r_{\text{ctx}} = \eta \cdot F_\beta(Z,G) + (1-\eta)\cdot r_{\text{ans}}(y)\cdot F_\beta(Z,G)$, using $\eta = 0.1$ and $\beta = 2$. The first term gives unconditional grounding signal; the second unlocks full reward only when the answer is also correct.

4. A synthetic data pipeline and dataset. Because grounded QA data with explicit evidence annotations is rare, the authors build an automated pipeline (Algorithm 1) using semantic clustering, per-cluster QA generation with a generator LLM (Qwen3-235B-A22B), and two-stage rejection sampling, yielding 46K long-context QA pairs from book, arXiv, and code documents filtered to lengths between 8K and 64K tokens.

Main Findings

• LongRLVR beats SFT and naive RLVR on every model and benchmark tested. Training used LLaMA-3.1-8B, Qwen2.5-7B-1M, and Qwen2.5-14B-1M with identical synthetic data. On LongBench v2, LongRLVR reached 46.5 vs. 39.8 for naive RLVR on Qwen2.5-14B-1M, 38.6 vs. 32.4 on Qwen2.5-7B-1M, and 36.2 vs. 32.4 on LLaMA-3.1-8B.

• Large absolute gains on RULER-QA for the 14B model. LongRLVR raised Qwen2.5-14B-1M's RULER-QA average from 73.17 (naive RLVR) to 88.90, with per-length scores of 95.4 at 32K, 87.8 at 64K, and 83.5 at 128K.

• LongReason improvements as well. Qwen2.5-14B-1M went from 72.33 (naive RLVR) to 78.42 average with LongRLVR; Qwen2.5-7B-1M reached 79.22 and LLaMA-3.1-8B reached 53.23.

• Small models overtake much larger conventional models. The Qwen2.5-7B-1M LongRLVR model scored 79.22 on LongReason, above LLaMA-3.1-70B (57.59) and Qwen2.5-72B-YaRN (72.76). The 14B LongRLVR model's 46.5 on LongBench v2 exceeds the 72B model's 43.5.

• Competitive with newer reasoning models. Qwen2.5-14B-1M trained with LongRLVR reached 88.90 vs. Qwen3-14B (Thinking)'s 87.60 on RULER-QA, and 78.42 vs. 77.50 on LongReason, and was comparable to QwenLong-L1-32B.

• Answer-only rewards plateau in contextual recall. Figure 1 shows contextual recall for naive RLVR stagnating quickly on training data, capping answer accuracy.

• Both reward components are necessary. In a study on Qwen2.5-7B-1M, the answer-only variant stagnated in recall, the context-only variant ($F_\beta$ score alone) improved rapidly at first but eventually degraded without answer reward steering, and the combined LongRLVR objective gave the best and most stable LongBench v2 performance.

• Data quality matters substantially. On LongBench v2 with Qwen2.5-7B-1M, best-rated samples scored 38.6, median-rated 36.6, and worst-rated 34.8. Filtering only easy questions worked best; filtering out hard questions dropped performance to 35.8, close to no filtering at 35.6.

• Hyperparameter sensitivity has clear optima. Performance peaked at $\eta = 0.1$ (the purely synergistic $\eta=0$ was too sparse, the purely unconditional $\eta=1$ decoupled grounding from correct answers) and at $\beta = 2$ (moderately favoring recall over precision).

• Robust to chunk segmentation. Performance remained high when varying the number of chunks per document from 16 to 128, which the authors contrast with the sensitivity of traditional retrieval systems.

Methodology in Plain English

The authors start from an observation: when a model gets only a right-or-wrong signal on its final answer, and the answer depends on finding several pieces of evidence scattered through a long document, there is almost no feedback telling the model which pieces it should have found. They show mathematically that the learning signal for picking any one required chunk shrinks toward zero unless the model has already happened to pick all the other required chunks in the same attempt — a very unlikely event for a long, hard question.

Their fix is to change the output format. Instead of only producing an answer, the model is trained to first list the identifiers of the context chunks it considers useful, and then give its answer. Because the training data includes the ground-truth set of essential chunks for each question, the authors can automatically score that list. That score becomes an extra reward, on top of the answer reward, computed throughout the GRPO training run. The score itself is an $F_\beta$ measure that balances retrieving all the needed chunks against not selecting irrelevant ones, with $\beta = 2$ leaning slightly toward recall. A small weight $\eta = 0.1$ is put on the pure grounding score, with the rest of the grounding reward gated on also getting the answer right.

To make this trainable they needed data that mostly did not exist, so they built a pipeline: split long documents into chunks, embed and cluster them, ask Qwen3-235B-A22B to write candidate questions from each cluster along with the evidence chunks needed, have the same model score each candidate, keep only the best pair per document through two-stage rejection sampling, and require a quality rating above 9 out of 10. Training used GRPO with the AdamW optimizer, a constant learning rate of 1e-6, a 5-step linear warmup, prompt batch size 512, 8 sampled responses per prompt, maximum context length 64K, response length 4096, and one epoch over the 46K examples. Easy questions the base model could already answer at a judge rating of 8 or higher were removed before training. Evaluation covered RULER-QA, LongBench v2, and LongReason at 32K, 64K, and 128K lengths.

Why This Matters

Impact on research. The paper reframes long-context RLVR as a two-part problem — finding evidence and using it — rather than a single monolithic reasoning task, and backs the reframing with a provable gradient argument. That gives a concrete theoretical target for future reward design, and it suggests that other failure modes of RLVR on external-information tasks may have similar structural explanations.

Real-world applications (the data domains and benchmarks point to these settings):

• Question answering over long technical or scientific documents, such as the arXiv-length sources used in the training data.
• Code and repository understanding, drawing on the code documents used in the synthetic pipeline.
• Long-document analysis where evidence must be assembled from many scattered sections, as tested by LongBench v2.
• Multi-hop synthetic reasoning over very long inputs, as tested by RULER-QA and LongReason at 32K, 64K, and 128K.

Industry relevance. Practical value comes from the parameter-efficiency result: the 7B and 14B tuned models matched or exceeded far larger baselines on these benchmarks, meaning strong long-context behaviour may be reachable at lower inference cost. The robustness to chunk count from 16 to 128 also means the method does not require a carefully tuned retrieval segmentation scheme, which is usually a significant engineering burden in deployed retrieval-augmented systems.

Future Directions

• Tightening the joint training dynamics. The context-only variant initially improved then degraded, and the authors note the model could in principle perfect grounding in isolation; better ways to bind the two objectives than a fixed $\eta = 0.1$ are an open question.
• Extending beyond QA and multi-choice formats. Validation covered RULER-QA, LongBench v2, and LongReason; whether the context reward transfers to other grounded long-context tasks is not reported.
• Combining with agentic workflows. The related-work section explicitly states that chunk-by-chunk agentic approaches are orthogonal to this work and that LongRLVR could let such agents process larger chunks per step, scaling them to longer contexts.
• Comparing against other retrieval-reward RLVR methods. The paper cites Wang et al. (2025), which rewards the appearance of correct context in the thinking process, but no head-to-head comparison with that approach is reported.

Target Audience

This paper is most useful for researchers and engineers working on reinforcement learning post-training, long-context LLM alignment, and retrieval-augmented generation. It suits readers who already understand policy gradient methods such as PPO and GRPO and are comfortable with reward shaping, but it does not require deep familiarity with long-context attention mechanisms. Practitioners deciding how to post-train a mid-sized model for document-heavy applications will find the hyperparameter ablations ($\eta$, $\beta$, chunk count) and the data-quality ablations directly actionable.

Authors’ abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) by optimizing them against factual outcomes. However, this paradigm falters in long-context scenarios, as its reliance on internal parametric knowledge is ill-suited for tasks requiring contextual grounding--the ability to find and reason over externally provided information. We identify a key reason for this failure: a reward based solely on the final answer is too sparse to effectively guide the model for identifying relevant evidence. We formally prove that the outcome-only reward leads to significant vanishing gradients for the context grounding process, rendering learning intractable. To overcome this bottleneck, we introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward. This auxiliary signal directly incentivizes the model for selecting the correct grounding information, providing a robust learning gradient that solves the underlying optimization challenge. We validate our method on challenging long-context benchmarks using Qwen and LLaMA models. LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks, e.g., boosting a 14B model's scores on RULER-QA from 73.17 to 88.90 and on LongBench v2 from 39.8 to 46.5. Our work demonstrates that explicitly rewarding the grounding process is a critical and effective strategy for unlocking the full reasoning potential of LLMs in long-context applications. Our code is available at https://github.com/real-absolute-AI/LongRLVR.