LongZero: Enhancing Long-Context Reasoning in LLMs from Zero Data

AI summary

Overview

Research area: Reinforcement learning for large language models, specifically long-context reasoning (Natural Language Processing).

Technical level: Advanced. The paper assumes familiarity with reinforcement learning with verifiable rewards (RLVR), GRPO, and long-context benchmarks, though the central idea is explained geometrically enough to follow at an intermediate level.

Scope: LongZero is a training method that lets an LLM improve its own long-context reasoning by writing its own long-context tasks -- question, evidence, and answer first, long documents second -- with no external corpus, human annotation, or teacher model.

What This Paper Is About

Reinforcement learning can strengthen long-context reasoning in LLMs, but it demands training data that are long, diverse, difficult, and automatically verifiable, and such data are expensive to annotate, scarce in the wild, and normally synthesized only with the help of external corpora or a stronger teacher model. LongZero asks whether an LLM can improve its long-context reasoning "from zero data" -- that is, from tasks it proposes and answers itself. The key trick is to reverse the usual generation order: instead of writing a long document and then asking a question about it, the model first writes a compact structured specification of the question, its supporting evidence, and the answer, and only afterwards expands the evidence into a long multi-document context, so the answer is fixed and correct before any long text exists.

Key Contributions

1. The LongZero method itself: a long-context RL procedure in which the model being trained first synthesizes the question, evidence, and answer in a structured format, then expands them into long-context data, and is finally trained on those data with GRPO.

2. A dual data recipe: self-written multi-hop reasoning tasks (multi-hop questions over a fictional world, answered from eight model-written documents) combined with self-written multi-round retrieval tasks in the style of MRCR (reproducing one specific earlier response in a long, model-written conversation), plus difficulty filtering that keeps only questions the model answers inconsistently.

3. An empirical evaluation across two base models and five long-context reasoning benchmarks (LB1, LBv2, FRAMES, AA-LCR, LongReason), showing that LongZero generalizes from self-generated tasks to general long-context reasoning tasks.

4. A comparison against two synthetic-data baselines built from external corpora (Enhanced HotpotQA from LoongRL, and teacher-synthesized QA written and verified by Qwen3.5-397B-A17B over Ultra-FineWeb clusters from SPELL), showing that LongZero outperforms both on average without any external data.

Main Findings

• Best average score on both base models. With Qwen3-4B-Thinking-2507, LongZero reaches an average of 57.42 versus 54.68 for the base model, 56.57 for Enhanced HotpotQA, and 55.99 for teacher-synthesized QA. With Qwen3-8B, it reaches 45.66 versus 44.63 (base), 45.03 (Enhanced HotpotQA), and 44.83 (teacher-synthesized).

• Wins on four of five benchmarks for both models (counting a tie on AA-LCR for Qwen3-8B); the exceptions are AA-LCR on Qwen3-4B-Thinking-2507 and LBv2 on Qwen3-8B. For Qwen3-4B-Thinking-2507 the per-benchmark numbers are LB1 67.22, LBv2 43.34, FRAMES 55.76, AA-LCR 39.50, LongReason 81.30; for Qwen3-8B they are 65.49, 32.01, 49.39, 3.00, and 78.40.

• A larger teacher does not guarantee better data. The teacher-synthesized baseline, whose questions and answers are written and verified by a 397B-parameter model (Qwen3.5-397B-A17B), does not surpass LongZero. The authors attribute this to the mismatch between another model's questions over real documents and the policy's own ability, whereas LongZero's questions are written by the policy, verifiable by construction, and filtered to its current ability.

• Gains transfer from fictional to real documents. Every training world is fictional, yet Qwen3-4B-Thinking-2507 improves on every benchmark, including LB1 and FRAMES, whose questions concern real Wikipedia entities. Because fictional entities cannot be answered from parametric memory, the policy can only improve by reading the context more carefully.

• Gains transfer from short training contexts to long inputs. Training samples contain tens of thousands of tokens at most, while LBv2 and AA-LCR require inputs several times longer; LongZero still improves Qwen3-4B-Thinking-2507 on both.

• The stronger long-context reasoner benefits more. Qwen3-4B-Thinking-2507 improves clearly more than Qwen3-8B. Consistent with the self-synthesis loop, Qwen3-8B keeps a smaller share of its own questions as "unstable" (13.1% versus 17.7% for Qwen3-4B-Thinking-2507) and therefore needs more synthesis for the same amount of data.

• AA-LCR is near zero for Qwen3-8B across all methods. The 72K–115K-token inputs leave little room within the 131K-token YaRN window, so the model often runs out of tokens before answering: 120 of the 200 AA-LCR responses of LongZero on Qwen3-8B contain no answer.

• The two data types are complementary. On Qwen3-4B-Thinking-2507, reasoning data alone gives an average of 55.38 but falls below the base model on AA-LCR (34.00) and LongReason (78.59); retrieval data alone lowers the average to 53.85; a 1:1 mixture gives 55.95; and the 2:1 mixture used by LongZero is best at 57.42, improving over the base model on all five benchmarks.

• Self-synthesis is cheap to scale. The whole pipeline runs on the model being trained, with no calls to a larger model, so data cost scales with the size of the policy rather than with that of a teacher.

Methodology in Plain English

The authors let the model train itself in a loop with three stages.

1. Write a compact "world" first. A program samples a domain, document genres, and a style card (time period, naming style, banned names); 12 domains, 8 genres, and 12 naming styles are used, with a 40-year period starting in some decade from 1870–1990. The model then writes a strict JSON specification containing fictional entities (with deliberately confusable names, e.g., at least two people sharing a surname), atomic facts forming question chains of 3–5 hops, 2–3 distractor facts per chain, and questions with their reference answers and supporting facts. Questions refer to intermediate entities by description ("the founder of the magazine's parent press") so each hop must be resolved, and answers must be names, places, products, titles, or numbers appearing verbatim in a fact.

2. Expand into long documents. The model assigns every fact to one of 8 documents, spreading each question's supporting facts across different documents and placing distractors away from the facts they imitate. It writes each document independently in the planned genre. A program shuffles the documents and keeps a question only if its answer or an alias appears verbatim in the expanded text -- a "faithfulness gate" that discards questions broken by the expansion.

3. Build retrieval tasks. Using only 10 formats and 434 nouns borrowed from the public OpenAI MRCR dataset, the model generates a pool of responses to "write a {format} about {noun}" requests (100 responses each, 4,340 requests, 434K responses). A program assembles long conversations by inserting 8 near-duplicate "needles" at random positions among distractor turns until the conversation hits a length drawn from 8,193–16,384 tokens, then asks for the k-th needle prefixed by a random 10-character alphanumeric string. Because the program chooses the needle, the reference answer is correct by construction.

4. Keep only questions of the right difficulty. GRPO normalizes rewards within a group, so questions always answered correctly or always answered incorrectly give no gradient. The model answers each question in small waves (two samples per wave, at most four answers); a question is kept once it has both a correct and an incorrect answer. The paper also notes that consistently failed questions were frequently defective, e.g., two documents assigning conflicting values to the same attribute.

5. Train. The final dataset is 2,304 reasoning questions and 1,152 retrieval conversations (a 2:1 mixture, 3,456 samples total, with 256 and 128 held out for validation). Training uses GRPO as implemented in verl, with a DAPO-style token-level loss, the k3 KL estimator, 64 prompts × 8 responses per step, learning rate 10⁻⁶, ε = 0.2, β = 0.001, 72 steps for Qwen3-4B-Thinking-2507 and 108 for Qwen3-8B. Rewards are rule-based: whole-word matching against the reference answer plus aliases (with a 128-character length tolerance) for reasoning, and a SequenceMatcher similarity to the needle for retrieval.

Why This Matters

Long-context RL currently depends on a supply chain of external documents, existing QA datasets, knowledge graphs, or bigger teacher models -- resources that get harder to obtain exactly as the trained models get stronger. LongZero removes that dependency entirely, and it also sidesteps two confounds the authors highlight in corpus-based pipelines: models answering from parametric knowledge instead of the given context, and public text overlapping with evaluation data. Its use of fictional worlds makes memory-based shortcuts impossible, which is a clean way to force genuine reading.

Real-world applications that stand to benefit:

• Collections of reports: the paper's motivating example, where evidence for one question is scattered across many documents and near-identical distractors.
• Long conversations: customer-support or assistant transcripts where a specific earlier statement must be located and reproduced exactly.
• Whole books and long-form archives: legal, regulatory, or research corpora that exceed a model's comfortable reading length.
• Self-improving deployed systems: teams that want to keep improving a model on long-context tasks without licensing a corpus or paying for a larger teacher model.

Industry relevance: the method needs no annotation budget and no teacher API calls, and its data cost scales with the policy rather than the teacher. Code and data are released at https://github.com/zhennan1/LongZero. The authors also report that no hyperparameter was tuned on the evaluation benchmarks -- the configuration was fixed during earlier development runs and shared by all methods.

Future Directions

• Iterating the loop. Regenerating and re-filtering data with the trained model, so each round of training produces data matched to the newly stronger policy.

• Longer contexts. Synthesizing longer training contexts by writing more documents per world, since current training samples are much shorter than the LBv2 and AA-LCR inputs.

• Harder question types. Adding questions that require counting, comparison, or arithmetic over the specification, beyond the current multi-hop lookup chains.

• Better correctness guarantees. Exploring improved ways to ensure the correctness of synthesized data, since the current pipeline relies on a JSON specification, a plan repair step, and a verbatim faithfulness gate -- and question filtering discards a large share of synthesized questions (17.7% unstable for Qwen3-4B-Thinking-2507, 13.1% for Qwen3-8B).

Target Audience

Researchers and engineers working on reinforcement learning for LLMs, long-context training, and synthetic data generation will get the most from this paper, particularly those who already work with GRPO-style training and want to extend it beyond short-context domains like mathematics and code. It is also useful for practitioners who need to improve long-context performance without access to large annotated corpora or a stronger teacher model. Readers new to RLVR will find the core idea (write the answer first, expand the context later) intuitive, but the training details assume familiarity with policy-gradient methods.

Authors’ abstract

Reinforcement learning (RL) is an effective way to strengthen the long-context reasoning of large language models (LLMs), but it needs training data that are long, diverse, challenging, and verifiable. Human annotation of such data is expensive, naturally occurring data are too scarce to sustain RL, and existing synthesis methods depend on external corpora or stronger teacher models, which become harder to obtain as the trained models get stronger. Recently, RL from zero data, in which a model learns from tasks it proposes itself, has achieved excellent performance, but existing methods target short-context tasks. Therefore, we propose $\textbf{LongZero}$, a method that lets an LLM improve its own long-context reasoning without any external data. Instead of writing questions and answers over existing documents, LongZero first has the model synthesize the question and answer with its supporting evidence in a compact structured form, and only then separately expands the evidence into a long multi-document context. LongZero combines these self-written multi-hop reasoning tasks with self-written multi-round retrieval tasks and trains the model with GRPO. We evaluate LongZero on two LLMs across five long-context reasoning benchmarks and compare it with RL on data synthesized from external corpora. Experimental results show that LongZero can generalize to real tasks, and outperforms both baselines, suggesting a viable and promising path to improving long-context reasoning of LLMs.