Learning from Synthetic Data Improves Multi-hop Reasoning

AI summary

Authors: Anmol Kabra, Yilun Yin, Albert Gong, Kamilė Stankevičiūtė, Dongyoung Go, Johann Lee, Katie Z. Luo, Carla P. Gomes, Kilian Q. Weinberger arXiv: 2603.02091v1 [cs.LG], 02 Mar 2026

Overview

Research area: Machine learning — reinforcement learning fine-tuning of large language models (LLMs), synthetic training data, and multi-hop reasoning.

Technical level: Intermediate. The paper is readable with a basic understanding of LLMs and reinforcement learning, though it assumes familiarity with concepts such as RLVR (reinforcement learning with verifiable rewards), GRPO, supervised fine-tuning, and multi-hop question answering.

Scope: The paper studies whether RL fine-tuning on rule-generated synthetic data — questions built from templates, context-free grammars, and logic programs over entirely fictional entities — can teach LLMs a general "knowledge composition" skill that transfers to real-world multi-hop reasoning benchmarks.

What This Paper Is About

Reinforcement learning fine-tuning has substantially improved LLM reasoning, but it depends on abundant high-quality verifiable data, which is scarce, expensive, or unreliable depending on whether it comes from human annotation, frontier LLMs, or LLM-based verifiers. The authors ask whether a much cheaper alternative — synthetic data generated by rules on fictional worlds, containing no real facts and only simple language — can still teach models something useful. Their answer is yes: models trained only on fictional multi-hop problems get measurably better at real-world multi-hop question answering.

Key Contributions

1. Demonstrating synthetic-to-real transfer in multi-hop reasoning. The authors RL fine-tune four LLMs (Qwen3-0.6B, Qwen3-1.7B, Qwen2.5-1.5B-Instruct, and Phi-4-mini-reasoning) on four rule-generated synthetic datasets and evaluate on five real-world multi-hop benchmarks, finding consistent improvement across model families and sizes (0.6 to 4 billion parameters), with the smallest models benefiting the most.

2. Isolating knowledge composition as the transferred skill. By evaluating on held-out synthetic test sets with zero factual overlap with training data, and on question complexities never seen during fine-tuning, the authors show that models cannot be succeeding by surfacing memorized pretraining knowledge — they must be learning composition as a domain-independent skill.

3. Comparing RL fine-tuning against supervised fine-tuning (SFT) and against format-only reward. They show that SFT on ground-truth synthetic solution traces improves synthetic task performance but does not transfer to real-world benchmarks, while format-only reward training helps only the model that was not already producing correct output formatting (Qwen2.5-1.5B-Instruct).

4. Showing that synthetic data scales and improves reasoning traces. Real-world benchmark performance improves monotonically with the number of synthetic training samples with no observed overfitting, and reasoning traces contain increasing proportions of correct intermediate answers as training progresses, even though the reward only supervises the final answer.

Main Findings

• Transfer is large. A Qwen3-0.6B model RL fine-tuned on PhantomWiki achieves relative F1 score improvements of 56% to 131% on real-world benchmarks, despite the synthetic training data containing no factual knowledge useful to those benchmarks.

• Transfer holds across generators, benchmarks, and models. Training on PhantomWiki, GSM-∞, RG-Family, and RG-Knights all transfer to HotpotQA, 2WikiMultihopQA, MuSiQue, CofCA, and SynthWorlds-RM, across Qwen and Phi model families.

• Format-only training does not explain the gains for most models. In the ablation with binary reward for using <answer>...</answer> tags over 3K steps, Qwen3-0.6B, Qwen3-1.7B, and Phi-4-mini-reasoning showed little or no improvement, while Qwen2.5-1.5B-Instruct improved substantially (for example, HotpotQA F1 went from 0.02 ± 0.01 to 0.43 ± 0.02). The authors conclude that Qwen2.5-1.5B-Instruct's transfer involves learning output formatting in addition to correct answering, whereas all of the transfer for Qwen3 and Phi-4-mini-reasoning must come from learning to compose knowledge.

• Models generalize to unseen knowledge and unseen difficulty. On held-out PhantomWiki and GSM-∞ test sets that share no facts with training data, intermediate checkpoints improve across all complexity levels, including out-of-domain difficulties beyond those seen during training.

• SFT does not transfer. Table 2 shows SFT improving GSM-∞ accuracy dramatically (Qwen3-0.6B from 0.0241 to 0.7735; Qwen3-1.7B from 0.1354 to 0.7742) while HotpotQA F1 changed from 0.3569 to 0.3995 and from 0.5935 to 0.5761 respectively. RL fine-tuning improved both: GSM-∞ accuracy of 0.6452 and 0.8532, and HotpotQA F1 of 0.4786 and 0.6664. The authors describe SFT as overfitting to the synthetic task.

• Reasoning traces become more grounded. As RL fine-tuning progresses on PhantomWiki and GSM-∞, models produce reasoning traces containing increasingly higher proportions of the ground-truth intermediate answers from MuSiQue and CofCA — an effect of outcome-only reward.

• Performance scales with synthetic data. Qwen3-0.6B, evaluated every 500 training steps on a single epoch over each dataset, improves steadily on all real-world benchmarks with more training steps, equivalently more synthetic samples, with no sign of overfitting.

• In-domain real-world data still wins. The authors note that RL fine-tuning on in-domain real-world data outperforms synthetic-to-real transfer, though such data is expensive and limited in scale.

• Model malleability differs. Smaller models such as Qwen3-0.6B start weaker but improve steeply, while larger models such as Qwen3-1.7B are stronger at initialization but improve more gradually.

Methodology in Plain English

The authors take four LLMs and fine-tune each of them with reinforcement learning, specifically GRPO (a variant of PPO that estimates advantages from a group of sampled completions per prompt) as implemented in the Hugging Face TRL v0.21.0 GRPOTrainer. The KL-divergence penalty β is set to 0, advantages are computed per batch on each GPU, and each training run uses 10K samples for a single epoch. Training ran on 4 NVIDIA H100 GPUs with 80GB VRAM each, taking roughly one day for 1 to 4B parameter models (about 100 H100 hours for the longer chain-of-thought models, and about 20 H100 hours for Qwen2.5-1.5B-Instruct).

The training questions come entirely from procedural generators. PhantomWiki builds random universes of 25 fictional people each, described in Wikipedia-like articles, and generates multi-hop questions with verifiable answers; the authors generated 34 universes with 330 questions each, held out 3 universes of about 1K samples for testing, and used 31 universes with 10K samples for training. GSM-∞ builds random computation graphs and converts them into grade-school-style math word problems of 2 to 20 arithmetic operations, yielding about 600 questions per difficulty level and about 12.5K problems total, of which a random 10K were used for training. ReasoningGym supplied two logical environments: Family-Relationships (family graphs of sizes 3 to 20) and Knights-Knaves (balanced across configurations of 2 to 6 people), 10K training samples each.

Each prompt presents all relevant evidence, instructs the model to place its final answer in <answer>...</answer> tags, and includes chain-of-thought examples (3 ground-truth examples for GSM-∞, 11 curated by Gong et al. for PhantomWiki, and 11 written by the authors for the ReasoningGym environments). Rewards are binary exact match for GSM-∞, RG-Family, and RG-Knights, and F1 score for PhantomWiki because those questions can have multiple answers.

Evaluation uses the distractor versions of HotpotQA (over 100K questions, 2-hop), 2WikiMultihopQA (over 190K questions across compositional, inference, comparison, and bridge-comparison categories), MuSiQue (2-4 hop, Answerable split), CounterfactualQA (rewritten 2-4 hop questions with factual knowledge removed), and SynthWorlds-RM (2-6 hops and constraints from Wikidata graph motifs). For each, 500 question-answer pairs were randomly subsampled, and each synthetic dataset fine-tuning was run with 2 random training seeds.

Why This Matters

Impact on research. The paper challenges the assumption that useful reasoning training data must be human-written, LLM-distilled, or grounded in real facts. It suggests the field can scale reinforcement learning for LLMs along a new axis — the quantity of free, unlimited, fully verifiable rule-generated training data — and it offers a mechanistic account (knowledge composition) and supporting evidence (unseen-world generalization, improved intermediate-step groundedness, monotonic scaling) rather than just a benchmark result.

Real-world applications:

• Retrieval-augmented question answering over document collections, where systems must chain information across multiple passages to answer a query.
• Enterprise and scientific knowledge assistants that combine facts scattered across many sources, a setting that closely mirrors the multi-hop benchmarks used here.
• Domain-specific reasoning systems where labeled data is scarce or expensive — the paper's approach lets experts design verifiable task generators rather than hand-label individual examples.
• Resource-constrained training pipelines, since the synthetic data requires no GPU or API access to produce and can be generated on any standard computer for free.

Industry relevance. The practical bottleneck for reinforcement learning with verifiable rewards is data cost and verifier reliability, not compute alone. This work points to a workflow in which domain experts write generators instead of annotations, which is cheaper, contamination-resistant (the synthetic benchmarks provide unlimited fresh examples), and easier to scale. It is especially relevant for teams training small models, which the paper finds benefit the most.

Future Directions

• Understanding how synthetic and real-world knowledge-intensive data interact during training. The authors note that models can learn memorization and generalization simultaneously, and that memorization may complement composition in standard settings even while hurting performance in counterfactual settings.

• Testing transfer for other reasoning skills. The paper suggests exploring whether causal reasoning, counterfactual inference, or analogical thinking show similar synthetic-to-real transfer.

• Identifying boundary conditions. When does this transfer work, when does it fail, and how does it interact with other training objectives — the authors call this an important open question.

• Explaining model malleability. Why smaller models improve steeply from a weak start while larger models improve gradually from a strong start remains an open problem, as does how model initialization influences RL fine-tuning.

• Closing the gap with in-domain real-world data. The authors report that in-domain real-world RL fine-tuning still outperforms synthetic-to-real transfer, leaving room for improvement.

Target Audience

This paper is most useful to machine learning researchers and engineers working on LLM post-training, reinforcement learning with verifiable rewards, and reasoning evaluation. It is also relevant to practitioners building question-answering or retrieval systems who face data-scarcity or annotation-cost constraints, and to researchers studying synthetic data generation and generalization. Readers without a background in LLM training or reinforcement learning will still follow the main argument, but the experimental setup and the SFT-versus-RL comparison will require some familiarity with the field.

Authors’ abstract

Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.