Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifies

AI summary

Overview

Research area: Large language model post-training (supervised fine-tuning and reinforcement learning), compositional generalization, and knowledge-intensive multi-hop reasoning.

Technical level: Intermediate. Readers should be comfortable with the SFT-then-RL pipeline, reward-based policy optimization, and standard generalization terminology (in-distribution, compositional, zero-shot).

Scope: This paper asks whether reinforcement learning creates genuinely new reasoning skills or merely sharpens what supervised fine-tuning already installed, using a controlled synthetic biography dataset that separates parametric memory from in-context reasoning.

What This Paper Is About

The paper studies "Complementary Reasoning," the ability to combine facts stored in a model's weights with new facts supplied in the context window, which the authors identify as a prerequisite for continual learning and retrieval-augmented generation. The authors build a contamination-free synthetic dataset and split this capability into two atomic skills, Parametric Reasoning and Contextual Reasoning, to test when reinforcement learning produces real generalization and when it only amplifies existing behavior. The goal is to determine what training recipe actually lets a model handle reasoning paths it has never seen.

Key Contributions

1. The authors define Complementary Reasoning and release a semantic-synthetic biography dataset that decouples reasoning into atomic Parametric and Contextual skills, using novel entities with real-world relations to prevent pre-training contamination while preserving natural-language semantics.

2. They document the "SFT Generalization Paradox": direct supervised fine-tuning on the composite task reaches high in-distribution accuracy (90.30% IID) but collapses to 18.41% on structural zero-shot paths, while training on the atomic skills transfers far better after RL.

3. They establish that RL acts as a skill synthesizer rather than a mere amplifier, but only when the base model has first acquired sufficient atomic skills through SFT, and they support this with pass@k analysis, sample-efficiency sweeps, and error-position analysis.

4. They propose the recipe SFT on Mem + Ctx followed by RL on Comp data as a scalable alternative to collecting expensive traces of complex reasoning.

Main Findings

• SFT memorizes rather than generalizes. A model trained directly on the composite task (SFT_Comp) achieves 90.30% IID and 76.25% Composition accuracy but only 18.41% on zero-shot paths with unseen relations.

• Atomic skills do not compose spontaneously. Training on Parametric plus Contextual data (SFT_Mem+Ctx) yields only 35.18% IID, 28.20% Composition, and 24.07% zero-shot on the Comp test set, lagging far behind the explicit composite baseline of 90.26% IID.

• Complementary Reasoning is data-hungry relative to its components. Reaching high IID performance on Comp requires 180,919 training samples, compared with 88,031 for Parametric and 2,651 for Contextual data.

• RL benefits atomic-first training enormously. Across all SFT/RL data partitions (x from 10% to 90%), SFT_Mem+Ctx gains substantially during the RL phase on Comp data, while SFT_Comp gains almost nothing.

• Zero-shot gains favor atomic-first. In the structural zero-shot setting, SFT_Mem+Ctx consistently outperforms SFT_Comp regardless of data scale.

• The SFT Generalization Paradox. Once the SFT data ratio x exceeds 70%, SFT_Comp slightly surpasses SFT_Mem+Ctx in IID accuracy, yet still collapses in zero-shot, indicating overfitting to specific training relation paths.

• Both atomic skills are necessary. Models trained only on Mem or only on Ctx fail to generalize after identical RL (12.8k Comp samples). Even SFT_10%Comp and SFT_20%Comp, which match SFT_Mem+Ctx's pre-RL performance, show negligible post-RL gains, while SFT_Mem+Ctx nearly doubles its performance across all levels.

• RL is uniquely necessary. With identical Comp samples, further SFT yields the highest IID performance but lags badly in zero-shot, while RL more than doubles SFT and LoRA (rank 256) performance on unseen combinations.

• Sample efficiency. With a fixed RL subset of roughly 90k Comp samples, SFT_Mem+Ctx outperforms SFT_Comp across all SFT budgets, and is "primed" for RL even at roughly 18k SFT samples. With fewer than 10% of composite data, SFT_Mem+Ctx matches the upper bound of SFT_Comp trained on the entire dataset, and just 50 samples already produce significant improvement.

• Pass@k shows synthesis versus amplification. For SFT_Mem+Ctx, RL performance stays above SFT even at k = 2^9, indicating a mechanism absent from the SFT distribution. For SFT_Comp, the curves converge by k = 2^5, indicating RL only boosted likelihood of paths the SFT model could already sample.

• Error profiles shift. SFT_Comp, SFT_Comp → RL_Comp, and SFT_Mem+Ctx all show over 85% contextual errors with failures at under 55% progress. SFT_Mem+Ctx → RL_Comp flips this to 70% parametric errors occurring later, at 71.8% progress. SFT_Comp terminates 62% of its errors at the very first hop.

• Replication on a second model family. With Llama-3.2-1B, SFT_Mem+Ctx → RL_Comp reaches 36.93 zero-shot versus 17.10 for SFT_Comp → RL_Comp.

• Longer-hop extrapolation. On 4-hop and 5-hop paths, SFT_Mem+Ctx → RL_Comp reaches 61.88 and 66.67 Composition accuracy versus 50.50 and 46.67 for SFT_Comp → RL_Comp; zero-shot 4-hop is 47.87 versus 35.64.

Methodology in Plain English

To rule out the possibility that a model is answering from memorized internet text, the authors build a world from scratch. They define 39 relations (including eight symmetric relations such as spouse and sibling, and eight inverse pairs such as child and parent), populate a knowledge graph with entirely synthetic people using the Python Faker library and GPT-4o, and render it as natural-language biographies. Relations are split into a parametric set (facts meant to live in model weights) and a contextual set (facts meant to arrive in the prompt). They generate 10k biographies for each type with 5k shared entities so the two sources can be bridged. Questions are built by walking 2-to-5-hop relational paths through the graph, and targets include chain-of-thought steps, so the researchers can see whether a fact was pulled from weights or from context.

They then define three difficulty tiers. In IID, the exact relational path was seen in training, though with different entities. In Composition, each individual relation was seen, but the specific sequence was not. In Zero-shot, at least one relation never appeared in any training question.

The training pipeline is standard: supervised fine-tuning first, then reinforcement learning with Group Relative Policy Optimization using binary outcome rewards, evaluated by exact match. The main model is Qwen-2.5-1.5B, with Llama-3.2-1B used for replication and a Qwen scaling check across 0.5B, 1.5B, and 3B.

Why This Matters

The findings reframe a live debate in post-training. Rather than asking whether RL synthesizes or amplifies in the abstract, the paper shows the answer depends on what the underlying model already knows, and it offers a concrete diagnostic (pass@k curve divergence) for telling the two apart. For research on continual learning and retrieval-augmented generation, this suggests that the hard problem is not teaching a model to merge memory with context, but arranging the training curriculum so that RL can do it.

Real-world applications:

• Retrieval-augmented generation systems that must answer questions by combining a user's private documents with general world knowledge.
• Continual-learning assistants that absorb new facts over time without degrading existing abilities.
• Enterprise knowledge-base agents that traverse relations across internal records and general knowledge.
• Any agentic workflow that must chain several operations where the exact combination was never seen in training data.

Industry relevance: The recipe implies that teams may not need to collect large volumes of expensive multi-step reasoning traces. Investing in cheap, well-scoped atomic skill data and then applying RL to composite tasks appears to be both more effective and far more sample-efficient, which changes the cost calculus for post-training pipelines.

Future Directions

• Localizing mechanistically how RL recruits and recombines the attention heads responsible for parametric and contextual retrieval.
• Validating the recipe on naturalistic benchmarks under controlled contamination, for example news-QA splits dated strictly after a model's pre-training cutoff, to confirm transfer to deployed RAG settings.
• Studying how to jointly mix Comp, Mem, and Ctx data during training, and how that mixing schedule should evolve across the SFT and RL phases.
• Extending the analysis beyond the controlled multi-hop factual setting to open-ended compositional reasoning where the boundary between stored and supplied knowledge is harder to enforce.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning for reasoning, retrieval-augmented generation, and continual learning. It is also useful for practitioners who must decide how to allocate limited annotation budget between atomic-skill data and complex reasoning traces, and for anyone interested in rigorous, contamination-free evaluation of whether a model is generalizing or memorizing.

Authors’ abstract

Does Reinforcement Learning (RL) merely amplify existing skills, or synthesize novel skills? We investigate this question through the lens of Complementary Reasoning: the critical practical capability of integrating internal knowledge with external context, a prerequisite for reliable Continual Learning and Retrieval-Augmented Generation. To avoid pre-training contamination, we construct a controlled semanticsynthetic dataset of biographies and decompose this capability into two atomic skills: Parametric Reasoning (retrieving facts encoded in model weights) and Contextual Reasoning (processing novel in-context information). We present two findings. First, models supervised directly on the composite task reach high accuracy on seen facts and reasoning paths (90%) but collapse on novel facts and reasoning paths (18%), indicating that Supervised Fine-Tuning (SFT) relies on rote memorization rather than genuine skill integration. Second, RL bridges this generalization gap, acting as a skill synthesizer rather than a mere amplifier--but only under a strict prerequisite: it synthesizes new composite strategies only when the base model has first mastered the independent atomic skills via SFT. These results suggest that decoupled atomic training followed by RL offers a scalable path to complex novel reasoning.