RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

AI summary

Overview

Research area: Natural Language Processing / large language model pretraining data curation (synthetic data, reinforcement learning, web data recycling).

Technical level: Intermediate — the paper assumes familiarity with LLM pretraining pipelines, reward-based RL (GRPO), and standard evaluation benchmarks, but its central idea can be grasped without deep RL background.

Scope: The paper proposes RePro, a reinforcement-learning method that trains a small (1B–4B parameter) language model rephraser to turn low-quality web text into higher-quality pretraining data while faithfully preserving the original semantics, structure, and length, and validates it by pretraining 400M, 1.4B, and 2.8B models on DCLM-RefinedWeb.

What This Paper Is About

High-quality text for pretraining LLMs is running short, while standard data pipelines throw away much of the web as "low-quality." Prior work tried to "recycle" this discarded data by prompting very large models (e.g., Llama-3.3-70B-Instruct) to rephrase it, but that is computationally expensive and prompts alone do not reliably preserve the meaning and structure of the original text. RePro instead trains a small LM with reinforcement learning, using one quality reward and three faithfulness rewards, so it can cheaply produce rephrasings that are both higher-quality and faithful to the source.

Key Contributions

1. A new web recycling method (RePro) that trains a relatively small LM with RL to perform effective and faithful rephrasing of organic web data for pretraining, rather than relying on prompting a very large model.
2. A reward design combining quality and faithfulness: one quality reward built on DataMan, plus three faithfulness rewards — BERTScore for semantics, a structure comparison for text structure, and a length constraint — optimized jointly via GRPO.
3. Empirical demonstration of large gains at low cost: RePro doubles the gains of the state-of-the-art recycling method while using a rephraser 70× smaller, and boosts organic data efficiency by 2–3×.
4. Analyses confirming faithfulness: individual and distributional analyses show RePro preserves more critical information and better mirrors the characteristics of organic data than prompting-based methods.

Main Findings

• Downstream accuracy gains: Across 22 downstream tasks covering commonsense reasoning (3 tasks), reasoning (6), language understanding (3), reading comprehension (5), and world knowledge (5), RePro achieves 3.7%–14.5% relative accuracy gains over the organic-only baseline in the Core (DCLM Core) score.
• Doubles the prior state of the art: RePro roughly doubles the performance gains of all baselines, including ReWire, which prompts Llama-3.3-70B-Instruct with chain-of-thought reasoning.
• Much smaller and faster rephraser: RePro uses a rephraser as small as 1B parameters, 70× smaller than the 70B model ReWire prompts, and achieves up to 87.2× speedup compared to ReWire.
• Base model robustness: Initializing with OLMo2-1B-Instruct achieves comparable, and in some cases better, gains than Qwen3-4B. The authors hypothesize that although Qwen3 has stronger general capabilities, OLMo2 may be more flexible for adapting to the rephrasing task.
• Organic data efficiency: Varying the unique token budget in the 1B setup (14.4B, 21.6B, 28.8B), RePro's best performance is at 14.4B, while 21.6B is comparable and gives the largest improvement over baselines — implying a 2–3× improvement in organic data efficiency. Pushing to 28.8B causes a performance drop, attributed to including more moderate-to-low-quality data.
• Extends to longer training: In the extended 3B setup, continuing pretraining for an additional 28.8B tokens (84.7B total) grows the gap from +3.7% to +4.7%.
• WRAP can hurt at scale: In the 3B setup, WRAP can even harm performance, likely due to limited diversity and weaker preservation of the original distribution.
• Faithful key-point preservation: On 30,000 randomly sampled instances, RePro achieves the highest number of 95% supported key points, while reducing the ratio of omitted key points by up to 92% compared to prompting-based WRAP and ReWire.
• Converts low-quality data: For RePro, 63.5% of high-quality recycled examples come from low-quality organic sources, versus 31.0% for Prompting, 42.5% for WRAP, and 39.0% for ProX. RePro's top-10% fastText cutoff exceeds 0.2, while the best baseline only reaches around 0.03.
• Rewards work together, not as trade-offs: With all rewards enabled, DataMan, BERTScore, and structure rewards all steadily improve during training while the length reward stays stably high. Without faithfulness rewards, the DataMan reward quickly converges but the other three rewards drop significantly, indicating overfitting.
• Quality shift: RePro raises the proportion of recycled data scoring 5 (the maximum DataMan score) from 20% to 60%, versus only 25% for direct prompting.
• Semantic and structural fidelity: RePro maintains an average BERTScore of 0.75, versus 0.69 for direct prompting and 0.56 for RL without the BERTScore reward. Direct prompting tends to convert Markdown-style text to plain text and produce shorter, summary-like output; RL without the structure or length reward over-generates Markdown or generates much longer text with potential extraneous information. RePro's distributions align most closely with organic data.
• RL beats prompting and SFT: In the 400M ablation, prompting alone (Core 0.19847), an enhanced prompt with reward descriptions (0.19910), SFT with GPT-4o rephrasings (0.19216), and SFT+ (0.20278) all remain below full RL (0.21658). Removing faithfulness rewards drops Core to 0.19456.
• Gains come mostly from low-quality recycling: Excluding the high-quality organic subset from recycling still recovers 77.2% of the total gains. The best results, however, come from merging organic and recycled high-quality data, indicating the recycled data complements rather than replaces organic data.
• Faithful rephrasing differs from generative synthetic data: A Nemotron-CC Diverse QA baseline using Qwen3-30B-A3B helps on reading comprehension and world knowledge but lags RePro on the overall Core score (0.20847 vs. 0.21658), which the authors attribute to the absence of a faithfulness constraint.

Methodology in Plain English

The pipeline starts with an organic data pool sampled from DCLM-RefinedWeb (72B or 144B tokens). A quality classifier (DCLM-fastText, with threshold 0.018112) keeps only the high-quality portion, giving about 7.2B/14.4B tokens. Because that is not enough to fill the training budget, a rephraser model is trained to convert organic samples into recycled ones, which are filtered the same way so that the recycled high-quality pool matches the organic high-quality pool in size — a 1:1 organic-to-rephrased mixture.

The rephraser is trained with GRPO. Its reward has four parts:

• DataMan (quality): the DataMan score of the recycled text minus that of the organic text, incentivizing improvement. DataMan is a rubric-based LLM-as-a-judge scoring 13 criteria such as coherence, topic focus, and knowledge novelty.
• BERTScore (semantic faithfulness): a binary reward if BERTScore between organic and recycled text meets a threshold of 0.65.
• Structure (structural faithfulness): a binary reward from an in-context structure-comparison model that checks whether high-level structure (e.g., Markdown) is preserved.
• Length (length faithfulness): a binary reward if the recycled text is no longer than 1.25× the organic length.

The four rewards are combined with weights 3, 1, 1, 1. The rephraser is initialized from either OLMo2-1B-Instruct or Qwen3-4B. The RL dataset has 41,000 organic examples, all with DataMan scores below 5 (since 5 is the maximum and cannot be improved). GRPO uses clipping 0.2, β 0.005, and 8 rollouts per input. An optional SFT warm-up stage uses 50,000 GPT-4o-generated example rephrasings; unless stated otherwise, RePro means RL without SFT. Generation at inference uses vllm with temperature 1.0 and top-p 0.9.

Evaluation pretrains decoder-only models from scratch at 400M (28.8B tokens), 1.4B (28.8B tokens), and 2.8B (55.9B tokens, plus an extended 84.7B run) on 22 downstream tasks in zero-shot or few-shot settings, using centered accuracy averaged into a Core score. Baselines are organic-only, WRAP (Mistral-7B-Instruct-v0.1), ProX, and ReWire (Llama-3.3-70B-Instruct). Because ReWire's code is not open-sourced, the authors randomly sample 7.2B tokens from its released data, which they note may give ReWire an advantage since it effectively draws on a larger pool.

Why This Matters

Research impact. The paper reframes synthetic pretraining data as a faithful recycling problem rather than a free-form generation problem, and shows that faithfulness rewards actively prevent the distribution drift and model-collapse risks that prior work warns about. It also shows that a small model trained with RL can beat a prompted 70B model at this task, challenging the assumption that bigger prompters are always better for data synthesis.

Real-world applications:

• Cheaper pretraining corpora: Organizations with limited compute can generate extra high-quality pretraining tokens with a 1B–4B rephraser instead of paying for a 70B model, with the paper reporting up to 87.2× speedup.
• Reviving filtered web data: Corpus builders can reprocess the large fraction of Common Crawl-style data that rule-based pipelines discard, converting it into usable pretraining material.
• Accessible model development: The 2–3× organic data efficiency gain could lower the data barrier for academic labs and smaller teams training models in the 400M–2.8B range.
• Controllable data augmentation: Because faithfulness is reward-shaped, practitioners get tunable control over how much a corpus is transformed versus preserved, which matters for compliance and distributional fidelity.

Industry relevance. Pretraining data scarcity is a stated bottleneck for frontier model development. A method that increases the effective supply of high-quality tokens at low cost, and that can be tuned to preserve source characteristics, is directly relevant to any organization building or curating large pretraining corpora. The code is released at https://github.com/cxcscmu/RePro.

Future Directions

• Comparison with closed-source methods: The authors explicitly leave comparison with concurrent work such as RefineX to future work, since its data and code are not open-sourced.
• Scaling the analysis: The main results cover 400M, 1.4B, and 2.8B models; whether the advantages of faithful recycling persist at frontier scale is not established in the paper.
• Reconciling SFT and RL: Adding an SFT prior improved world-knowledge tasks but lowered the overall Core score, which the authors attribute to distillation-style GPT-4o rephrasings hindering generalization — how best to combine the two stages remains open.
• Optimizing the token budget trade-off: Performance peaked at a 14.4B unique token budget and dropped at 28.8B because of moderate-to-low quality data; how to set budgets and quality thresholds optimally is an open practical question.
• Unifying recycling and generation: The Diverse QA baseline suggests generative synthetic data and faithful recycling capture different benefits; how to combine them is not resolved.

Target Audience

Researchers and engineers working on LLM pretraining data pipelines, synthetic data generation, and RL-based model post-training. It is most useful to those who need to expand or improve a pretraining corpus under compute constraints, and to readers interested in reward design for controlling the fidelity of generated training data. Readers without background in RL or pretraining data curation will still follow the high-level argument, but will need to consult the appendices (referenced but truncated here) for full task and hyperparameter detail.

Authors’ abstract

High-quality data is a cornerstone of large language model (LLM) pretraining, yet its growth has not kept pace with the needs of frontier models. In this paper, we introduce RePro, a novel web recycling method that trains a relatively small LM with reinforcement learning to generate effective and faithful rephrasings of pretraining data. Specifically, we design one quality reward and three faithfulness rewards, optimizing the LM rephraser to convert organic data into high-quality rephrasings while maintaining its core semantics and structure. In our experiment, we train a rephraser as small as 1B parameters to recycle 72B tokens sampled from DCLM-RefinedWeb. Pretraining results on 400M, 1.4B, and 2.8B models demonstrate that RePro delivers 3.7%-14.5% relative accuracy gains over organic-only baseline on 22 downstream tasks, doubling the performance gains achieved by the state-of-the-art web recycling method that prompts a 70B rephraser. Experiments with different amounts of recycled data highlight that RePro improves organic data efficiency by 2-3x. Individual and distributional analyses validate that RePro preserves more critical information and faithfully reflects the characteristics of organic data compared to prompting-based methods. Together, these results show that RePro provides an efficient and controllable path to effectively recycle organic data for pretraining. Our code is available at https://github.com/cxcscmu/RePro.