LM-mixup: Text Data Augmentation via Language Model based Mixup

AI summary

Overview

Research area: Natural language processing and data-centric AI, specifically instruction-tuning data augmentation and data curation for large language models.

Technical level: Intermediate. The dataset construction and evaluation are accessible to a general machine learning reader, while the reinforcement learning reward design and clustering procedure require familiarity with supervised fine-tuning, GRPO, and embedding-based retrieval.

Scope: This paper defines the task of Instruction Distillation, builds the 144K-sample Mixture dataset for it, and introduces LM-Mixup, a pipeline that fuses multiple low-quality instruction inputs into a single high-quality instruction-output pair.

What This Paper Is About

Real-world instruction datasets contain large amounts of redundant, noisy, or shallow data that is usually discarded, while genuinely high-quality instruction data remains scarce and expensive. The paper asks whether those low-quality samples can be turned into a valuable training resource instead of being thrown away. Its goal is to transform multiple topic-related but imperfect inputs into one coherent, information-dense instruction-output pair, and to show that training on such distilled data beats training on the full original dataset.

Key Contributions

1. A new task definition: Instruction Distillation. Given a set of low-quality samples describing the same topic or task (X = {ℓ1, …, ℓk}), the goal is to learn a generator that produces a single high-quality instruction-response pair that preserves salient information, denoises conflicts, and follows task-specific format.

2. The Mixture dataset. A Wikipedia-based dataset of 144,884 instances across five task types (QA pairs, True/False (TFQ), Paragraph, Multiple-Choice Question (MCQ), and Category-Statement (CS)), with hierarchical mappings from multiple low-quality variants to one high-quality target. Each high-quality pair has 2 to 20 controlled low-quality variants, optional chain-of-thought supervision, cross-topic mixing, and noise injection.

3. The LM-Mixup method. Built on Qwen-2.5-1.5B-Instruct, it performs cold-start supervised fine-tuning on a subset of Mixture, then optimizes with Group Relative Policy Optimization (GRPO) using three reward signals: quality, semantic alignment, and format compliance, with weights λq = 0.5, λa = 0.4, and λf = 0.1.

4. Evidence that low-quality data can be a resource. Fine-tuning downstream models on distilled data plus original high-quality data, totaling only about 3% of a 300K pool, matches or exceeds full-dataset training and advanced data selection methods.

Main Findings

• LM-Mixup leads on the Mixture test set. It reaches an average score of 3.66 across the five tasks (cs 3.85, mcq 3.55, para 3.31, qa 4.17, tfq 3.32), above Qwen-2.5-7B-SFT (3.46), GPT-4o-mini (3.37), Qwen-2.5-1.5B-SFT (3.28), Qwen-2.5-7B-Instruct (3.28), LLaMA-3.1-8B-Instruct (3.27), LLaMA-3.2-3B-Instruct (3.21), and DeepSeek-R1-Distill-Qwen-7B (3.02). The test set is a held-out, non-overlapping 20% split of Mixture, rated by ChatGPT-4o-mini.

• Distilled low-quality data beats high-quality-only baselines on OpenLLM. With LLaMA-3.1-8B as the base model and a 10K fine-tuning set, the 50% MIXUP + 50% ORI setting achieves the top average of 61.5 ± 0.1, exceeding DS2 (61.4), DS2 w/o curation (60.2), Deita (59.7), AlpaGasus (58.1), LESS (58.0), Full data 300K (57.7), and Random Selection (57.4).

• Mixup outperforms the full 300K data pool with 3.3% of the data. The best mixup configuration uses only 10K training samples yet surpasses the 300K full-data-pool baseline for both LLaMA-3.1-8B and Mistral-7B. The abstract describes the distilled data as about 3% of the entire dataset.

• Applying LM-Mixup to low-quality data helps at every mixture ratio. In the 70% LOW + 30% ORI setting, the average rises from 53.6 to 59.9 (up 6.3); the 50% setting rises by 3.6, and the 30% setting by 1.5.

• The mixup pipeline beats standard augmentation and untrained generation. Mixup + ORI configurations (59.9, 61.5, 58.5) exceed Back-translation (57.7), EDA (56.6), and Rephrasing (56.5), and far exceed the zero-shot Qwen-1.5B-Instruct BASE settings (47.1, 45.4, 45.6).

• Both rewards are necessary. Removing the alignment reward causes reward hacking, where the model memorizes reference answers regardless of input and loses semantic similarity with ground truth. Removing the quality reward makes the model behave like standard SFT with limited quality improvement.

• Scaling helps. Extending the pipeline from Qwen-2.5-1.5B-Instruct to Qwen-2.5-7B-Instruct raises the Mixture average from 3.66 to 4.18.

• LLM rating bias has limited effect in this setup. Re-running the MIXUP 70% + ORI 30% experiments with the DS2 pipeline produces only marginal changes in overall performance. The authors attribute this to GRPO's multi-dimensional rewards attenuating upstream rating noise and to diversity being governed by embedding-based long-tail selection.

• Not reported: Training hardware, compute budget, GRPO hyperparameters such as the KL coefficient β and clipping parameter, and the exact capacity bounds cmin and cmax used in clustering are not given in the provided content.

Methodology in Plain English

The authors start from English Wikipedia, keeping only plain text, filtering out entries shorter than 200 tokens, and removing duplicates with exact matching plus Sentence-BERT near-duplicate filtering (discarding cosine similarity above 0.95). About 10,000 Wikipedia entries survive. Each article is split into sentences and greedily concatenated into segments of up to 512 tokens (cl100k_base tokenizer), with roughly 10% overlap; segments under 128 tokens are merged and over-long sentences are split at weaker punctuation.

Those segments are converted into task-specific samples by prompting ChatGPT-4o-mini. Every candidate is scored by ChatGPT-4o-mini on dimensions including rarity, complexity, and informativeness, aggregated into a single rating discretized to a 1-5 scale; only scores of 4 or above count as high quality. For each high-quality target, ChatGPT-4o-mini generates 2 to 20 degraded variants that preserve the topic but reduce information density or reasoning completeness. Cross-topic fusion and surface perturbations (spelling variations, synonym substitutions, minor formatting shifts) add robustness, with some clean samples retained.

LM-Mixup is then trained in two stages. First, cold-start supervised fine-tuning linearizes the k low-quality inputs as context and trains the model to generate the high-quality output with standard autoregressive cross-entropy loss, giving a stable initial policy. Second, GRPO fine-tuning samples multiple candidate outputs per input and scores them with three rewards: a quality reward estimated by a KNN-Bayes scheme that retrieves neighbors with pre-computed LLM scores and maps the expected score through a piecewise function (λ = 4, κ = 3, α = 1, β = 0.3); a semantic alignment reward using the BAAI/bge-m3 embedding model and a cosine similarity threshold; and a format compliance reward that checks the <think>...</think><answer>...</answer> template with regular expressions. GRPO normalizes rewards within each group and clips the importance ratio, with KL regularization against the reference policy.

For downstream use, a Capacity-Constrained Clustering method groups low-quality inputs: samples are encoded, a target capacity vector is drawn from a truncated normal distribution, MiniBatchKMeans produces initial centers, and samples are reassigned to similar clusters under capacity constraints with refinement steps to avoid imbalance and over-fragmentation. Models are evaluated on the Mixture test set and on OpenLLM Leaderboard tasks using accuracy on MMLU, BBH, and GSM8K, the Informative-Truthful Rate on TruthfulQA, and F1 on TyDiQA, with results averaged over three runs.

Why This Matters

Research impact. The paper reframes low-quality data as a resource rather than waste, and positions Instruction Distillation as distinct from both data augmentation (which expands sample count while preserving semantic content) and data curation (which refines labels for fixed content). Instruction distillation instead reduces volume through information aggregation while enriching semantic content. This suggests a different axis for data-centric AI research than selection, filtering, or repair.

Real-world applications (from the paper's framing):

• Low-resource machine translation, where high-quality parallel data is a recognized bottleneck.
• Domain-specific tasks such as law and medicine, where curated expert data is scarce.
• Instruction tuning on messy crowdsourced or web-scraped corpora containing redundant and shallow instructions.
• Converting discarded portions of large open instruction pools (Flan V2, Open Assistant 1, WizardLM, Dolly, Stanford Alpaca, totaling 300K in this study) into compact, high-density supervision.

Industry relevance. The finding that 10K distilled samples can beat a 300K pool means lower fine-tuning costs, shorter training cycles, and direct reuse of data that organizations would otherwise discard. The public release of code and data at https://github.com/yuu250/LM-mixup makes the pipeline directly applicable to proprietary instruction corpora.

Future Directions

• Reduce LLM-as-judge bias further. The authors note that rating bias is real and suggest multi-judge ensembling, cross-model adjudication, or light human spot-checks to unlock additional gains.
• Tune the data-composition ratio systematically. The paper evaluates only three representative settings (30%, 50%, 70% additional data) and states that a per-model, per-benchmark exhaustive tuning of the ratio was not performed.
• Extend beyond the current task set and scale. Mixture covers five task types and the largest LM-Mixup variant tested is Qwen-2.5-7B-Instruct, leaving larger models, additional task formats, and other domains unexplored in the reported content.
• Separate synthetic from real low-quality data. The training variants are synthetic, generated by ChatGPT-4o-mini rather than collected from real low-quality pools, and the paper reports gains on real pools (via the 300K study) but leaves broader validation of the synthetic-to-real gap open.

Target Audience

This paper suits NLP and LLM practitioners who work on instruction tuning, data curation, and dataset construction; researchers in data-centric AI interested in augmenting or reusing noisy corpora; and engineers building fine-tuning pipelines under budget or data constraints. Readers with some background in supervised fine-tuning and reinforcement learning will get the most from the reward design and GRPO sections, while the dataset construction and benchmark results are accessible to a broader audience.

Authors’ abstract

Instruction tuning is crucial for aligning Large Language Models (LLMs), yet the quality of instruction-following data varies significantly. While high-quality data is paramount, it is often scarce; conversely, abundant low-quality data is frequently discarded, leading to substantial information loss. Existing data augmentation methods struggle to augment this low-quality data effectively, and the evaluation of such techniques remains poorly defined. To address this, we formally define the task of Instruction Distillation: distilling multiple low-quality and redundant inputs into high-quality and coherent instruction-output pairs. Specifically, we introduce a comprehensive data construction pipeline to create MIXTURE, a 144K-sample dataset pairing low-quality or semantically redundant imperfect instruction clusters with their high-quality distillations. We then introduce LM-Mixup, by first performing supervised fine-tuning on MIXTURE and then optimizing it with reinforcement learning. This process uses three complementary reward signals: quality, semantic alignment, and format compliance, via Group Relative Policy Optimization (GRPO). We demonstrate that LM-Mixup effectively augments imperfect datasets: fine-tuning LLMs on its distilled data, which accounts for only about 3% of the entire dataset, not only surpasses full-dataset training but also competes with state-of-the-art high-quality data selection methods across multiple benchmarks. Our work establishes that low-quality data is a valuable resource when properly distilled and augmented with LM-Mixup, significantly enhancing the efficiency and performance of instruction-tuned LLMs.