QAQ: Bidirectional Semantic Coherence for Selecting High-Quality Synthetic Code Instructions

AI summary

Overview

Research area: Data curation and selection for training large language models, specifically synthetic code-generation and math-reasoning instruction data.

Technical level: Intermediate. The paper assumes familiarity with instruction tuning, conditional perplexity, and pass@1 code benchmarks, but its central idea (scoring a question by how well its answer predicts it) is explained in accessible terms.

Scope: The paper introduces QAQ, a data-selection framework that scores synthetic query-answer pairs by Reverse Mutual Information and by the disagreement between a strong and a weak model, and validates it on code generation (WarriorCoder, Magpie-Qwen2.5-Coder-Pro-300K) and math reasoning (OpenR1-Math-220k).

What This Paper Is About

Seedless synthetic data pipelines generate large volumes of instruction-response pairs by prompting aligned LLMs directly, which yields diversity but also noise that surface-level metrics cannot detect. Existing selectors such as Instruction-Following Difficulty (IFD) judge quality in the answer-given-query direction and therefore conflate genuinely hard tasks with hallucinated or misaligned ones. The paper's goal is to select a small, high-quality subset of synthetic data that matches full-data training performance by instead asking how well the answer explains the question.

Key Contributions

1. Identification of an overlooked dimension. The authors show that seedless synthetic pipelines such as Magpie and WarriorCoder focus on answer quality (e.g., WarriorCoder picks the highest-scored answer per query) while query quality is largely unaddressed, and they propose QAQ to fill that gap.

2. Reverse Mutual Information (RMI). They define RMI as log PPL(Q) minus log PPL(Q|A), an information-gain-like quantity measuring how much the answer reduces uncertainty about the query, and show empirically that both low and excessively high RMI values mark undesirable samples.

3. Model disagreement as a selection signal. They introduce Diff = r_s − r_w, the difference between the stratified RMI ranks assigned by a strong model (DeepSeek-Coder-6.7B-Base) and a weak model (Qwen3-0.6B), and retain samples that are valid according to the strong model yet challenging for the weak one.

4. Empirical validation across domains and a 25% recipe. Selecting 25% of data with the Diff score matches or approaches full-data performance on code benchmarks and math benchmarks, and is reported as competitive with or better than Random, IFD, RDS+, and SCAR.

Main Findings

• RMI and IFD measure different things. The Spearman rank correlation between RMI and IFD is only ρ = 0.252, which the authors attribute to length disparity, information-density differences, and model capability asymmetry rather than to redundancy between the two metrics.

• Mid-range RMI is the "sweet spot." On WarriorCoder, the middle-range variant (RMI 50-75%, 25% of data) reaches 72.56 on HumanEval+, matching the full-data figure of 72.56, and 71.43 on MBPP, versus 71.69 for full data. It outperforms Random, IFD, and SCAR at the 25% budget and exceeds RDS+ on HumanEval+ (72.56 vs. 71.34).

• Both extremes of RMI are harmful. RMI Top-25% scores 70.12 on HumanEval+ and 57.14 on MBPP+, which the authors describe as 2.4 and 2.4 points below full data; RMI Bottom-25% scores 67.68 and 54.76, described as 4.9 and 4.8 points below. The interpretation is that extreme-high RMI reflects shortcut patterns such as paraphrase and keyword echo, while low RMI reflects semantic misalignment.

• Illustrative failure cases. A top-percentile sample with RMI = 3.37 pairs a query about converting a "gritty sulla matrix" with an answer that simply restates the fabricated jargon. A misaligned sample with RMI = −0.13 pairs the query "Hi!" with an answer beginning "compensated, let's get you the help you need."

• Disagreement beats consensus. Diff-High reaches 77.44 on HumanEval, 71.95 on HumanEval+, 71.43 on MBPP, and 58.73 on MBPP+. Sum-High reaches 74.39, 68.90, 71.16, and 59.52 — a 3-point HumanEval+ gap (71.95 vs. 68.90). The overlap between the two selected sets is only 13.85%.

• Single-model RMI conflates causes. High RMI can mean the model has seen the pattern before or that a defect such as repetition is present; low RMI can mean the sample is too difficult, unrelated, corrupted, or already familiar. Comparing a strong and a weak model separates validity from learnability.

• RMI is stable across training but sensitive to model capability. Pre- and post-finetuning RMI rankings correlate at ρ = 0.9539, supporting one-time static selection. Strong and weak models show rank correlation ρ ≈ 0.7 and 76.1% overlap in their top-50% selections, yet their downstream performance differs substantially.

• The binning hyperparameter is not fragile. Performance is stable across K ∈ {5, 10, 20, 50}; with RMI 50-75% at 25% data, K = 5 gives 78.66 HumanEval, 70.73 HumanEval+, 74.07 MBPP, and 60.58 MBPP+, and K = 10 gives 77.44, 72.56, 71.43, and 58.47.

• The method does not require a strictly weaker partner. With Qwen2.5-Coder-7B as the second model, results are 77.44 HumanEval, 72.56 HumanEval+, 73.02 MBPP, and 60.32 MBPP+, matching or exceeding full data on three of four metrics. Mistral-7B-v0.3, a non-code-specialized model, yields 75.00, 69.51, 74.60, and 60.85.

• Template context matters for RMI, not for IFD. IFD correlates at ρ = 0.957 with its raw concatenation version, while RMI's correlation falls from 0.88 (simplified task prompt) to 0.79 (system prefix only), 0.70 (prefix-only) and 0.62 (pure raw cold start). The authors attribute this to the absence of context inflating PPL(Q) on short queries.

• Math transfer. On OpenR1-Math-220k (91K samples after filtering to a 16,384-token cutoff), QAQ at 25% data scores 91.6 on MATH-500 versus 90.6 for full data, and 39.9 on GPQA-Diamond versus 42.4 for full data, outperforming every other 25%-data baseline on both benchmarks (Random 87.8/33.3, IFD 87.2/37.4, SCAR 86.6/30.8, RDS+ 85.6/37.9).

• Second code dataset. On Magpie-Qwen2.5-Coder-Pro-300K (~120K samples after filtering to 2048 tokens), QAQ at 25% scores 71.95 HumanEval, 65.24 HumanEval+, 68.25 MBPP, and 56.35 MBPP+ against full-data values of 71.95, 66.46, 62.43, and 53.17. The paper notes SCAR leads on HumanEval but falls below full data on MBPP+ (49.74 vs. 53.17), which it describes as benchmark-specific overfitting.

Methodology in Plain English

The researchers start from a dataset of paired questions and answers and ask a model two things about each pair. First, how surprising is the question on its own, measured as its perplexity with a standard chat template. Second, how surprising is the question once the answer has been shown, framed as a reverse generation task where the model is told: given this answer, produce the most likely computer science question it responds to, or say "INVALID" if the answer is outside computer science. The difference between the log of those two perplexities is the Reverse Mutual Information — essentially how much the answer tells you about the question.

Because plain question difficulty confounds this score (simple questions naturally have low perplexity and therefore lower RMI), they sort samples into 10 bins by question perplexity and rank RMI only within each bin, normalizing the rank to the range 0 to 1. This "stratified RMI" is computed twice per sample: once with a strong model (DeepSeek-Coder-6.7B-Base) and once with a weak model (Qwen3-0.6B). The final score is the difference between the two ranks, Diff = r_s − r_w. Samples the strong model ranks highly but the weak model does not are kept, on the reasoning that the strong model vouches for validity while the weak model vouches for the sample being non-trivial to learn. The top 25% by Diff is the selected subset; on WarriorCoder this cutoff lands near Diff > 0.1.

Training uses LlamaFactory for 3 epochs with a cosine decay schedule and a warmup ratio of 0.2. Batch size and learning rate scale with subset size: batch size 512 and learning rate 1.2e-4 for full data, 256 and 0.8e-4 for 50% data, and 256 and 0.4e-4 for 25% data. Evaluation is greedy decoding on HumanEval, HumanEval+, MBPP, and MBPP+, plus MATH-500 and GPQA-Diamond (Pass@1) via LightEval for the math transfer experiment. Baselines include Random, IFD, RDS+ (hidden-state cosine similarity to 25 HumanEval and 25 MBPP seed examples), and SCAR (a trained style-consistency ranker scoring linguistic form and instructional surprisal). Code is released at https://github.com/XXSg559/QAQ.

Why This Matters

Impact on research. The paper argues that mutual-information-style quantities are not interchangeable across directions in practice, even though Shannon mutual information is symmetric, and ties this to the notion of predictive V-information for a bounded predictor. It also extends the use of model disagreement from token-level selection (as in Rho-1 and prior long-context work) to sample-level data selection, and offers a concrete diagnostic taxonomy of synthetic data defects: misalignment, keyword echo, paraphrase, and corruption.

Real-world applications.

• Curating large seedless synthetic corpora produced by LLM prompting pipelines before expensive fine-tuning runs.
• Reducing fine-tuning compute: selecting 25% of the data is presented as a way to reach comparable capability at a fraction of the cost.
• Quality control for domain-specific assistants, where off-topic queries and jargon-echo answers contaminate instruction data.
• Extending the same selection logic to math and other reasoning domains, where the paper reports the largest gaps between QAQ and naive reduction.

Industry relevance. Teams that generate instruction data at scale (tens or hundreds of thousands of pairs) face a selection problem that directly determines training cost and model quality. A one-time scoring pass with a strong and a weak model is cheap relative to full fine-tuning, and the paper's finding that the method works with a non-code-specialized partner model and with a partner that is actually stronger reduces the burden of choosing an exact model pair.

Future Directions

• Broaden dataset coverage. The main experiments use WarriorCoder, with validation on Magpie-Qwen2.5-Coder-Pro-300K and OpenR1-Math-220k. Both code datasets are seedless synthetic and may share noise patterns; the authors call for testing on seed-based corpora and general instruction-tuning datasets.
• Determine optimal model pairs. The disagreement signal depends on the pairing. Three pairings are tested, but the best pair for other domains or programming languages remains unexplored, and the key factor appears to be meaningful disagreement rather than a strict capacity gap.
• Strengthen statistical reporting. Results are single-run; the limitations section calls for multi-seed confidence intervals. Checkpoint selection also differs by domain: code results use the best checkpoint because training on noisy WarriorCoder is less stable across steps, while math results report the final training step.
• Characterize model-specific bias. The paper notes that RMI is sensitive to model capability (strong-weak rank correlation ρ ≈ 0.70) and refers to a discussion of model-specific bias in its appendix, leaving open how much of the signal is dataset-intrinsic versus scorer-intrinsic.

Target Audience

This paper suits machine-learning engineers and researchers who build or curate instruction-tuning datasets, particularly for code generation, and who need practical filtering criteria for synthetic data. It is also relevant to practitioners working on data-efficient fine-tuning who want to cut training cost without measurable capability loss, and to researchers studying perplexity-based or mutual-information-based data selection who want a concrete alternative to IFD-style scoring. Readers should be comfortable with perplexity, conditional probabilities, pass@1 evaluation, and fine-tuning workflows.

Authors’ abstract

Synthetic data has become essential for training code generation models, yet it introduces significant noise and hallucinations that are difficult to detect with current metrics. Existing data selection methods like Instruction-Following Difficulty (IFD) typically assess how hard a model generates an answer given a query ($A|Q$). However, this metric is ambiguous on noisy synthetic data, where low probability can distinguish between intrinsic task complexity and model-generated hallucinations. Here, we propose QAQ, a novel data selection framework that evaluates data quality from the reverse direction: how well can the answer predict the query ($Q|A$)? We define Reverse Mutual Information (RMI) to quantify the information gain about the query conditioned on the answer. Our analyses reveal that both extremes of RMI signal quality issues: low RMI indicates semantic misalignment, while excessively high RMI may contain defect patterns that LLMs easily recognize. Furthermore, we introduce a selection strategy based on the disagreement between strong and weak models to identify samples that are valid yet challenging. Experiments on the WarriorCoder dataset demonstrate that selecting just 25% of data using stratified RMI achieves comparable performance to full-data training, significantly outperforming existing data selection methods. Our approach highlights the importance of bidirectional semantic coherence in synthetic data curation, offering a scalable pathway to reduce computational costs without sacrificing model capability.