DataFlex-RL: An Evaluation Platform for RLVR Data Policies

AI summary

Overview

Research area: Machine learning / post-training of large language models, specifically reinforcement learning with verifiable rewards (RLVR) and data-centric training policy design.

Technical level: Intermediate. The paper is an empirical evaluation study; understanding it requires familiarity with GRPO-style RL fine-tuning, advantage weighting, and benchmark aggregation, but its findings are stated in terms of paired confidence intervals and mean scores.

Scope (one sentence): The paper builds a controlled platform that holds a GRPO recipe fixed and asks whether 13 selection, reweighting, and mixture-adaptation data policies produce reproducible gains over uniform sampling across Qwen and Llama base models and 12 math, logic, and science benchmarks.

What This Paper Is About

RLVR data policies decide which generated rollouts enter an update, how strongly they are weighted, or which domains supply the next batch of prompts, and many such methods have been proposed (solve-rate filtering, variance-based down-sampling, advantage reweighting, adaptive domain curricula). The problem is that these methods are usually introduced as one component of a larger RLVR recipe, so a reported gain may come from the signal, the intervention, or the surrounding training stack rather than the policy itself. DataFlex-RL addresses this by isolating the data policy from the rest of the GRPO pipeline and comparing all policies under matched models, data, rollout budgets, seeds, and a shared 12-benchmark evaluation.

Key Contributions

1. A controlled comparison finding no clear advantage over uniform sampling. The authors evaluate 13 selection, reweighting, and mixture configurations with 12 matched seeds on Qwen2.5-7B-base. Uniform GRPO improves Overall accuracy by 7.76 points, but no selection or reweighting method shows a clear improvement over uniform sampling, and no adaptive mixture improves over a fixed equal mixture at the measured precision.

2. A direct measurement of evaluation sensitivity. On nine Qwen2.5-7B-Instruct configurations, a math-heavy six-benchmark summary (five math sets plus GPQA-Diamond, with no logic benchmark) and the domain-balanced 12-benchmark summary produce negatively correlated rankings (ρ = -0.33), with method spread changing from 0.90 under Math-Heavy-6 to 3.31 under DB-12.

3. A platform for fair comparisons. DataFlex-RL isolates the data policy from the surrounding GRPO recipe through a shared rollout, verification, optimization, and evaluation stack, matched seeds, and a common 12-benchmark score format, so future policies can be compared under the same protocol without rebuilding the pipeline.

4. A released experimental matrix with auditable records. The matrix contains 591 runs, including a 156-run primary block comparing all 13 Qwen2.5-7B-base configurations over 12 seeds, with runs linked to configurations, training logs, and 12-benchmark records under an open licence.

Main Findings

• Training headroom is large. Uniform GRPO improves the domain-balanced Overall score from 42.01 to 49.77 on Qwen2.5-7B-base (+7.76, 95% CI [7.28, 8.25]) and from 10.87 to 21.12 on Llama-3.1-8B-base (+10.25, 95% CI [7.75, 12.33]), so the null data-policy result is not explained by the model failing to learn.

• No selection or reweighting method beats uniform sampling. Across 12 matched seeds, paired differences relative to uniform sampling were +0.08 [-0.67, 0.82] for difffilter, -0.58 [-1.39, 0.23] for maxvar, -0.90 [-1.87, 0.08] for gfpo, -0.38 [-1.14, 0.38] for topk, -0.28 [-1.15, 0.60] for ar, -0.86 [-3.28, 1.57] for per, -0.23 [-1.44, 0.98] for softmax, and -0.02 [-0.61, 0.57] for diffband. None of the eight intervals excludes zero.

• The spread among policies is small. Including the baseline, the nine policy means span 0.97 point, about one eighth of the 7.76-point gain from uniform GRPO. The four reweighting estimates range from -0.86 to -0.02.

• Selector ordering is unstable across seed counts. topk leads in the original three-seed subset, whereas difffilter has the highest 12-seed mean and topk falls below the baseline.

• Adaptive mixtures change the data but not the outcome reproducibly. Relative to the fixed equal static mixture (49.00), the paired differences are +0.45 [-0.09, 1.00] for reward_gap, +0.61 [-0.02, 1.25] for dump_ucb, and +0.16 [-0.61, 0.93] for tscl. Every interval includes zero and the means span only 0.61 point, even though training logs confirm the methods changed realized domain proportions.

• The corrected Llama evaluation finds no common winner. The three original selectors have paired Overall differences of -1.09 [-2.84, 0.65] for difffilter, -0.71 [-2.18, 0.76] for maxvar, and -0.26 [-1.76, 1.24] for topk, all including zero. Across the expanded set, means range from 18.66 (softmax) to 21.98 (diffband) against a reusable baseline of 21.12. Earlier 14–16 point Llama values were produced by an evaluator that silently scored unsupported math tasks as zero and are excluded.

• Cross-scale directions are inconsistent. In the six-configuration sweep, difffilter is below the uniform baseline at Qwen2.5-1.5B-base (-0.83) and Qwen2.5-3B-base (-0.79) but above it at Qwen2.5-7B-base (+0.08) and Qwen2.5-14B-base (+1.20); maxvar is below the baseline in every row; tscl is below its fixed control at 1.5B (-0.10) but above it at larger Qwen sizes and on Llama-3.2-3B-base (+0.22).

• Benchmark coverage can flip the apparent winner. Math-Heavy-6 ranks topk and diffband first, whereas DB-12 ranks gfpo and difffilter first. The reason is domain trade-offs: on Qwen2.5-7B-Instruct, diffband has the highest LOGIC mean (59.0) and the lowest SCIENCE mean (44.2). By contrast, Macro-12 and Item-12 both correlate with DB-12 at ρ = 0.88, and leave-one-out checks range from 0.78 to 0.98, so the disagreement is driven mainly by the omitted logic domain.

• The conclusion is explicitly scoped. The authors state the result does not establish equivalence, nor rule out gains under longer training, other hyperparameters, or different policy families.

Methodology in Plain English

The authors build one shared driver for rollout, verification, and GRPO optimization, and register each method at one of three intervention points so that only the data policy changes:

• Selection decides after rollout which generated responses enter the update, using a 0/1 mask that can set a response's contribution to zero. Methods: difffilter (retain a group only if its solve rate falls strictly between 0.2 and 0.8, a stricter DAPO-style filter), maxvar (retain half the responses in a group so the subset maximizes reward variance, following PODS), gfpo (retain the top three of five responses by reward per token), and topk (retain the highest-scoring 50% of responses in the batch by mean absolute advantage).

• Reweighting keeps responses but changes their continuous loss contribution, with weights normalized to mean one. Methods: ar (weights proportional to 0.5·token probability + 0.5, damping low-probability tokens), per (weights proportional to (a_i + ε)^0.5, PER-inspired weighting of the current batch rather than replay), softmax (weights proportional to exp(a_i/T) with T = 1), and diffband (2× weight to responses between the batch reward quartiles, 1× otherwise).

• Mixture adaptation changes which domains supply future prompts, acting before the next rollout. Methods: static (p_t(d) = 1/3, the fixed control), reward_gap (favoring lagging domains by rolling mean reward, with a 0.05 floor), dump_ucb (an upper-confidence-bound score over mean absolute advantage and domain counts), and tscl (favoring domains with greater learning progress or forgetting, using reward slope). Mixture probabilities use T = 1 and a 0.05 floor, summarizing each domain over a rolling 50-observation window.

A point the authors emphasize is that the signal and the intervention are separate: per, softmax, and topk all rank responses by mean absolute advantage a_gk, but the first two turn that score into a continuous weight while topk uses it as a hard keep-or-discard rule. Comparing them tests whether the same signal is better used for reweighting or for selection. Selection and reweighting occur after rollout, so they do not reduce generation cost; selection is not renormalized and can reduce the number of tokens used in the update, whereas reweighting preserves a mean weight of one.

All campaigns use verl v0.5+ and GRPO with five rollouts per prompt, a KL coefficient of 10^-3, prompt and response limits of 1024 and 8192 tokens, and 300 optimizer steps with checkpoints every 100 steps unless a campaign explicitly studies 1000-step training. The training corpus contains 15,000 prompts split equally among math (math_dapo, DeepScaler, and GSM8K with boxed-answer verification), logic (procedurally generated Knights&Knaves puzzles with an assignment checker), and science (SciQ multiple-choice with exact letter matching). The breadth grid uses seeds {1, 2, 3}, while the Qwen-base primary matrix and higher-seed replications use seeds {1, …, 12}.

Evaluation covers 12 benchmarks: five math sets (MATH-500, AIME-2024, OlympiadBench, MinervaMath, GSM8K), four logic sets (Knights&Knaves, two BBH tasks, and ZebraLogic), and three science sets (MMLU-Pro Chemistry, MMLU-Pro Physics, GPQA-Diamond). Decoding is deterministic; math and GPQA allow up to 8192 output tokens, and the shared logic and MMLU-Pro evaluator uses 4096. The Overall score averages within MATH, LOGIC, and SCIENCE first, then averages the three domain scores so each domain receives equal weight. Two safeguards address training–evaluation mismatch: a calibration smoke test checking boxed-answer and multiple-choice parsing on a reference checkpoint before each campaign, and an audit of the 15,000 training prompts against all evaluation items that finds no exact or normalized matches and no 13-gram Jaccard similarity above 0.5.

Why This Matters

The paper's central methodological contribution is a demonstration that a widely used class of RLVR data-policy interventions does not replicate in a controlled setting, and that the choice of evaluation summary can itself determine which method looks best. It also documents a concrete evaluation failure mode: an evaluator that silently scored unsupported math tasks as zero produced Llama values of roughly 14–16 points that had to be excluded from the corrected results.

Real-world applications implied by the work's setting (the paper does not report deployed systems):

• Reasoning-model post-training pipelines. Teams deciding whether to add solve-rate filtering, variance-based selection, or adaptive domain curricula now have paired-interval evidence under a fixed GRPO recipe rather than headline numbers from differently tuned stacks.
• Benchmark and evaluation design. The finding that omitting the logic domain changes the apparent winner, while retaining all 12 benchmarks yields correlated rankings, is directly relevant to anyone constructing or auditing reasoning benchmark suites.
• Reproducibility auditing of RL results. The released run-level records, logs, configurations, and reconstruction scripts provide a template for linking reported means back to matched seeds and a common score format.
• Domain-focused reasoning products in math, logic, and science. Because the platform separates MATH, LOGIC, and SCIENCE scores, it shows that methods trade off across these domains rather than improving uniformly, which matters for applications targeting one domain in particular.

Industry relevance: the negative result has direct cost implications, since selection and reweighting methods do not reduce generation cost (they act after rollout), and selection can reduce the number of tokens used in the update while reweighting preserves a mean weight of one. For organizations choosing where to spend rollout compute, the paper argues that effort spent on data-policy engineering may not outperform simply running uniform GRPO at the same budget.

Future Directions

• Test under conditions the paper explicitly leaves open. The authors state their result does not rule out gains under longer training (their campaigns use 300 optimizer steps unless a campaign studies 1000-step training), other hyperparameters, or different policy families.
• Resolve the logic-domain sensitivity. Since the ranking disagreement between Math-Heavy-6 and DB-12 is driven mainly by omitting logic, the relationship between domain coverage and reported method rankings needs systematic mapping.
• Determine whether any signal is useful when decoupled from the intervention. per, softmax, and topk share the mean absolute advantage signal but differ in how it is applied; the paper's comparison shows no benefit for either use at the measured precision, leaving open whether other signal–intervention pairings would.
• Establish the boundary between "no reproducible gain" and equivalence. The paper is deliberately scoped to say it does not establish equivalence, so the natural follow-up is a precision or power analysis that would distinguish these two possibilities.

Target Audience

This paper is most useful to researchers and engineers who post-train reasoning models with RLVR and are deciding whether to adopt a data policy; to benchmark designers and reproducibility-focused methodologists interested in how aggregation choices change conclusions; and to reviewers and practitioners who need paired-interval evidence rather than single-run comparisons. Readers unfamiliar with GRPO can follow the experimental logic and results, but the method taxonomy in Table 1 assumes comfort with advantage weighting, token-level loss terms, and group-relative baselines.

Authors’ abstract

Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which domains contribute to subsequent training batches. We introduce DataFlex-RL, an evaluation platform for comparing these choices under a common GRPO recipe. Our primary experiment evaluates 13 configurations across 12 matched seeds using Qwen2.5-7B-Base and 12 mathematics, logic, and science benchmarks. Uniform GRPO improves the domain-balanced average accuracy by 7.76 percentage points over the untrained checkpoint. None of the eight rollout-selection or reweighting methods achieves a paired 95% confidence interval that excludes zero relative to uniform sampling, and none of the three adaptive mixtures outperforms a fixed equal mixture at the same level of precision. A corrected 12-seed extension on Llama-3.1-8B-Base places the additional methods on the same score scale as the original controls, but does not reveal a consistent winner in terms of observed mean performance. We also quantify evaluation sensitivity by rescoring nine Qwen2.5-7B-Instruct runs using a math-heavy six-benchmark summary, consisting of five mathematics benchmarks and GPQA-Diamond but no logic benchmark, and comparing it with the domain-balanced 12-benchmark summary. The resulting rankings are negatively correlated, with a correlation coefficient of -0.33, whereas summaries that retain all 12 benchmarks largely agree. Across the controlled settings studied here, changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.