Dynamic Important Example Mining for Reinforcement Finetuning

AI summary

Overview

Research area: Data-centric reinforcement fine-tuning (RFT) for large language models and vision–language models — specifically, how training examples should be selected and weighted during policy optimization.

Technical level: Advanced. The paper assumes familiarity with policy gradient methods, PPO/GRPO, advantage estimation, and gradient-based influence functions.

Scope: The paper proposes DIEM, a fully automated framework that measures each sample's marginal contribution to the current policy update via gradient alignment and reweights the batch under a gradient-magnitude constraint, evaluated on mathematical and multimodal reasoning benchmarks.

What This Paper Is About

Reinforcement fine-tuning improves a model's reasoning by learning from reward signals, but its success depends heavily on how training data are chosen and weighted. Most existing data-centric RFT methods decide a sample's value once, before or early in training, using heuristics such as reward variance, pass rate, or difficulty score. This paper argues that a sample's usefulness changes as the policy evolves, and it introduces DIEM, which re-estimates and reweights each sample's contribution at every optimization step inside the training loop.

Key Contributions

1. A fully automated dynamic data selection framework for RFT. DIEM estimates importance and reweights data at every step, with the paper reporting a minimal additional training overhead of 1.2% (and, in the speed test, 0.9 hours on top of a 70.3-hour baseline, about 1.28%).
2. A gradient-alignment importance estimator. Instead of heuristic scores, DIEM scores a sample by the inner product between its individual policy gradient and the aggregate batch gradient, which the paper proves is a bounded-error proxy for the true step-wise importance.
3. A constrained batch-reweighting scheme with a quasi-closed-form solution. The method maximizes aggregate weighted utility subject to the constraint that the reweighted gradient's L2 norm equals the original unweighted gradient's norm, with negative weights clipped to zero.
4. Demonstration that reweighting induces a self-organizing curriculum. Tracking Pass@k-based difficulty groups over training shows easy and medium samples initially receive high weight, easy-sample weight declines, and hard-sample weight rises with oscillations — an emergent easy-to-hard progression.

Main Findings

• Consistent gains over GRPO on LLM math benchmarks. On the average of MATH-500, Gaokao23en, AMC-23, AIME24 and AIME25, DIEM outperforms GRPO by +1.74 on Qwen-3-1.7B (31.36 to 33.10), +2.50 on Qwen2.5-3B (27.82 to 30.32), +3.36 on Qwen3-4B (37.30 to 40.66), and +1.68 on Qwen2.5-7B (34.00 to 35.68).
• Largest gains on the hardest benchmarks. On AIME25, DIEM improves over GRPO by +2.1 on Qwen-3-1.7B (3.4 to 5.5) and +5.3 on Qwen2.5-7B (5.5 to 10.8), which the paper reports as relative gains of 62% and 96%.
• One reported exception. On AMC-23 with Qwen3-4B, DIEM scores 55.0 versus GRPO's 58.5; the paper calls this an outlier specific to that model–benchmark pair, noting DIEM improves AMC-23 on all other models.
• Strong VLM results. On Qwen2.5-VL-7B, DIEM reaches a 61.8% average across MathVista, MathVerse, MathVision, MMStar, MMMU and AI2D, versus 59.1% for Vanilla RFT and 60.0% for the best prior data-selection method, SPEED-RL; the paper reports these as gains of 3.6 and 1.8 percentage points. DIEM's 7B average of 61.8% also surpasses the commercial model GPT-4o's 60.9% average.
• Scaling to larger VLMs preserves the advantage. On Qwen2.5-VL-32B, DIEM achieves a 67.3% average, ahead of Vanilla RFT (64.9%) by 2.4 percentage points and SPEED-RL (65.6%) by 1.7 percentage points, and it posts the highest score on all six individual benchmarks.
• Both components are necessary (ablation on MathVerse with Qwen-2.5-VL-32B). Starting from DIEM's full score of 58.0, replacing the influence score with a random value drops to 53.0 (5.0 points), with raw Pass@k to 53.2 (4.8 points), with Pass@k-to-median-distance to 54.9, with raw PCL Difficulty score to 52.1 (5.9 points), and with Difficulty-to-median-distance to 53.6. Removing the reweighting step (NULL-operation) drops to 55.4 (2.6 points) and replacing it with Softmax normalization drops to 56.4 (1.6 points).
• Small measured overhead versus baselines. The baseline takes 70.3 hours and DIEM 71.2 hours, an increase of 0.9 hours. By comparison, the paper reports PCL at 79.1 hours, SPEED-RL at 94.6 hours, and both LIMR and HVS at 122.0 hours.

Methodology in Plain English

The method runs inside a standard policy-gradient RFT loop (the paper builds on PPO and GRPO) and adds two lightweight steps per minibatch.

Step 1 — measuring importance. The ideal measure of a sample's value is how much the batch's total reward changes if you remove that sample from the update and retrain. Computing that directly would require one full gradient update per sample, which is far too expensive. DIEM instead reuses gradients already computed during backpropagation: it takes each sample's individual policy gradient and computes its inner product with the summed batch gradient. A sample pointing in the same direction as the batch gets a high positive score; a sample pointing away gets a negative score. The paper proves this proxy's error is bounded by a term proportional to the learning rate — a bound that does not require convexity or near-stationarity assumptions, which matters because RFT training is non-convex and non-stationary.

Step 2 — reweighting. Given importance scores, DIEM solves for a weight per sample that maximizes the weighted total importance, subject to the constraint that the reweighted gradient has the same L2 norm as the unweighted gradient. That constraint keeps the update step size stable. Using Lagrange multipliers, the solution only requires inverting a Gram matrix of the per-sample gradients, whose size equals the minibatch size — small enough that the paper describes the cost as trivial next to a full RFT step. Negative weights are clipped to zero, and the resulting weighted gradient drives the parameter update.

Experiments. LLM experiments used the veRL framework with Qwen series models on 16 NVIDIA H200 GPUs, trained on 14,973 math problems (7,500 from MATH and 7,473 from dapo-math), with a prompt batch size of 64, 8 rollouts per prompt, mini-batch size 32, micro-batch size 8 (4 for 7B/8B models), maximum prompt and response lengths of 1,024 and 2,048 tokens, constant learning rate 1e-6 with no warmup, and KL penalty and entropy bonus disabled with clipping ratio 0.2. VLM experiments used Qwen2.5-VL-7B and 32B on 52K multimodal entries sampled from MM-Eureka, with the 7B trained on 16 NVIDIA A100 GPUs and the 32B on 32 GPUs, a global batch size of 128 and learning rate 1×10⁻⁶. Baselines included GRPO, HVS, LIMR, PCL, SPEED-RL, and random selection.

Why This Matters

Impact on research. The work challenges the default assumption in data-centric RFT that a sample's value is fixed, and moves importance estimation from a preprocessing heuristic to a quantity computed inside the policy optimization loop. Its gradient-alignment estimator is also notable for not requiring convexity or near-stationary assumptions, unlike many influence-function approaches.

Real-world applications:

• Training reasoning models for mathematics and competition problems, where gains concentrate on the highest-difficulty benchmarks (AIME25).
• Post-training multimodal assistants for diagram, chart, and textbook-image reasoning (MathVista, MathVerse, MathVision, MMStar, MMMU, AI2D).
• Cost-constrained training pipelines, since the method reuses existing gradients rather than training a separate surrogate model.
• Domains with verifiable rewards where correctness signals are cheap but data value shifts during training.

Industry relevance. The approach is presented as compatible with existing RFT algorithms (GRPO, PPO, and others), which lowers adoption cost, and the reported overhead is small relative to the training run itself. The comparison against LIMR and HVS at 122.0 hours versus DIEM's 71.2 hours (against a 70.3-hour baseline) is a direct argument for compute efficiency at scale.

Future Directions

• Extending beyond the evaluated settings. The paper's experiments cover Qwen-family LLM math benchmarks and Qwen2.5-VL multimodal benchmarks; whether the gains transfer to other model families, modalities, or reward types is not reported.
• Reducing the oscillation in hard-sample weighting. The training visualization shows the hard-group weight rising but with oscillations; understanding and damping these dynamics is an open question.
• Addressing estimation error and negative weights. The method clips negative weights to zero, which the paper says may reflect either genuine harm or misclassification from estimation error; sharper estimation could reduce that ambiguity.
• Combining with other RFT refinements. The paper notes broad compatibility with PPO, GRPO, DAPO, GPG and other schemes but does not report systematic combinations with all of them.
• More extensive ablations. The paper states that additional ablation results on more benchmarks appear in supplementary materials, implying the current ablation is limited to MathVerse with Qwen-2.5-VL-32B.

Target Audience

Researchers and engineers working on reinforcement fine-tuning, RLHF/RLVR pipelines, and post-training of large language and vision–language models, particularly those interested in data curation, curriculum learning, or gradient-based influence estimation. Readers need a working knowledge of policy gradient methods and PPO/GRPO to follow the derivations, but the core intuition — score samples by how well their gradients align with the batch, then reweight without changing update magnitude — is accessible to practitioners with a general machine learning background.

Authors’ abstract

Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.