Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation

AI summary

Overview

Research area: Reinforcement learning with verifiable rewards (RLVR) for mathematical reasoning in large language models, spanning both policy-optimization algorithm design and training-data augmentation.

Technical level: Advanced. The paper proves two theorems about gradient update magnitudes and proposes a modified policy-gradient objective; the summary below translates those ideas into plain language, but the underlying analysis assumes familiarity with policy-gradient methods such as GRPO and PPO.

Scope in one sentence: The paper argues that existing RLVR pipelines systematically under-train on hard-but-solvable math questions, and addresses this with a difficulty-balanced advantage estimator plus a question-reformulation strategy that makes training questions harder while preserving their original answers.

What This Paper Is About

Reinforcement learning with verifiable rewards has become a standard way to sharpen mathematical reasoning in large models, but the most widely used algorithm, Group Relative Policy Optimization (GRPO), implicitly gives the largest policy updates to questions of moderate difficulty and suppresses updates on both very easy and very hard questions. Separately, existing data-augmentation methods for reasoning mostly rephrase questions to add diversity rather than to make them harder. The paper's goal is to fix both problems at once: an algorithm (DGPO) that equalizes update magnitude across questions and then upweights the hard ones, and a data strategy (MQR) that reformulates questions along three difficulty-increasing dimensions while keeping the original gold answer.

Key Contributions

1. Difficulty-Aware Group Policy Optimization (DGPO). A new RLVR algorithm that first corrects the implicit update-magnitude imbalance in GRPO via difficulty-balanced group advantage estimation (DGAE), which normalizes group-relative advantages by the mean absolute deviation (MAD) of rewards instead of the standard deviation.
2. Difficulty-aware question-level weighting (DQW). A second stage inside DGPO that assigns each valid question a weight based on its current difficulty, measured as the negative mean accuracy across its generated responses, so harder questions receive larger weights.
3. Multi-Aspect Question Reformulation (MQR). A data-augmentation strategy tailored to RLVR that rewrites each question along three aspects — adding story background, introducing abstract terminology, and nesting a key numerical condition as an independent sub-problem — under the hard constraint that all reformulations preserve the original gold answer, so no new solutions need to be generated.
4. Empirical validation across models and domains. MathForge (DGPO + MQR) is tested on Qwen2.5-Math-7B, Qwen2.5-Math-1.5B, Qwen2.5-3B, DeepSeek-Math-7B, and in the multimodal setting on Qwen2.5-VL-3B-Instruct, with DGPO also shown to be compatible with GPG, DAPO, and GSPO.

Main Findings

• GRPO has a provable difficulty imbalance. Theorem 1 shows that, for a single question with binary rewards, the total unclipped update magnitude under GRPO's group relative advantage estimation is exactly 2G√(p(1−p)), where p is the accuracy rate and G the group size. This peaks at p = 0.5 and decreases toward either 0 or 1 — so both easy and hard questions receive smaller updates than moderate ones.
• MAD normalization makes update magnitude constant. Theorem 2 shows that replacing the standard deviation with the mean absolute deviation makes the total update magnitude for a single question equal to G regardless of accuracy rate, and the result no longer requires binary rewards.
• DGPO alone improves over GRPO. On Qwen2.5-Math-7B trained on MATH, DGPO reaches an average of 39.79% across six benchmarks versus 37.61% for GRPO (+2.18), outperforming Dr.GRPO (37.40), GPG (37.93), DAPO (37.94), GSPO (37.71), and GRPO-AD (38.26).
• MQR alone also improves over GRPO. Trained with GRPO on MQR-augmented data, the model reaches 41.04% average (+3.43 over GRPO), with notable gains on MATH500 (77.85 vs. 72.20) and Olympiad (40.81 vs. 37.33).
• The combination is best. MathForge achieves 42.17% average (+4.56 over GRPO), the top score in Table 1, with MATH500 at 79.95 and Olympiad at 42.67.
• Results hold across model sizes and types. On Qwen2.5-Math-1.5B, MathForge scores 33.84 (+4.45 over GRPO); on Qwen2.5-3B, 29.01 (+3.54); on DeepSeek-Math-7B, 17.77 (+2.86). DGPO and MQR each outperform GRPO individually on all four models.
• Both DGPO components pull weight. Ablations on Qwen2.5-Math-7B show valid token-level loss averaging contributes +0.10, DGAE contributes +0.94, and DQW contributes +1.14 in average accuracy; the full DGPO reaches +2.18.
• Temperature T = 2.0 is optimal for DQW. T = 1.0 gives +1.42, T = 5.0 gives +1.92, and T = 10.0 gives +1.66, all below T = 2.0's +2.18. Because the difficulty score is bounded in (−1, 0), T = 2.0 keeps the ratio between maximum and minimum weights in a batch below e^(1/2) ≈ 1.65.
• DGPO generalizes to other algorithms. Adding DGPO to GPG raises the average from 37.93 to 38.92, to DAPO from 37.94 to 39.91 (higher than standalone DGPO's 39.79), and to GSPO from 37.71 to 39.32.
• DGPO also works in the multimodal domain. On GEOQA-8k with Qwen2.5-VL-3B-Instruct, DGPO reaches 59.95% versus 57.43% for GRPO (+2.52), beating Dr.GRPO (57.96), GPG (59.02), DAPO (59.02), GSPO (57.16), and GRPO-AD (58.09).
• MQR's gains come from quality, not volume. With training data volume normalized across methods, MQR-augmented data still beats the original data using DGPO on every model tested: Qwen2.5-Math-7B 42.17 vs. 39.90, Qwen2.5-Math-1.5B 33.84 vs. 32.18, Qwen2.5-3B 29.01 vs. 27.22, and DeepSeek-Math-7B 17.77 vs. 16.64.
• Reformulated questions really are harder. Evaluating Qwen2.5-Math-7B-Instruct on the MQR subsets gives accuracy of 79.77% on Original, 77.31% on Background, 76.87% on Term, and 72.04% on Sub-Problem.
• Each MQR aspect helps on its own. Against Original at 39.90 with DGPO, Original + Background reaches 40.95 (+1.05), Original + Term 41.24 (+1.34), and Original + Sub-Problem 41.53 (+1.63). Combining all three (MQR) reaches 42.17 (+2.27). MetaMath-Rephrasing, a GPT-3.5-Turbo rephrasing baseline using 22.5k sampled questions from its 50k set, reaches 40.73 (+0.83).
• "Train harder, test better." Training accuracy on the MQR-augmented data is consistently lower than on the original data, yet the MQR-trained model ends up with higher accuracy on the unseen MATH500 benchmark.
• DGPO produces shorter outputs. On MATH500, DGPO tends to generate more concise responses than GRPO while also being more accurate.

Methodology in Plain English

The authors start from an observation about how GRPO computes its learning signal. For each question, GRPO samples a group of answers, scores them with a simple rule-based correctness check (1 for correct, 0 otherwise), and then rescales each answer's reward by subtracting the group mean and dividing by the group's standard deviation. Because of that division, the total amount the model moves on a question depends on how many of its answers were correct: the biggest pushes happen when roughly half the group is right.

The authors prove this algebraically, then change the denominator. Instead of dividing by the standard deviation, DGPO divides by the mean absolute deviation of the group's rewards. The proof shows this makes the total update magnitude constant — exactly the group size G — no matter how many answers were correct. This means a question the model gets right once out of eight now receives the same "pressure" as a question it gets right four times out of eight.

Once magnitudes are balanced, the second step adds back a deliberate preference for difficult questions. Each question in the batch gets a weight computed from its difficulty (the negative of its mean accuracy across the sampled responses), passed through a softmax-like formula with a temperature parameter T. Valid questions — those whose answers are not all correct or all incorrect — are the only ones counted in the loss. The authors call this a "balance-then-reweight" procedure and note it has fewer hyperparameters than the prior difficulty-reweighting method they compare against.

On the data side, MQR takes each original training question and produces three harder variants using a large reasoning model (OpenAI o3 by default): one wrapped in an unrelated story background that resembles mathematical context but is not; one restating the question using a newly invented abstract term as a central concept; and one converting a key numerical condition into a standalone sub-problem that may come from any branch of mathematics. The constraint is that every variant must still have the same gold answer, so no new solutions are generated and the original mathematical logic is preserved. These three variants target, respectively, the model's ability to find key information amid noise, to handle abstraction, and to chain multi-step cross-domain reasoning. Because the augmented data is harder, it pairs naturally with DGPO, which prioritizes hard questions.

Experiments use 8 NVIDIA H20 GPUs and the Open-R1 codebase. The main setup trains Qwen2.5-Math-7B on the MATH dataset and evaluates zero-shot on AIME24, AIME25, AMC23, MATH500, Minerva, and Olympiad, with 32 runs for AIME24, AIME25, and AMC-23 and 4 runs for the others. For a fair algorithm-level comparison, resampling components in GPG and DAPO are disabled, and GRPO-AD applies the advantage reweighting of Zhang and Zuo (2025) on top of the GRPO baseline. DeepSeek-Math-7B receives a cold start via fine-tuning on 80k samples from NuminaMath-CoT. The temperature T in DQW is set to 2.0; remaining implementation details are placed in the paper's appendices, which are not included in the content summarized here.

Why This Matters

Impact on research. The paper reframes a widely used algorithm not as a neutral baseline but as one with a provable, quantifiable bias toward medium-difficulty problems. The closed-form update-magnitude results give other researchers a concrete quantity to reason about when designing advantage estimators, and the demonstration that DGPO stacks with GPG, DAPO, and GSPO suggests difficulty balancing is an orthogonal axis rather than a competing algorithm. The MQR result also argues that rephrasing for difficulty — not just diversity — is an underexplored lever in RLVR data curation, and it does so without requiring regenerated solutions.

Real-world applications:

• Automated math tutoring and homework help. A model that handles deliberately harder, noisier, and more abstractly worded problems is more useful to students encountering realistic textbook or competition phrasing.
• Educational assessment generation. The MQR recipe — add irrelevant background, invent a term, nest a sub-problem, keep the answer fixed — is a template for producing harder practice variants from an existing problem bank without writing new answer keys.
• Competition and olympiad training tools. The largest gains in the reported tables appear on Olympiad and MATH500, the benchmarks closest to competition-style reasoning, which is directly relevant to contest preparation platforms.
• Multimodal math education. The GEOQA-8k result shows the approach transfers to diagram-based geometry questions, relevant to tools that read figures as well as text.

Industry relevance. The method is a training-time modification to an existing RLVR loop rather than a new architecture, so it can be adopted on top of current post-training stacks. OpenAI o3 is used only as the data reformulator, not in the trained model, and the paper notes that some smaller open-source models can handle the reformulation task, which lowers the cost barrier for teams without frontier-model access. The data augmentation cost is reported in the paper's Appendix E, which is outside the content summarized here.

Future Directions

• Extending difficulty balancing beyond verifiable rewards. Theorem 2 drops the binary-reward assumption, which suggests the MAD-based normalization could apply to settings with graded or continuous rewards — an avenue the paper raises but does not test.
• Reducing reliance on a strong reformulator. The paper states that smaller open-source models "can also competently handle this task" but the specific evaluation of that claim appears in a section on generality to less capable reformulators that is truncated in the provided content, leaving the exact results unreported here.
• Characterizing when harder data stops helping. The temperature analysis hints at a ceiling — too sharp a weighting focus on the single hardest question degrades performance — but the paper does not explore how far the MQR difficulty increase can be pushed before the answers become effectively unreachable for the policy.
• Broader domain transfer. The multimodal test covers a single dataset (GEOQA-8k) and one model (Qwen2.5-VL-3B-Instruct); whether the same difficulty machinery helps in other visual-reasoning or code-generation settings is left open.

Target Audience

Researchers and engineers working on post-training and reinforcement learning for reasoning models — particularly those already using GRPO, DAPO, GSPO, or GPG and looking for a drop-in improvement. It also suits data-curation teams interested in augmentation strategies for RLVR, and readers who want a mathematically grounded critique of a standard algorithm rather than an empirical comparison alone. Readers unfamiliar with policy-gradient objectives will find the theorems dense, though the two-part intuition (equalize update size, then weight by difficulty) is easy to grasp.

Code and augmented data are available at https://github.com/AMAP-ML/MathForge

Authors’ abstract

Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting. Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks. The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.