FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning

AI summary

Overview

• Research area: Reinforcement learning for large language models, specifically reinforcement learning with verifiable rewards (RLVR), reward modeling, and reasoning reliability.
• Technical level: Advanced. The paper involves GRPO advantage estimation, a theoretical analysis of group-relative advantage shifts, and a step-wise reinforcement learning procedure for training a reward model.
• Scope: The paper studies "flawed-positive" rollouts (correct final answers reached through unreliable reasoning), proposes a reward-penalization algorithm called FAPO, and trains a compact generative reward model to detect flawed positives during RL training.

What This Paper Is About

RLVR typically rewards a model with a binary signal for whether its final answer is correct. This means a rollout that guesses an answer or skips reasoning steps ("flawed positive") receives exactly the same reward as a fully correct solution, so the policy can internalize unreliable reasoning. The paper first analyzes how these flawed positives behave across the RL process, then proposes FAPO (Flawed-Aware Policy Optimization), which penalizes flawed positives with a parameter-free reward term and uses a small generative reward model with process-level rewards to detect them.

Key Contributions

1. A systematic study of flawed-positive rollouts in RL. The authors measure flawed-positive prevalence in three models, run a simulated learning-stage experiment, and track flawed-positive ratio through RL training on Qwen2.5-Math-7B, identifying a twofold effect: flawed positives accelerate early capability gains but later constrain reasoning quality.
2. FAPO, a flawed-aware reward penalization algorithm. FAPO adds a penalty term R_Δ = -λ to rollouts that are both correct and flagged as flawed positives, with a majority-guided strategy that sets λ = 1. The penalty is described as parameter-free, and can serve as a drop-in replacement for rule-based outcome rewards in other RLVR methods.
3. FAPO-GenRM, a compact generative reward model with process-level rewards. Built on Qwen3-4B-Instruct and trained with a combination of outcome reward and a distance-sensitive step-wise process reward, it detects and localizes errors. The training data, FAPO-Critic-85K, is derived from DAPO-Math-17K using Qwen3-32B for step-level error localization.
4. A detection benchmark and an infrastructure design. The paper introduces FlawedPositiveBench, built from positive samples in ProcessBench, with precision, recall, and F1 metrics; and describes an asynchronous architecture that decouples rollout inference from generative reward modeling, limiting the training-time overhead to less than 20% relative to the baseline.

Main Findings

• Flawed positives are common at the starting point. Evaluating Qwen2.5-Math-7B-Base, Llama3.3-70B-Instruct, and Qwen3-1.7B on the DAPO-Math dataset with Qwen3-32B as judge, flawed positives account for 20%–40% of correct rollouts.
• Flawed positives concentrate in early learning stages. In a simulated experiment where samples are grouped by rollout accuracy, flawed positives are most prevalent in the early learning stages and diminish significantly as training progresses, supporting their role as stepping stones.
• The flawed-positive ratio stays roughly flat during training. Training Qwen2.5-Math-7B with RL on DAPO-Math, rollout accuracy improves steadily while the flawed-positive ratio remains almost constant at around 30%, indicating that optimization does not naturally shift toward reliable reasoning.
• Penalizing flawed positives helps but is slow early on. A preliminary trial using Qwen3-32B to detect flawed positives and assign them the same negative signal as incorrect rollouts produced significant gains on AIME24 compared with baseline RLVR, though improvements emerged more gradually in the early stages.
• Existing detectors are misaligned on capability versus efficiency. Models such as Qwen3-4B-Instruct and Qwen2.5-Math-PRM-72B show an "over-critic" pattern: high recall but low precision, often flagging minor issues such as unsimplified fractions. Stronger models are more accurate but impractical for online RL because of slow inference.
• FAPO-GenRM-4B beats much larger models and its teacher. Built on Qwen3-4B-Instruct, it achieves substantial improvements on both FlawedPositiveBench and ProcessBench, even outperforming the teacher model Qwen3-32B.
• FAPO improves reasoning across benchmarks. FAPO maintains a clear advantage over baselines on AIME24, AIME25, and GPQA-Diamond, with advantages at nearly all intermediate evaluation checkpoints. Specific numeric benchmark scores are not reported in the provided content.
• Process reliability improves. FAPO responses exhibit a substantially lower flawed-positive ratio; the paper also reports a manual verification of unreliable reasoning patterns in Table 6.
• Training is more stable and no longer requires extra tokens. Learning curves are smoother, FAPO does not show the notable late-stage performance drop seen in baselines, and gains do not come from longer responses.
• Detection quality translates into RL gains. Comparing FAPO-GenRM against the base Qwen3-4B-Instruct during RL training shows that stronger detection capability leads to improved final performance.
• Self-correction behavior is gradually replaced. Both FAPO and the baseline initially depend on self-correction (longer rollouts), but FAPO shifts toward fully correct rollouts over time, producing shorter rollouts and consistent performance gains.
• Dense process-reward designs can be hacked. A step-ratio reward (scoring by the proportion of correct steps before the first error) gives some early gains but then stalls, producing an obvious jump-in-reasoning pattern.

Methodology in Plain English

The authors start from GRPO, a policy-gradient method that samples a group of rollouts per question and normalizes their rewards into per-token advantages. In standard RLVR, the reward is +1 for a correct final answer and -1 otherwise, so flawed positives are indistinguishable from fully correct solutions.

To study the problem, they (1) measure how often flawed positives appear at the start of training, (2) group samples by rollout accuracy to approximate learning stages, and (3) actually run RL training and track the flawed-positive ratio over time.

For detection, rather than calling a large model like Qwen3-32B online (too slow), they train a small generative reward model with RL. Its reward has two parts: an outcome reward of +1/-1 for whether the predicted label matches ground truth, plus a process reward that applies only when both prediction and label are "flawed positive." That process reward is -|t̂θ - t*| / n, where t̂θ is the predicted error step index, t* the true first-error step index, and n the number of steps, keeping the term in [-1, 0]. This rewards predictions close to the true error location and penalizes distant ones, discouraging mere guessing: early in training the outcome term dominates, and as correctness saturates the process term becomes more influential without any extra control hyperparameter.

For the final RL stage, FAPO adds a penalty to the reward: correct rollouts flagged as flawed positives get -λ added; all other rollouts are unchanged. A theoretical analysis shows the optimization shifts from a warm-up stage to a refinement stage once the ratio ρ = α/β (positive-to-negative proportion) reaches 2/λ - 1, and that positive-sample advantages get downscaled once ρ exceeds 4/λ - 1, which stabilizes training. A majority-guided strategy (shifting when positives and negatives are balanced, ρ_shift = 1) yields λ = 1 as the default.

The reasoning models trained are Qwen2.5-Math-7B and Qwen2.5-32B, with GRPO using clip-higher, token-level loss, and overlong reward shaping as the baseline, implemented in the verl framework. The GenRM is deployed as an external service on a cluster with multiple server workers and a load-balancing router, decoupled asynchronously from rollout inference and actor training.

Why This Matters

Impact on research. The paper reframes the standard binary RLVR reward as a source of systematic noise: identical rewards for reliable and unreliable reasoning. It provides both an analytical account of when penalizing flawed positives helps (after the model can produce correct rollouts) and when it hurts (during warm-up), and shows that a small, RL-trained generative reward model can carry process-level signal into large-scale RL without an impractical compute cost.

Real-world applications (plausible domains, not enumerated as such in the paper):

• Mathematical reasoning systems, where the paper evaluates on AIME24 and AIME25.
• General-domain question answering and science questions, covered by GPQA-Diamond.
• Code generation, which the paper cites as a verifiable RLVR task alongside mathematical reasoning.
• Reinforcement learning training pipelines for reasoning models, where FAPO is described as a drop-in replacement for rule-based outcome rewards.

Industry relevance. The paper targets the practical bottleneck of deploying reward models online: a 4B generative reward model that outperforms a 32B teacher, an asynchronous serving design, and a measured training-time overhead of less than 20% are all aimed at making process-level rewards affordable in large-scale training.

Future Directions

• A fully synchronous GenRM design. The authors explicitly state that while a fully synchronous design may be a better system-level solution, it lies beyond the paper's research focus and is left as important future work.
• Generalizing the reward-shift theory. The analysis derives when optimization shifts stages based on proportions of correct, flawed-positive, and negative rollouts; extending it to other reward shapes and tasks is open.
• Robustness against reward hacking. The paper shows a step-ratio reward induces jump-in-reasoning and argues FAPO is more robust, but broader reward-model hacking scenarios remain a stated algorithmic challenge.
• Scaling and domain expansion. The evaluation covers AIME24, AIME25, and GPQA-Diamond; the authors frame FAPO's effectiveness as needing demonstration in future large-scale RL systems more broadly.

Target Audience

Researchers and engineers working on RL for LLM reasoning, especially those building RLVR pipelines, reward models, or verifier-based training systems. It is also relevant to practitioners who need to make process-level reward models affordable in production training loops, and to readers interested in the reliability and failure modes of reasoning traces rather than only final-answer accuracy.

Authors’ abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models (LLMs). In this context, models explore reasoning trajectories and exploit rollouts with correct answers as positive signals for policy optimization. However, these rollouts might involve flawed patterns such as answer-guessing and jump-in-reasoning. Such flawed-positive rollouts are rewarded identically to fully correct ones, causing policy models to internalize these unreliable reasoning patterns. In this work, we first conduct a systematic study of flawed-positive rollouts in RL and find that they enable rapid capability gains during the early optimization stage, while constraining reasoning capability later by reinforcing unreliable patterns. Building on these insights, we propose Flawed-Aware Policy Optimization (FAPO), which presents a parameter-free reward penalty for flawed-positive rollouts, enabling the policy to leverage them as useful shortcuts in the warm-up stage, securing stable early gains, while gradually shifting optimization toward reliable reasoning in the later refinement stage. To accurately and comprehensively detect flawed-positive rollouts, we introduce a generative reward model (GenRM) with a process-level reward that precisely localizes reasoning errors. Experiments show that FAPO is effective in broad domains, improving outcome correctness, process reliability, and training stability without increasing the token budget.