On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training

AI summary

Overview

• Research area: Large language model post-training, specifically the geometry of parameter updates and how it relates to generalization.
• Technical level: Advanced. The paper combines a gradient-level theoretical analysis (policy gradient versus cross-entropy gradient) with large-scale controlled experiments across multiple models, datasets, and benchmarks.
• One-sentence scope: The paper shows that on-policy post-training (GRPO, OPD) continuously changes its parameter update direction while SFT updates along nearly fixed directions, and that constraining SFT updates to the on-policy direction (a method called OPSFT) transfers much of the on-policy generalization advantage to SFT.

What This Paper Is About

Prior work has observed that on-policy post-training paradigms generalize better than supervised fine-tuning (SFT) and has catalogued behaviors such as sparse update locations, but has treated these behaviors as byproducts rather than as usable optimization principles. This paper asks whether one specific behavior — the cumulative parameter update direction — can be extracted from on-policy training and used to improve SFT. The authors answer yes, proposing OPSFT, which restricts SFT updates to the sign vector (direction) of the cumulative on-policy update.

Key Contributions

1. Analysis of on-policy update direction. The authors show theoretically (via a comparison of the SFT and policy-gradient formulations) and experimentally (via cosine similarity of interval and cumulative updates) that SFT updates parameters along consistent directions, whereas on-policy paradigms continuously adjust their update direction during training.

2. OPSFT (On-Policy direction-constrained Supervised Fine-Tuning). A method that takes the sign vector v = sign(θ_on − θ_base) from an on-policy run and, at each SFT step, masks gradient elements whose induced movement opposes v, with an additional hard projection after the optimizer step. The paper reports that OPSFT substantially outperforms vanilla SFT and reaches performance comparable to the on-policy paradigm supplying the direction.

3. Demonstration that direction matters beyond location. OPSFT constrained only by update locations (the {0,1}^d sign vector) performs worse than vanilla SFT, while extending the constraint to directions ({−1,0,1}^d) yields substantial gains — indicating that location alone does not transfer the generalization advantage.

4. Two practical applications. (a) Using a small number of on-policy steps to identify the direction and then running OPSFT, achieving strong accuracy with less training time than GRPO; (b) using OPSFT to continue improving an already post-trained model along its original on-policy direction using newly acquired high-quality trajectories, without degrading the post-trained capability.

Main Findings

• Directional difference between paradigms. For interval updates, SFT is positively correlated across training steps while on-policy paradigms explore nearly orthogonal directions. For cumulative updates, SFT shows cosine similarity close to 1.0, while on-policy methods show roughly 0.5. These measurements use Qwen3-1.7B on DeepMath.

• Theoretical explanation. The on-policy gradient projection equals the projection of the covariance between the advantage A_θ(x,τ) and the trajectory gradient s_θ(x,τ); because trajectories are sampled from the evolving policy, this direction shifts. The SFT gradient projection instead depends on s_θ(x,τ) averaged over a fixed teacher distribution.

• Generalization transfer. OPSFT consistently outperforms vanilla SFT and approaches GRPO across models and benchmarks. On out-of-domain benchmarks (trained on DeepMath), Qwen3-4B mean accuracy is 73.43 (Base), 73.65 (SFT), 73.95 (GRPO), and 74.11 (OPSFT); Qwen3-8B mean is 75.85 (Base), 75.54 (SFT), 76.20 (GRPO), and 76.39 (OPSFT).

• Location constraints are insufficient. OPSFT restricted only to the on-policy update locations performs worse than vanilla SFT, whereas the direction constraint produces substantial gains.

• Training efficiency. OPSFT (50 GRPO steps for direction identification plus 100 OPSFT steps) beats 100-step GRPO while taking less than half the time. On Qwen3-8B, OPSFT reports 8.9h and 41.67 mean accuracy versus GRPO's 19.3h and 40.31; on Qwen3-4B, 6.4h and 40.11 versus GRPO's 16.5h and 38.96; on Qwen3-1.7B, 2.3h and 15.11 versus GRPO's 4.9h and 12.09; on DeepSeek-R1-Distill-LLaMA-8B, 8.1h and 26.56 versus GRPO's 18.2h and 25.21. DFT at 14.1h reaches 19.90 on that last model.

• Improving post-trained models. Directly applying SFT to a GRPO-post-trained model degrades performance (Qwen3-4B drops from 38.96 to 36.25), while OPSFT improves it (38.96 to 41.36). Similar patterns hold for Qwen3-1.7B (12.09 → 16.77), Qwen3-8B (40.31 → 42.39), and DeepSeek-R1-LLaMA-8B (25.21 → 26.15).

• Reusability across datasets within a domain. Update-location overlap between same-domain and cross-domain datasets is similar (around 0.4), but directions are much more similar within a domain. Even within math, DAPO and DeepMath directions have a cosine similarity of only about 0.2, yet OPSFT trained on DeepMath using the DAPO-identified direction slightly outperforms using the DeepMath-identified direction. Directions from the code domain (Eurus) do not transfer well to math.

• Later-stage directions help more, with diminishing returns. Directions identified at later GRPO steps produce better OPSFT performance, and the gain from 100 steps over 50 steps exceeds the gain from 150 steps over 100 steps.

• Parameter precision effects (Qwen3-8B). Under BF16, SFT updates 2.702% of parameters (mean 38.33) while OPSFT updates 0.408% (mean 40.94, a 24.59% improvement over the base model's 16.35). Under FP32, SFT updates 89.286% (38.41), OPSFT 9.454% (42.81), and GRPO 9.499% (42.01) — so OPSFT can exceed the GRPO run used to derive its direction.

• Reasoning behavior. Vanilla SFT, OPSFT, and GRPO have similar response lengths, but OPSFT shows higher standard deviation in accuracy across responses, approaching GRPO. Analysis of case-splitting cues (e.g., if, otherwise, consider a case) and logical cues (e.g., therefore, because) indicates the direction constraint makes OPSFT's reasoning more closely resemble on-policy paradigms rather than simply lengthening responses.

• Code-domain results. With Qwen3-4B on Eurus, OPSFT reports 9.5h and 58.07 mean versus GRPO's 20.3h and 57.19 and SFT's 9.8h and 54.30. When continuing a GRPO-post-trained model on Eurus, OPSFT reaches 60.05 versus the base 57.19 and SFT 48.71.

Methodology in Plain English

The authors start by writing down the gradient of SFT and the gradient of policy-gradient methods in comparable form and comparing them through their projection onto an arbitrary sign vector. This shows that SFT pulls toward a fixed teacher distribution's trajectory gradients, while on-policy methods pull in the direction of the covariance between how good a trajectory is and how its log-probability changes — a quantity that moves as the policy moves.

They then verify this empirically by tracking the cosine similarity of parameter update vectors at different training steps, separately for interval updates (between consecutive stages) and cumulative updates (from the initial weights).

From there, they define OPSFT: run an on-policy method (GRPO) for a limited number of steps, record the sign of the total parameter change θ_on − θ_base, and then run standard SFT while (a) masking gradients that would push a parameter opposite to that sign and (b) resetting, after each optimizer step, any parameter whose cumulative displacement from the initial weights violates the sign constraint. The second step exists because the AdamW optimizer's weight decay and momentum could otherwise break the constraint.

They evaluate on math (AIME24, AIME25, HMMT25-Feb, HMMT25-Nov) and code (HumanEval+, MBPP+, LiveCodeBench v6), using Qwen3-1.7B/4B/8B and DeepSeek-R1-Distill-Llama-3-8B, with DeepMath, DAPO, and Eurus-RL-Code as training data and Qwen3-30B-A3B as the trajectory generator. They also run ablations on parameter precision, on which GRPO step provides the direction, and on reusing directions across datasets.

Why This Matters

Impact on research. The paper challenges the common framing that "SFT memorizes while RL generalizes" by showing that SFT can generalize strongly once its update direction is constrained. It also reframes on-policy update behaviors — previously studied as descriptive properties — as transferable optimization principles, and it argues that update directions, not just locations, are the key object of interest in post-training analysis.

Real-world applications:

• Training reasoning models in domains with reliable verifiers (math, code), where a short RL phase can be used to lock in a generalization-friendly direction and a longer, cheaper SFT phase does the bulk of the work.
• Incremental improvement of deployed post-trained models on newly collected high-quality trajectories, without restarting the full pipeline from the base model.
• Code assistants, where the paper reports OPSFT gains on HumanEval+, MBPP+, and LiveCodeBench v6 in both standalone training and post-training continuation.
• Compute-constrained post-training pipelines, where the reported OPSFT runs use less wall-clock time than GRPO while matching or exceeding its accuracy.

Industry relevance. The reported training-time reductions (for example, 8.9h versus 19.3h on Qwen3-8B) and the ability to reuse previously trained models for further improvement are directly relevant to teams that produce reasoning or coding models under compute and schedule constraints. The paper's public code is at https://github.com/ssfgunner/OPSFT.

Future Directions

1. Determining how individual components of on-policy paradigms (for example, the reverse KL term in OPD or negative samples in GRPO) contribute to identifying generalization-friendly directions.

2. Understanding how properties of reasoning trajectories affect model generalization through OPSFT.

3. Characterizing why multiple distinct update directions can support effective generalization within the same domain (the paper reports DAPO and DeepMath directions with cosine similarity around 0.2 that both work), and what determines cross-domain transferability.

4. Extending the analysis and method beyond math and code to other domains and tasks.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning for language models, and fine-tuning methodology. The paper is most useful to readers already comfortable with policy gradients, advantage functions, and optimizers such as AdamW, since the central argument rests on a comparison of gradient formulations. Practitioners who run GRPO or SFT pipelines at scale will also find the efficiency and post-training-continuation results directly applicable, while those new to post-training can still follow the high-level claim that update direction, not just update location, drives generalization.

Authors’ abstract

The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these studies treat the observed behaviors only as byproducts in on-policy training, overlooking their potential to serve as optimization principles for improving the generalization of other paradigms such as supervised fine-tuning (SFT). To address this limitation, we investigate whether there exists a specific on-policy update behavior that can achieve such improvements. First, our analyses reveal that SFT updates parameters along consistent directions, while the on-policy paradigm continuously adjusts the direction during training. This difference inspires us to focus on the cumulative update direction of each parameter as a promising behavior. Then, we evaluate its effectiveness for improving generalization by proposing On-Policy direction-constrained Supervised Fine-Tuning (OPSFT), which constrains SFT updates to the direction identified by on-policy paradigms. The strong performance of OPSFT indicates that the generalization advantage of on-policy paradigms can be transferred to SFT through the parameter update direction. Once such a direction is identified, even SFT can generalize with its updates constrained to this direction. This finding offers two practical benefits by combining the strong generalization of on-policy paradigms with the advantages of SFT, including the high training efficiency and ability to leverage high-quality trajectories. For efficiency, we identify update directions that support strong generalization using a few on-policy training steps, and subsequently apply OPSFT to achieve high training efficiency. For leveraging high-quality trajectories, OPSFT can utilize these trajectories to continue improving a post-trained model along its update direction without disrupting the ability learned from on-policy training.