Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs

AI summary

Overview

• Research area: LLM post-training and safety alignment, specifically reinforcement learning with verifiable rewards (RLVR) versus supervised fine-tuning (SFT) and RLHF.
• Technical level: Intermediate. The empirical sections are accessible, but the core argument rests on a KL-constrained optimization result, a covariance bound, and a chi-squared divergence bound that require comfort with probability and reinforcement learning notation.
• Scope: The paper provides a theoretical and empirical analysis of whether RLVR on math and coding tasks degrades the safety guardrails of large language models, evaluated across five adversarial safety benchmarks.

What This Paper Is About

Fine-tuning an LLM to get better at a downstream task is widely believed to damage its safety alignment, a pattern the authors call the safety-capability trade-off, and this damage has been observed even when the training data is benign. The paper asks whether reinforcement learning with verifiable rewards avoids this problem, since its reward signal comes from objectively checkable answers such as math correctness or code execution rather than from human preferences. The authors derive conditions under which safety is provably preserved and then test those conditions empirically on Qwen2.5-family models.

Key Contributions

1. Theoretical framework for RLVR safety: The authors formalize LLM generation as a two-stage process (sampling a token path, then tokens conditioned on that path), derive the optimal RLVR policy, and prove an upper bound on safety drift under KL-constrained optimization.
2. Elimination of safety degradation under independence: They prove that when the verifiable reward and the safety outcome are statistically independent, the optimal RLVR policy leaves the expected safety score exactly unchanged, and provide a worst-case bound of the square root of the chi-squared divergence between the tuned and reference policies when independence fails.
3. Empirical validation across five adversarial safety benchmarks: Using a paired before/after statistical design, they show RLVR leaves harmfulness scores and rates essentially unchanged while SFT substantially increases both.
4. Ablation studies over algorithm, scale, and task: They compare GRPO against REINFORCE++, 7B against 32B parameters, and math against code training, reporting only small differences in each case.

Main Findings

• RLVR does not degrade safety on the aggregate: For Base Model vs RLVR, the mean paired difference in harmfulness score is -0.019 with a 95% confidence interval of [-0.040, 0.006] and a p-value of 0.971; the mean paired difference in harmfulness rate is -0.032 with a 95% confidence interval of [-0.041, -0.023] and a p-value of 1.000.
• SFT does degrade safety on the aggregate: For Base Model vs SFT, the mean paired difference in harmfulness score is 0.828 with a 95% confidence interval of [0.784, 0.871] and a p-value below 0.001; the mean paired difference in harmfulness rate is 0.2475 with a 95% confidence interval of [0.227, 0.267] and a p-value below 0.001.
• The optimal RLVR policy is an exponential tilt of the reference model: The path distribution is proportional to exp{g_x(r)/β} times the reference path probability, normalized by Z(x). This means RLVR reweights the base model's density according to success probability rather than introducing an unrelated objective.
• Safety drift is bounded by normalized covariance: The change in safety score is bounded by |Cov(s_x(R), w_x(R))| / E[w_x(R)], where w_x(r) = exp(g_x(r)/β). The authors use this to argue that safety is preserved whenever the verifiable reward is uncorrelated with unsafe token trajectories.
• Worst-case safety drift is controlled by chi-squared divergence: Even without the independence assumption, the safety score change is bounded by the square root of the chi-squared divergence between the tuned and reference policies.
• Algorithm choice barely matters: Comparing GRPO and REINFORCE++ on Qwen2.5-7B-Math, the difference is slight relative to the score scale, which ranges from -4 to 4, consistent with the shared exponential-tilt optimum.
• Scale barely matters: Comparing Ours-RLVR-7B and Ours-RLVR-32B, only a slight difference appears, consistent with the size-agnostic bound.
• Task domain barely matters: Comparing code-trained CodeR1 against math-trained Ours-RLVR-7B, the change in harmfulness score for the code-trained model is negligible.
• Reasoning is maintained as a sanity check: The authors state that RLVR maintains or modestly improves reasoning performance on GSM8K, MATH500, and AIME24, though the specific accuracy values are in Table 5, which is not included in the provided content.

Methodology in Plain English

The authors start from a simple modeling idea: an LLM's answer can be thought of as first picking a "token path" (a style or trajectory of reasoning), and then generating tokens along that path. Prior work has argued that these paths largely determine whether an answer is correct and whether it is safe. Assuming the outcome of a path is fixed, the authors write the expected success and safety rates as averages over paths and derive what the best RLVR policy looks like.

That derivation shows the best policy simply upweights paths with higher predicted success, scaled by the KL regularization coefficient β. Because safety is a separate function of the path, the change in the safety score depends on how correlated safety is with that upweighting. If they are independent, safety does not move at all. If they are not, the change is still bounded, first by a covariance ratio and then by a chi-squared divergence.

For the empirical side, the authors take open-weight Qwen2.5 models and their instruction-tuned variants, plus RLVR models from SimpleRL-Zoo and CodeR1, and train their own models (Ours-RLVR-7B, Ours-RLVR-7B-REINFORCE, Ours-RLVR-32B) with GRPO and REINFORCE++ using an extended SimpleRL-Zoo recipe with zero RL training plus early stopping. Training used 8 × A100-80G GPUs per node, with 2 nodes for the 7B models and 8 nodes for the 32B model, and KL coefficients of 1e-4 for 7B and 1e-3 for 32B. The experiments section reports GSM8K as the training dataset, while Table 4 lists the MATH training dataset (levels 1-5).

Rather than comparing group averages, they use a paired design: each fine-tuned model is compared against its own base model on the same benchmark. Continuous harmfulness scores use a paired t-test, and binary harmfulness rates use the method of Newcombe (1998). Safety is measured with a DeBERTa-based harmful score predictor that outputs a score from 0 (most safe) to 4 (most harmful), and with the HarmBench classifier that outputs a binary harmful/not-harmful judgment. Evaluation uses a minimal prompt template of "<User> [Instruction] <Assistant> Let's think step by step." with temperature 0.6 and top-k 40. The five safety datasets are I-CoNa (n=178), I-Controversial (n=40), I-PhysicalSafety (n=100), Q-Harm (n=100), and I-BeaverTails (n=1000).

Why This Matters

The paper challenges a widely held assumption that any capability-improving fine-tuning must erode safety alignment, and it locates the deciding factor in reward verifiability rather than in post-training itself. If the theory holds, it gives practitioners a principled reason to prefer RLVR over preference-based or supervised fine-tuning when safety is a constraint.

Real-world applications:

• Deploying reasoning-capable assistants in domains like math tutoring or code assistance, where models must be both capable and resistant to adversarial prompting.
• Post-training pipelines for enterprise LLMs, where avoiding safety regression reduces the cost of re-running alignment stages after every capability upgrade.
• Safety evaluation and red-teaming, where the paired base-versus-fine-tuned statistical design provides a reusable way to quantify fine-tuning side effects instead of reporting only aggregate benchmark means.
• Model selection and procurement, giving teams a documented safety profile for RLVR-trained checkpoints such as the SimpleRL-Zoo and CodeR1 models.

Industry relevance: the work originates partly from the AWS Generative AI Innovation Center and uses open-weight models and public benchmarks, so its conclusions map directly onto applied fine-tuning workflows where teams fine-tune general-purpose base models for narrow tasks and need to know whether guardrails survive.

Future Directions

• Stronger reward-verification strategies: The theory shows safety preservation depends on the verifiable reward staying orthogonal to unsafe token trajectories, so designing reward functions that guarantee this property is a natural next step the authors themselves propose.
• Testing beyond math and code: The ablations cover mathematical correctness and code execution only; whether the independence condition holds for other verifiable domains such as structured extraction or tool use is untested here.
• Extending the analysis past the independence assumption: Proposition 2 bounds drift but does not say how tight that bound is in practice, leaving open how much safety drift is possible with poorly crafted rewards or unexpected model behavior.
• Efficient RLVR training grounded in theory: The authors point toward more efficient RLVR training procedures, which would connect the exponential-tilt optimum to concrete savings in compute and data.

Target Audience

Researchers and engineers working on LLM post-training, safety alignment, and reinforcement learning for language models will get the most from this paper. It is also useful for applied teams who fine-tune open-weight base models and need evidence for choosing between SFT and RLVR. Readers who want only the empirical safety comparison can rely on Table 2 and the ablation figures, while the theoretical sections require a background in probability inequalities and KL-regularized policy optimization.

Authors’ abstract

Fine-tuning large language models (LLMs) for downstream tasks typically exhibit a fundamental safety-capability tradeoff, where improving task performance degrades safety alignment even on benign datasets. This degradation persists across standard approaches including supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF). While reinforcement learning with verifiable rewards (RLVR) has emerged as a promising alternative that optimizes models on objectively measurable tasks, its safety implications remain unexplored. We present the first comprehensive theoretical and empirical analysis of safety properties in RLVR. Theoretically, we derive upper bounds on safety drift under KL-constrained optimization and prove conditions under which safety degradation is eliminated. Empirically, we conduct extensive experiments across five adversarial safety benchmarks, demonstrating that RLVR can simultaneously enhance reasoning capabilities while maintaining or improving safety guardrails. Our comprehensive ablation studies examine the effects of optimization algorithms, model scale, and task domains. Our findings challenge the prevailing assumption of an inevitable safety capability trade-off, and establish that a specific training methodology can achieve both objectives simultaneously, providing insights for the safe deployment of reasoning-capable LLMs.