Unlocking Token Rewards via Training-Free Reward Attribution

AI summary

Overview

Research area: Reinforcement fine-tuning (RFT) of large language and vision-language models, specifically credit assignment and reward design for reasoning tasks.

Technical level: Intermediate. The paper assumes familiarity with reinforcement learning (policy optimization, advantage estimation, GRPO), reward models (outcome vs. process rewards), and token embeddings, but the core idea is explained plainly enough for readers with a general ML background.

Scope: The paper introduces P2T (Process-to-Token), a training-free method that converts coarse process-level rewards from an existing Process Reward Model (PRM) into token-level rewards using a first-order gradient approximation, and evaluates it across multimodal and text-only reasoning benchmarks. (Published at CVPR 2026; code at https://github.com/stonewst/P2T.)

What This Paper Is About

Reinforcement fine-tuning for reasoning relies either on outcome rewards (was the final answer correct?) or process rewards from a PRM (was each reasoning step good?), but both are coarse: a process reward still covers hundreds or thousands of tokens in a single step, so most tokens get identical credit regardless of their actual contribution. Existing attempts at token-level rewards either require training a dedicated token reward model on noisy pseudo-labels or rely on cheap heuristics like token entropy that are not aligned with real token quality. P2T asks how to get token-level reward signals that are simultaneously cost-effective, principled, and robust — and answers by attributing a PRM's reward to individual tokens without training anything new.

Key Contributions

1. A principled, training-free algorithm that converts process-level rewards from any differentiable PRM into token-level rewards, bridging coarse and fine-grained supervision for RL without training a dedicated token reward model.
2. An efficient gradient-based estimator for token attribution: a first-order Taylor approximation replaces the naive per-token substitution (which would need N forward passes for an N-token sequence) with a single forward and backward pass, reducing complexity from O(N) to O(1).
3. Empirical validation across many settings showing that RL training with the derived token-level rewards beats both sparse (outcome) rewards and other fine-grained reward methods — across model families from 1.5B to 7B parameters, multiple reasoning paradigms, and both text-only and multimodal inputs.
4. Additional demonstrated benefits: P2T improves models already converged under outcome-based RL, and reaches the best test accuracy of the outcome-reward counterpart with 3–4x fewer training steps.

Main Findings

• Consistent gains everywhere: GRPO+P2T outperformed GRPO on every benchmark for every model — across 40 model–benchmark pairs (5 for the multimodal model, 35 for the text-only settings), with no exceptions. Average gains ranged from +2.5% to +9.4%.

• Headline numbers: P2T beats the outcome reward by +4.9% on MathVista for Qwen2.5-VL-7B-Instruct, by +11.5% on AIME24 for Qwen2.5-Math-7B, and by +14.2% on MinervaMath for DeepSeek-R1-Distill-Qwen-1.5B. On Qwen2.5-Math-7B, average accuracy rose from 27.3 (base) to 48.0 (GRPO) to 53.8 (GRPO+P2T).

• Largest gains where outcome RL stalls: On DeepSeek-R1-Distill-Qwen-1.5B, GRPO improved average accuracy by only +1.6% over the initial model, while GRPO+P2T improved it by +8.4%. On MathVista, GRPO added only +2.0% to Qwen2.5-VL-7B-Instruct, while GRPO+P2T added +6.9%.

• Larger gains on harder benchmarks: Averaged over AIME24, AIME25, and AMC23, GRPO+P2T improved over GRPO by +4.0% to +12.2% across the five text-only settings — roughly two to seven times its gain on MATH-500 (+0.6% to +5.8%).

• Raises the ceiling of already-converged models: Continuing to train Vision-R1-7B and DeepScaleR-1.5B-Preview — both already fully trained with outcome-based RL — on their original data (Vision-R1-10K and DeepScaleR-40K) improved both on every benchmark, e.g., +6.8% on AIME24 and +12.1% on MinervaMath for DeepScaleR-1.5B-Preview.

• Better training efficiency: Tracking Qwen2.5-Math-1.5B on AIME25, GRPO+P2T surpassed GRPO's best accuracy much earlier — reaching the same test performance with 3–4x fewer training steps.

• Beats existing token-level rewards: Starting from Eurus-2-7B-SFT on Eurus-2-RL-Data, P2T achieved the best result on all six benchmarks, improving average accuracy over GRPO by +7.2%, over PRIME by +4.7%, and over SPRO by +2.3% (40.7 average vs. 38.4 for SPRO, 36.0 for PRIME, 33.5 for GRPO, 23.3 for the SFT baseline).

• Within-step credit assignment is the real source of gain (ablation): Adding the step-level process reward to the outcome reward raised accuracy from 48.0 to 50.6; redistributing that same reward among tokens with P2T raised it further to 53.8. Of the overall +5.8% gain, +3.2% came from within-step credit assignment rather than the step-level scores themselves.

• Robust to the redistribution strength η: Accuracy varied by at most 0.6% as η ranged from 0.25 to 1.0 (53.5, 53.8, 53.4, 53.2 for η = 0.25, 0.5, 0.75, 1.0), all well above the process-reward baseline of 50.6 at η = 0.

• Magnitude beats sign: The magnitude-based redistribution (53.8) outperformed the signed variant (52.6), though the signed variant still beat the process-reward baseline (50.6). The authors suggest the positive and negative terms are generally imbalanced within a step.

• Better PRMs yield better token rewards: Among text-only PRMs, ReasonFlux-PRM-7B scored best (53.8), followed by Qwen2.5-Math-PRM-7B (52.4) and Skywork-o1-Open-PRM-Qwen-2.5-7B (50.0); among multimodal PRMs, VisualPRM (57.5) beat MM-PRM (55.3). Every PRM tested improved over the outcome-reward baseline (48.0 text-only, 54.9 multimodal). The 3.8% gap between best and worst text-only PRMs is comparable to P2T's own gain, showing PRM quality matters substantially.

• Null token choice matters: The PRM's padding token <|endoftext|> performed best (53.8), followed by the vocabulary mean embedding (52.5), the all-zero embedding (51.4), and the EOS token <|im_end|> (50.3). The authors attribute this to how neutral each baseline is: <|im_end|> is tied to response completion, precisely what the PRM judges, so substituting it injects an evaluation-related signal instead of removing information.

Methodology in Plain English

The core idea. Take a PRM that gives one score per reasoning step. For each token in that step, ask: how much would the step's score change if this token's content were removed? Define the Token Attribution Score as the difference between the original reward and the reward after replacing that token's embedding with a "null" token embedding (the padding token is used, since it almost never appears inside a reasoning response and carries little content meaning there). A positive score means the token helped; negative means it hurt; near zero means it barely mattered.

The efficiency trick. Computing this naively means running the PRM once per token — N forward passes for an N-token sequence. Instead, the authors use a first-order Taylor expansion, approximating each token's influence as the inner product of two things: (a) the gradient of the reward with respect to that token's embedding, and (b) the difference between the token's embedding and the null token embedding. Because gradients for all token embeddings come out of a single backward pass, the whole sequence's scores are obtained with just one forward and one backward pass — turning O(N) into O(1).

Turning scores into rewards. Rather than using the raw scores as rewards, the method normalizes their magnitudes within each step to form weights, then redistributes the original step reward using those weights. The formula keeps the original step reward as a baseline term and adds an η-weighted term proportional to each token's normalized influence. The baseline is important because gradient-based attributions can be noisy; anchoring to the step reward prevents the policy from being whipsawed by a few bad estimates. The most influential tokens in a correct step get the most credit; the most influential tokens in an erroneous step get the heaviest penalties.

Plugging into RL. For GRPO, the standard advantage is the normalized outcome reward within a group. P2T adds a weighted token-level term: the token advantage equals the outcome-derived advantage plus α times the P2T token reward, with α = 1.0 by default and η = 0.5 by default. When the PRM is less reliable, the paper suggests a smaller α or a schedule that increases α gradually during training.

Experimental setup. Multimodal training used Qwen2.5-VL-7B-Instruct on a mixture of VerMulti-65K and Vision-R1-10K, trained for 1 epoch with global batch size 128. Text-only models were trained for 500–1000 steps with global batch size 256, using a constant learning rate of 1×10⁻⁶ and weight decay 0.01, with 8 responses per prompt at temperature 1.0 and top-p 1.0, and a KL coefficient of 0.001. Evaluation sampled 8 responses at temperature 0.6 and top-p 0.95, reporting average Pass@1. Process rewards came from VisualPRM for multimodal models and ReasonFlux-PRM for text-only models.

Why This Matters

Impact on research. The paper offers a route out of a documented dilemma in token-level supervision: either pay heavily to train an uncertain token reward model, or settle for heuristic proxies with no semantic grounding. P2T shows that a differentiable coarse reward model already contains enough information to recover fine-grained credit, and that the extraction can be done with standard autodiff. It also suggests that the value of a process reward model is not fixed — the same PRM signal becomes substantially more useful once attributed to the tokens responsible for it. Because the method is training-free and model-agnostic, it can be layered onto existing RL pipelines and improves automatically as stronger PRMs appear.

Real-world applications:

• Mathematical and competition reasoning assistants — the biggest reported gains are on competition-level problems (AIME-style), the setting where a single faulty step invalidates a long chain of reasoning.
• Multimodal tutoring and diagram-based problem solving — demonstrated on multimodal math and general multimodal benchmarks with a vision-language model, relevant to educational tools that reason over images and text.
• Code generation and competitive programming — the Eurus-2-7B-SFT comparison includes CodeForces and CodeContests, where step-level correctness is similarly critical.
• Cost-constrained RL training pipelines — reaching comparable accuracy in 3–4x fewer training steps directly reduces compute expenditure.

Industry relevance. The method requires no additional reward model to be trained or maintained during RL, which matters for teams that already have a PRM in production. Training-cost savings from fewer steps and the ability to squeeze more performance out of a model that has already converged under outcome-based RL are both practically attractive. The main dependency is PRM quality, which puts the operational burden on selecting or validating an appropriate PRM for the target domain and response format.

Future Directions

1. Making better use of the sign of attribution scores. The signed redistribution variant underperformed the magnitude-based one (52.6 vs. 53.8), which the authors attribute to imbalanced positive and negative terms within a step. They explicitly leave a more effective use of sign information to future work.
2. Reducing dependence on PRM quality. Since P2T inherits its rewards from a PRM, an unreliable PRM limits its benefits; the paper recommends verifying that a PRM's step-level judgments are reliable for the target domain and response format before use. Improving robustness here is an open problem.
3. Mitigating noise in gradient-based attribution. The paper introduces the step reward as a stabilizing baseline specifically because first-order attribution scores can be noisy or imprecise; further work could improve the fidelity of the estimator itself or adapt the noise-control weight α during training.
4. Extending beyond the settings tested. The paper covers math and code reasoning with text-only and multimodal inputs across Qwen and LLaMA families; whether the approach transfers to other domains and reward models is not established here.

Target Audience

Researchers and practitioners working on reinforcement fine-tuning of large models who need finer-grained credit assignment than outcome or step-level rewards provide; engineers building RL training pipelines who already have access to a PRM and want to improve reasoning performance without training additional reward models; and readers interested in attribution methods and gradient-based approximations applied to language model training. Readers comparing token-level reward approaches (PRIME, SPRO, OREAL, CAPO, TVM, T-SPMO) will find the direct empirical comparison with PRIME and SPRO particularly relevant.

Authors’ abstract

In this paper, we propose an extremely efficient, training-free method to extract token-level reward signals directly from an existing deep reward model. Our core idea is to attribute the overall process reward to individual tokens by estimating each token's influence. This influence is defined as the change in the final macroscopic reward (e.g., the process reward) when a token is replaced with a semantically null token. Naively calculating this influence is computationally infeasible, requiring N forward passes through the PRM for an N-token sequence. We overcome this bottleneck by proposing a highly efficient gradient-based estimator. Specifically, we use a first-order Taylor approximation, which simplifies the influence calculation to the inner product of the difference between the token embedding and the null token embedding, and the gradient of the reward with respect to the token embedding. This requires only a single forward and backward pass. The resulting token-level rewards enable standard RL algorithms to perform precise credit assignment without requiring additional reward model training. Experiments on challenging reasoning benchmarks demonstrate that our method substantially improves policy optimization efficiency and enhances the generalization of LLM reasoning capabilities. Our P2T outperforms the outcome reward by +4.9% on MathVista for Qwen2.5-VL-7B-Instruct, and +11.5% on AIME24 for Qwen2.5-Math-7B. The code will be released via https://github.com/stonewst/P2T.