TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs

AI summary

Overview

Research area: Reinforcement learning with verifiable rewards (RLVR) for large language models, specifically exploration efficiency during rollout generation.

Technical level: Advanced — the paper combines policy-gradient theory (unbiased estimators, advantage decomposition, Jensen–Shannon divergence bounds) with large-scale empirical training runs across math, code, and agentic benchmarks.

Scope: The paper proposes TGRL, a framework that partitions each prompt's rollout group into low-temperature reference and high-temperature exploration subsets, estimates prompt-level exploration gain from their reward gap, and distributes that signal as token-level credit via Jensen–Shannon (JS) divergence between the temperature-scaled next-token distributions.

What This Paper Is About

RLVR methods such as GRPO and DAPO improve LLM reasoning by optimizing over sampled completions, but they generally assume the rollout group already contains sufficiently diverse trajectories. Existing ways of adding exploration — test-time scaling, temperature schedules — either expand the rollout budget or leave the benefit of exploration unquantified.

TGRL's goal is to turn temperature-induced diversity into an explicit training signal: it measures whether broader exploration actually improves verifiable reward for each prompt, then directs that measured gain to the token positions where the temperature switch most changes the decoding distribution.

Key Contributions

1. Theoretical characterization of exploration gain and token-level credit allocation. The paper formalizes temperature as a controllable perturbation of the same policy, showing that the reward contrast between low- and high-temperature groups is an unbiased estimate of prompt-level exploration gain (Lemma 1), that mixed-group normalization shifts high-temperature advantages by an additive term proportional to that gain (Proposition 1), and that token-level JS divergence bounds the change in expected continuation value under a temperature switch (Lemma 2, Proposition 2).

2. Exploration-gain estimation with JS-based token credit allocation. TGRL combines two mechanisms: mixed-temperature grouping estimates whether broader exploration improves the verifiable reward for each prompt, and JS-based credit allocation maps this grouped signal into token-level policy-gradient credit for high-temperature exploratory trajectories.

3. Empirical gains across reasoning, coding, and agent tasks. Evaluated on 11 benchmarks with Qwen3-4B, Qwen3-14B, Qwen3-32B, and Qwen2.5-7B-Instruct, TGRL improves the six-benchmark math average by 1.6% at 32B, raises CodeForces rating by 196.7 points and LiveCodeBench Pass@16 by 4.4%, and improves ALFWorld/WebShop success rates by 6.3%/4.9%.

4. Training-efficiency and mechanism validation. Wall-clock comparisons, ablations, and training-dynamics diagnostics support the grouping-and-credit-allocation design; TGRL reaches equivalent accuracy up to 36% faster than strong RLVR baselines without expanding the rollout budget.

Main Findings

• Math benchmark average. TGRL achieves the best six-benchmark average at both Qwen3-14B (69.4%) and Qwen3-32B (70.2%), outperforming the strongest baseline by 1.4% and 1.6% respectively. At 14B it improves over the best prior result by 2.2% on AIME24, 1.5% on AIME25, and 1.6% on Olympiad; at 32B it is best on all six benchmarks, with the largest margin on Olympiad (+4.2%) plus a further gain on AIME25 (+0.4%). Each AIME set contains 30 problems, so the authors also ran 5-seed Avg@16 evaluations on AIME24, AIME25, and AMC23 for TGRL and GRPO at both scales, finding modest cross-seed variance.

• Code generation. On Qwen3-4B, TGRL obtains the best LiveCodeBench Avg@16 (43.6, +13.1) and Pass@16 (62.1, +21.2), raises CodeForces rating from 1377.6 to 1574.3 (+196.7, percentile 71.9% to 84.3%), and reaches 96.9 Pass@16 on the near-saturated HumanEval+ (+7.9).

• Long-horizon agent tasks. On Qwen2.5-7B-Instruct, TGRL reaches 86.7% overall ALFWorld success (+6.3% over the best RL baseline) and 74.2% WebShop success (+4.9%), while improving WebShop task score to 85.7%. The largest ALFWorld sub-task gain is on Pick2 (95.0% vs. 68.8% for PPO).

• Comparison to exploration-oriented baselines. On Qwen3-14B, TGRL's 69.4% average exceeds the best entropy-regularized GRPO run (67.8%) and TAMPO, a temperature-adaptive method (68.0%).

• Temperature alone is not sufficient. Raising GRPO's sampling temperature on AIME24/AIME25 improves performance from 54.8%/37.5% to 60.0%/44.2% but still trails TGRL's 61.9%/49.6%, while yielding longer, more truncated responses.

• Ablation decomposition. Raising temperature alone (GRPO @ T1 vs. GRPO @ T0) improves the average by only 0.5% at 14B and 1.1% at 32B. Mixed-temperature grouping raises the average from 67.0% to 68.2% at 14B and from 66.2% to 67.8% at 32B. Adding JS-based credit allocation further raises the average to 69.4% and 70.2%, with especially large gains on AIME25 (+2.5% at 14B; +3.5% at 32B).

• Update statistics. During steps 121–150, TGRL attains the largest gradient norm (0.331) and policy KL (0.160) while keeping the PPO clip fraction below 0.11%, indicating larger effective policy updates without widespread ratio clipping.

• Response length and truncation. TGRL produces the shortest late-stage responses, averaging 2476 tokens versus 3915 for GRPO @ T0 and 3482 for GRPO @ T1, and the lowest response clipping ratio, 7.2% versus 16.0% and 12.4%.

• Reward scale is not the source of gains. TGRL and TGRL (Uniform Token Credit) have nearly identical late-stage training rewards, 0.394 versus 0.388, yet differ in downstream accuracy, response length, and truncation.

• Temperature sensitivity. Performance degrades only when the temperature gap is too narrow (T1 = 0.8), whereas T1 in [1.0, 1.6] remains competitive; T1 = 1.2 achieves the best efficiency–stability trade-off and is the default. T0 = 0.3 serves as the conservative reference.

• Wall-clock efficiency. Within a matched 12-hour budget, TGRL attains an AIME-Combined score of 1.0833 versus GRPO's 1.0354 (+0.048 absolute); against DAPO, TGRL reaches an equivalent accuracy target 36% faster (6.4 h vs. 10.0 h).

• JS as a token-selection criterion. A local-cooling probe comparing JS-, entropy-, and margin-based span selectors finds JS-targeted spans produce the highest flip rate (12.1%) and score drop (0.241) at matched position ratios.

Methodology in Plain English

For each prompt, TGRL draws a small group of rollouts split by sampling temperature: a low-temperature reference group (via T0 = 0.3) and a high-temperature exploration group (via T1 = 1.2). The authors use a rollout budget of four responses, allocating one sample to the low-temperature group and three to the high-temperature group, reserving most of the budget for exploration.

The mean-reward difference between the two subgroups estimates how much broader exploration helps on that specific prompt. This gain is folded into a group-normalized advantage computed over the combined mixed-temperature group, so a high-temperature rollout's advantage reflects both its ranking within the group and the exploration benefit.

The scalar group-level signal is then spread across token positions. For each high-temperature trajectory, the model replays the response, recomputes next-token distributions at both temperatures from the same logits, and measures their Jensen–Shannon divergence per position. Positions with larger JS divergence are those where the temperature switch most changes what the model would say. These divergences are compressed with a monotone log transform and normalized so the average token weight equals one, then multiplied into the rollout advantage.

The low-temperature group contributes only to the group statistics (mean and variance) and receives zero token advantage gradient; optimization is restricted to the high-temperature group, since allowing low-temperature gradient updates would introduce a competing update direction that weakens the exploration-gain correction. Training uses a clipped importance-ratio (PPO-style) objective, with a 20-step warmup in which all trajectories are sampled at high temperature and optimized with the standard single-temperature group-normalized advantage, preventing the low-temperature group from becoming a degenerate baseline before the policy develops stable reasoning patterns.

Training data: DeepScaleR for math, DeepCoder for code, and WebShop and ALFWorld for agent tasks. Math and code experiments run in Qwen3 think mode with a maximum response length of 8192 tokens; math and code evaluations use decoding temperature 0.6 and top-p 0.95.

Why This Matters

The paper reframes sampling temperature from a decoding knob into a measurable training signal. Instead of simply sampling more stochastically or spending more rollout budget, it quantifies per-prompt exploration benefit and routes that benefit to the token positions where the temperature intervention actually matters — a design pattern other RLVR methods could adopt.

Real-world applications:

• Mathematical and scientific reasoning assistants — the six math benchmarks (AIME 2024/2025, AMC 2023, MATH500, Minerva, Olympiad) target competition-level and STEM reasoning, relevant to tutoring, quantitative analysis, and research-assist tools.
• Competitive programming and code assistants — gains on LiveCodeBench, CodeForces, and HumanEval+ map to code generation, algorithm selection, and debugging support.
• Long-horizon agents for household and web tasks — ALFWorld (multi-step household task completion) and WebShop (sparse-reward web shopping) represent multi-turn agents that must commit to plans under sparse feedback.
• Compute-constrained post-training — the wall-clock results (equivalent accuracy 36% faster; 1.0833 vs. 1.0354 within a 12-hour budget) matter for teams training within fixed GPU budgets.

Industry relevance: Efficient exploration under a fixed rollout budget directly affects the cost of RLVR post-training for LLM developers. Because TGRL does not expand the rollout budget and uses a standard clipped policy-gradient objective, it is compatible with existing group-relative pipelines such as GRPO, DAPO, Dr.GRPO, and RLOO that it was compared against.

Future Directions

• Adaptive multi-temperature grouping — the authors explicitly name this as future work; currently T0 and T1 are fixed, with T1 = 1.2 chosen as a default and T1 = 0.8 identified as too narrow.
• Richer token-credit allocation criteria — the paper notes JS weighting redistributes rather than creates exploration; alternative or combined criteria beyond JS, entropy, and margin (probed at matched position ratios) remain open.
• More open-ended interactive environments — extending beyond ALFWorld and WebShop to more open-ended tasks.
• Generalization of the low/high-temperature rollout split — the paper evaluates different low-/high-temperature rollout splits (Table 12 referenced but not shown in the provided content) and the low-temperature reference is excluded from actor updates, leaving the optimal partitioning scheme an open design question.

Target Audience

This paper is most useful to RL and post-training researchers and engineers working on RLVR for LLMs, particularly those concerned with exploration efficiency, rollout-budget allocation, and temperature-controlled decoding. It also suits graduate students with a background in policy-gradient methods and information theory, since the theoretical sections rely on advantage decomposition and Jensen–Shannon divergence bounds. Practitioners seeking compute-efficient alternatives to test-time scaling for math, code, and agentic training will find the empirical suite and wall-clock analysis directly applicable. The code is available at https://github.com/1229095296/TGRL/tree/main.

Authors’ abstract

Efficient exploration often remains a central bottleneck in reinforcement learning with verifiable rewards (RLVR). Although temperature control and test-time scaling strategies can increase rollout diversity of large language models (LLMs), they either expand the sample budget at rollout time or leave the benefit of exploration unquantified. To this end, we propose Temperature-Grouped Reinforcement Learning (TGRL), which turns temperature-induced diversity into an explicit training signal. For each prompt, TGRL partitions its rollout group into low- and high-temperature subsets, estimates exploration gain through their reward contrast, and allocates this group-level signal as token-level credit using Jensen--Shannon (JS) divergence between the corresponding temperature-scaled next-token distributions induced by the same logits. Notably, TGRL reaches equivalent accuracy up to 36% faster than strong RLVR baselines without expanding the rollout budget. Across 11 benchmarks from diverse domains, TGRL broadly improves over strong RLVR baselines: it improves the six-benchmark math average by 1.6% at 32B, raises CodeForces rating by 196.7 points and LiveCodeBench Pass@16 by 4.4%, and improves ALFWorld/WebShop success rates by 6.3%/4.9%. Comprehensive ablations and wall-clock analysis confirm the efficacy of all proposed components. Code is available at https://github.com/1229095296/TGRL/tree/main.