Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error

AI summary

Overview

• Research area: Machine learning / large language model post-training — specifically reinforcement learning with verifiable rewards (RLVR) for mathematical and scientific reasoning.
• Technical level: Intermediate. The paper assumes familiarity with policy-gradient RL, importance sampling, and on-policy versus off-policy training, though the central idea (reusing the model's own mistakes as hints) is intuitive.
• Scope in one sentence: The paper proposes LTE, a method that helps a language model break out of "exploration stagnation" during RLVR by feeding it its own previously generated wrong answers as hints in extra rollouts, without any external expert guidance.

What This Paper Is About

RLVR trains language models by sampling responses and rewarding correct ones, but if a training problem is too hard for every one of the sampled rollouts, all rewards are zero, all advantages collapse to zero, and the model learns nothing from that problem. This creates "exploration stagnation": the model can never solve problems it could not already solve, and it stays bounded by its initial capability. The paper's goal is to let the model break that ceiling using only information it produced itself — its own failed attempts — rather than human-written solutions or reasoning traces from stronger models.

Key Contributions

1. LTE (Learning to reason from Trial and Error): a method that targets none-pass training samples (problems where no rollout is verified correct) by running extra rollouts with hints drawn from the model's own mistaken answers, requiring no explicit external guidance from humans or stronger models.
2. Empirical validation across two language models and six mathematical benchmarks, in both Pass@1 and Pass@k, showing LTE outperforms GRPO, GRPO with vanilla extra rollouts, EvoCoT, ReLIFT, DAPO, and LUFFY, and that its entropy-loss-enabled variant LTE† performs better than methods that rely on external guidance.
3. Training-data and training-dynamics analysis showing LTE reduces the number of unsolved (none-pass) samples and preserves more some-pass samples during training, keeps relatively higher entropy in the long tail, and increases response length — evidence that it mitigates exploration stagnation while encouraging both exploitation and exploration.
4. Ablations and cost analysis isolating the contribution of the two hint types (incorrect answers and the conciseness instruction) and comparing training time in days against all baselines.

Main Findings

• Headline improvement over GRPO: LTE outperforms normal GRPO by 5.02 in Pass@1 and 9.96 in Pass@k on average across six mathematical reasoning benchmarks for Qwen3-8B-Base. Averaged over both models, LTE beats GRPO and GRPO with vanilla extra rollouts by +4.94 and +5.22 in Pass@1.
• Hinted rollouts beat plain extra rollouts: LTE outperforms its counterpart that simply performs extra rollouts by +7.29 Pass@1 and +10.04 Pass@k averaged across the six benchmarks. Merely scaling rollouts (GRPO with extra rollouts) brought only marginal or even negative improvement to GRPO.
• No external guidance needed to beat guided methods: LTE† exceeds LUFFY — which uses reasoning traces from stronger models — by +2.41 Pass@1 and +2.15 Pass@k on Qwen3-8B-Base, and by +2.66 points on average for Pass@1 across the two models. It also beats DAPO by +4.40 on average in Pass@k and LUFFY by +1.79 on average in Pass@k.
• Exploration upper bound expanded: LTE improves Pass@k over the base models by +7.30 and +13.91 for the two models, which is 3.1× and 3.6× the improvement produced by GRPO with extra rollouts (+2.39 and +3.87). LTE† improves Pass@k over the base models by +13.23 and +14.71.
• Stagnation is visibly reduced: In Figure 4(a), normal GRPO becomes stuck after about 150 training steps and cannot solve more none-pass samples; LTE and LTE† keep reducing the number of unsolved problems even in the second half of training. LTE and LTE† also keep a higher number of some-pass samples, i.e., non-zero-gradient groups that carry learning signal.
• Validation dynamics: In Figure 5(a), MATH-500 Pass@1 for GRPO and GRPO with extra rollouts stagnates after 100 training steps; LTE and LTE† keep improving. In Figure 5(b), the baselines' Pass@4 slightly shrinks after 300 steps, while LTE and LTE† keep raising the upper bound. LTE entropy stays relatively high in the long tail after 100 steps, and LTE increases response length more, especially LTE†.
• Ablation: Removing the incorrect-answer hint costs 4.54 Pass@1 and 4.30 Pass@k on average; removing the concise hint costs 1.82 and 1.71; removing both (i.e., GRPO with extra rollouts) causes a degradation of 7.29 and 10.04. The authors state in appendices that the off-policy and importance-sampling components are necessary and that the policy-shaping trick is not the source of the improvement.
• Cost: Training time in days (Qwen3-4B-Base / Qwen3-8B-Base): GRPO 5.0/7.0, GRPO with extra rollouts 5.5/7.0, EvoCoT 1.5/2.0, LTE 6.0/9.0, ReLIFT 9.0/12.5, DAPO 6.5/7.0, LUFFY 5.5/8.0, LTE† 7.5/9.0. The authors note LTE requires roughly two more days (around 30%) than GRPO on average. EvoCoT is fastest because it learns to emit extremely short responses but does not outperform the GRPO baseline. With cost roughly matched (LTE at 400 steps instead of 500), LTE trained for both 5.0 days still improves Qwen3-8B-Base's Pass@1 and Pass@k over GRPO by +4.63 and +8.83; on Qwen3-4B-Base, LTE† at 300 steps beats LUFFY by +0.63 Pass@1 and +2.71 Pass@k with 1.0 fewer day.
• Beyond mathematics: On SciQ with Qwen3-4B-Base, LTE reaches 55.58 / 58.18 (Pass@1/Pass@4) on the validation set versus GRPO's 51.13 / 52.72, and 57.75 / 59.83 on the test set versus GRPO's 55.43 / 57.35. Base model scores are 15.50 / 28.89 (valid) and 17.60 / 32.37 (test).
• Baseline caveat: The authors ran the exact training scripts from EvoCoT and ReLIFT but obtained relatively poor results, which they attribute to foundation-model compatibility and leave for future work.

Methodology in Plain English

The method works inline with standard GRPO training. For each training batch, the model samples 8 rollouts per prompt, and answers are extracted and checked. Most samples are left untouched. The intervention applies only when all 8 rollouts fail verification (a none-pass sample). The system then inspects whether responses were truncated by the length limit:

• If all responses were truncated, the failure is blamed on verbosity, and the model is re-prompted to think concisely.
• If some or none were truncated, the extracted wrong answers are inserted into the prompt as a warning not to repeat them, and for the some-truncated case the model is also asked to think concisely. The prompt tells the model not to explicitly use or mention the hint.

A further 8 hinted rollouts are then sampled. If any succeed, that many of the original failed responses are randomly replaced with the correct ones so the group contains a learning signal. Because the correct responses were generated under the hinted prompt rather than the plain prompt, they are treated as off-policy: the importance-sampling ratio divides by the hinted sampling distribution, and a regularized importance-sampling function is applied — the same mixed-policy update style used by LUFFY. Everything else, including the KL term and clipping for the on-policy responses, follows GRPO.

Training setup: Qwen3-4B-Base and Qwen3-8B-Base, the OpenR1-Math-46k-8192 dataset (45,792 instances), the verl framework, temperature 1.0, maximum response length 8,192, 500 training steps, batch size 128. Evaluation covers MATH-500, Minerva, OlympiadBench, AMC'23, AIME'24, AIME'25; MATH-500, Minerva and OlympiadBench are sampled 4 times reporting Mean@4 and Pass@4, and AMC'23, AIME'24, AIME'25 are sampled 16 times reporting Mean@16 and Pass@16, with context length 32,768, temperature 0.6, top-p 0.95, top-k 20. Hints are used only during training; evaluation follows the standard protocol.

Why This Matters

The paper targets a structural weakness of RLVR: a model cannot learn from problems it cannot solve at all, so the training signal vanishes exactly where it is most needed. The paper's distinctive claim is that this ceiling can be raised using the model's own failed attempts rather than human-annotated solutions (costly, limited in scale) or reasoning traces from stronger models (which may not exist when training flagship models). This shifts the source of guidance from external experts to the model's own trial-and-error history.

Real-world applications:

• Automated tutoring and homework-help systems that explain why a common wrong answer is wrong, using the system's own error patterns rather than a curated error bank.
• Post-training pipelines for reasoning models where gold reasoning traces or a stronger teacher model are unavailable or too expensive to produce.
• Domain assistants in mathematics and science where answers are objectively checkable and short, the setting LTE was validated on.
• Self-improving agents in low-resource or proprietary domains where no external expert data exists but the system can generate and verify its own attempts.

Industry relevance: the results suggest that a training run can recover learning signal from an otherwise wasted fraction of the batch, and the cost analysis (roughly two extra days versus GRPO in these experiments, but still fewer steps needed when training budget is matched) makes the trade-off explicit for teams deciding how to allocate compute. The comparison against LUFFY, DAPO, and EvoCoT gives practitioners direct reference points, and the authors report that drop-in replication of EvoCoT and ReLIFT did not transfer well to these base models — a caution for anyone planning to reproduce published recipes.

Future Directions

• Extending LTE to longer-horizon and multi-turn tasks such as coding and tool-use agents. The authors note the current method fits short-answer reasoning, and suggest using a summarizer whose summaries of previously failed interaction trajectories serve as hints.
• Applying hints at more fine-grained intervals rather than only for whole none-pass groups.
• Incorporating ground-truth answers into the hint while avoiding reward hacking — a direction the authors explicitly flag.
• Scaling to larger language models and longer maximum response lengths. The paper's limitations state that training used a relatively short maximum response length of 8,192, which may not fully unlock reasoning capability, and that larger-scale models were not tested due to limited computational resources.
• Investigating why EvoCoT and ReLIFT underperformed with these base models, which the authors attribute to foundation-model compatibility and leave open.

Target Audience

This paper is most useful to machine learning researchers and engineers working on RLVR and reasoning-model post-training, particularly those who need to raise a model's capability ceiling without access to human-annotated reasoning traces or a stronger teacher model. It also benefits practitioners building evaluation or training pipelines for mathematical and scientific reasoning, and students with some grounding in reinforcement learning who want a clear example of diagnosing and fixing a training pathology. Readers focused on long-horizon agentic tasks, coding, or multimodal reasoning will find the scope explicitly limited by the authors' own stated constraints.

Authors’ abstract

Reinforcement learning with verifiable rewards (RLVR) has significantly boosted the reasoning capability of language models (LMs). However, existing RLVR approaches train LMs based on their own on-policy responses and are constrained by the initial capability of LMs, thus prone to exploration stagnation, in which LMs fail to solve more training problems and cannot further learn from the training data. Some approaches try to address this by leveraging off-policy solutions to training problems, but rely on external expert guidance that is limited in availability and scalability. In this work, we propose LTE (Learning to reason from Trial and Error), an approach that hints LMs with their previously self-made mistakes, not requiring any external expert guidance. Experiments validate the effectiveness of LTE, which outperforms the normal group relative policy optimization (GRPO) by 5.02 in Pass@1 and 9.96 in Pass@k on average across six mathematical reasoning benchmarks for Qwen3-8B-Base and even performs better than methods that require external guidance. Further analysis confirms that LTE successfully mitigates exploration stagnation and enhances both exploitation and exploration during training. Our code is available at https://github.com/JamyDon/LTE.