The Pulse
Apple Tests Short Self-Written Hints for Hard RL Tasks
Apple’s RLTL;DR method trains a language model to turn failed attempts and verifier feedback into brief insights it can apply on later tasks. In tests on difficult coding and tool-use problems, the approach raised success rates where standa
AI.info Team · Written with AI ·
On October 1, Apple highlighted RLTL;DR, a method that gives a language model a short note from its own failed attempts and trains it to carry the lesson into future tasks. The paper’s claim is narrow but notable: brief, self-generated feedback can give reinforcement learning a path forward when the model almost never solves a task by chance.
When 128 attempts produce no success
Many reinforcement-learning systems use verifiable rewards: a model tries a task, a checker judges whether it succeeded, and training favors successful attempts. That feedback loop stalls on problems so difficult that none of a model’s attempts succeeds. The authors study this low-signal setting, where they also assume there is no stronger teacher model or known solution to copy.
The experiments use a Qwen 3.5 9B Thinking policy on challenging coding and tool-calling datasets filtered to tasks with a Pass@128 of zero—that is, no success in 128 attempts. Standard Group Relative Policy Optimization, or GRPO, stayed at a Pass@1 of 0% to 1% on those tasks, according to the paper. Pass@1 measures success on a single attempt.
A verifier’s feedback becomes a short hint
After a failed attempt, RLTL;DR gives the policy the verifier’s output and asks it to write one concise insight. The next attempt receives the accumulated insights, so the model can use feedback from earlier failures while it keeps trying. The authors say the insights average 17 tokens and tend to name one corrective action, rather than provide a full answer or corrected code.
“We identify that the key is the task to insight internalization,” write Michael Kirchhof and co-authors in their paper. During training, the method also backpropagates through the insight tokens, teaching the policy to associate types of tasks with useful guidance. Apple’s research page reports Pass@1 of 14% to 31% when insights appear in the training context; on evaluation attempts without an insight present, the reported range is 12% to 13%.
The gains depend on what gets measured
The no-insight evaluation matters because it tests whether training changed the policy, rather than whether a hint simply helps during the attempt. The results suggest that the model can internalize some of the guidance and apply it without seeing the note again. They do not show that the method solves every hard task: in the paper’s detailed results, performance varies across datasets, and the authors report overfitting concerns on LeetCode.
The work also separates the proposed method from a stronger claim about reinforcement learning in general. On the authors’ normal-difficulty dataset splits, where some tasks already provide learning signal, RLTL;DR generally neither helps nor hurts compared with GRPO, with an exception on LeetCode. The paper frames the approach as a way to train on especially difficult tasks, not as a replacement for standard RL across the board.
Four thousand task-and-insight examples
In a further experiment, the authors simplify the method to SFTL;DR. That version trains only on task-and-insight pairs, without showing the model full attempts or backpropagating through them. Training on 4,000 such pairs recovered almost the full performance of RLTL;DR and conventional supervised fine-tuning on complete rollouts, according to the paper.
The result points to a compact training signal: a short reminder about what to keep in mind for a type of task. But the findings remain tied to the paper’s selected tasks, model and evaluation setup. The authors’ central measured result is that, on tasks with no success in 128 baseline attempts, RLTL;DR reached 12% to 13% Pass@1 at evaluation without an insight in context.