Language Models Learn to Improve Language Models

AI summary

Overview

• Research area: Automated AI research (AutoResearch), reinforcement learning for LLMs, and language-model development.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning for LLMs (DAPO/GRPO-style policy optimization), reward modeling, and language-model training pipelines.
• Scope: The paper introduces LM2LM, a framework that trains an LLM proposer with reinforcement learning to suggest effective architecture, objective, and optimization changes for language models, using a frozen LLM "research world model" as a cheap reward source that is periodically recalibrated with real experimental runs.

What This Paper Is About

Language models are increasingly used as agents that propose, implement, and run experiments to improve other language models, but deciding what to try (the paper calls this "research judgment") remains a bottleneck. Training a model to make better proposals with real experimental feedback is prohibitively expensive, since every proposal requires a full model-training run to score. The paper's goal is to learn this proposal-generation ability at scale using a surrogate predictor of experimental gain, while keeping that surrogate honest as the proposer's behavior changes during training.

Key Contributions

1. LM2LM framework. A method that trains an LLM proposer with DAPO (a GRPO-family RL algorithm) where rewards come from a frozen LLM acting as a research world model (RWM) that predicts a proposal's experimental gain relative to a reference configuration, given real experimental records in context.
2. Structured reward design for predicted gains. Rewards are normalized by a task-specific scale and clipped, then adjusted for proposal validity, repetition, and mechanism diversity, so that tasks with large gain scales or extreme predictions do not dominate policy optimization.
3. On-policy calibration. Instead of fixing the RWM's evidence, the method periodically executes the current policy's highest-rated proposals and adds their measured outcomes to the RWM's in-context records, targeting real execution exactly where RL is most likely to be misled by overestimation.
4. Empirical validation across eight environments. Real-execution evaluation on held-out tasks spanning four autoregressive families (OLMo-3, Qwen3, NanoChat, Marin) and four diffusion families (E2D2-SW, E2D2, MDM, BD3LM), plus a generalization test on a model family (Qwen3) held out entirely from training.

Main Findings

• Past experience helps, but only if used well. On autoregressive (AR) tasks, giving the base model 300 experimental records in context (Base-ICL) raises Avg@4 from −7.94 to 4.04; on diffusion tasks it rises from −4.18 to 4.03. In contrast, supervised fine-tuning (SFT) reaches −12.19 (AR) and −3.09 (diffusion), and rejection-sampling fine-tuning (RFT-Real) reaches −22.95 and −3.72, so fine-tuning on a few hundred past proposals does not consistently help.
• LM2LM leads on both environment types. It achieves Avg@4 / Best@4 of 17.06 / 23.40 on AR tasks and 7.10 / 9.93 on diffusion tasks, beating Base on 93.3% and 100.0% of held-out states respectively. For comparison, GPT-6-sol reaches 6.51 / 8.79 (AR) and 1.98 / 4.43 (diffusion), and GPT-6-sol-ICL reaches 6.15 / 11.27 and 4.94 / 5.93.
• Fewer harmful proposals, and larger upside. LM2LM cuts the negative-gain rate from 70.4% to 39.2% on AR tasks and from 59.7% to 7.6% on diffusion tasks. On AR tasks it has a higher negative rate than GPT-6-sol-ICL (39.2% vs. 29.6%) but a much higher Avg@4 (17.06 vs. 6.15), which the authors read as evidence it pursues higher-gain proposals rather than just avoiding failures.
• Lower data-generation cost. At step 96, LM2LM matches ICL with 200 in-context records at comparable cost ($40.1k vs. $39.5k) while outperforming ICL with 300 records by roughly 33% lower data-generation cost and time ($40.1k vs. $60.0k; 343 vs. 510 hours). Including RL training, total cost is still lower ($56.1k vs. $60.0k).
• Cheap improvement over training. Between steps 40 and 96, Avg@4 rises from 10.85 to 17.06 (AR) and from 2.28 to 7.10 (diffusion) for only $11.8k in additional data-generation cost, covering over 14k sampled proposals with just 69 additional calibration records.
• Generalizes to an unseen model family. With all Qwen3 records excluded from training and RWM grounding, LM2LM reaches Avg@4 / Best@4 of 53.72 / 86.69, versus 21.63 / 25.27 for the strongest baseline, GPT-6-sol-ICL. Out-of-family in-context records alone give little: Base-ICL 2.97 vs. Base 3.53, and GPT-6-sol-ICL 21.63 vs. GPT-6-sol 20.19.
• Calibration beats simply adding records. With the same budget of 200 real runs, on-policy calibration reaches 17.06 Avg@4 versus 14.76 for a static record set augmented with 100 random records, and wins on 74.2% and 70.0% of held-out AR states against the two static variants.
• Proposal selection matters. Replacing top-rated selection with random selection of valid proposals drops AR Avg@4 / Best@4 from 17.06 / 23.40 to 10.84 / 18.60.
• RWM feedback carries real signal. On held-out AR states, LM2LM maintains group-relative advantage cosine similarity of 0.50–0.71 and within-state pairwise ranking accuracy of 0.61–0.69 (chance being 0 and 0.5).
• Without calibration, feedback degrades below chance. The static RWM's advantage cosine falls from 0.69 at step 16 to −0.42 at step 128, and its pairwise accuracy falls from 0.72 to 0.37, while LM2LM holds 0.61. Scoring LM2LM's own proposals with the initial record set instead of calibrated records lowers pairwise accuracy from 0.61 to 0.46 at step 96.
• Calibration limits overestimation. The static RWM's predicted–real reward gap climbs to about 2.0 at step 96, whereas LM2LM's stays below 1 and falls from 0.85 at step 96 to 0.17 at step 128 after a fourth calibration.
• Concrete overestimation case. The initial RWM overestimates a NanoChat optimizer change (predicted +0.7 to +1.25 vs. real −1.46), and the static variant's share of NanoChat training proposals rises from 30% to 79% by step 64; after LM2LM's first calibration adds the real outcome, the share falls to 8%.

Methodology in Plain English

The setup treats model development as a repeated loop. A proposer (the model being trained) looks at a task — a model family, size, training budget, and a reference recipe — plus a history of previously tried proposals and their measured outcomes, and suggests the next change. A separate builder model turns the suggestion into an executable modification, and a training run measures the resulting validation loss. The "gain" is the reference loss minus the new loss, so positive means improvement.

The obstacle is that scoring a proposal this way is expensive. The authors estimate that training a 486M-parameter model on 8B tokens takes about 45 B200 GPU-hours, and a GRPO-style RL setup with 100 updates, 32 states per update, and 8 proposals per state would need 25,600 evaluations — roughly 1,152,000 GPU-hours and over $16 million.

Their fix is a research world model: a frozen LLM that reads a task, a proposal, and a set of real experimental records provided in its context, and predicts the gain. That turns scoring into an LLM call rather than a training run. Predicted gains are converted into rewards by (a) dividing by a task-specific scale τ_e defined as the max of the reference-run standard deviation and 1.4826 times the median absolute deviation of predicted gains, (b) clipping to ±6, (c) penalizing repeated proposals below all valid ones, assigning −6 to invalid ones (and −3 to proposals whose modifications are all inert), and (d) adding a diversity bonus favoring less common "mechanism directions" within a group of proposals.

Because RL will drift toward whatever the world model overrates, the authors add on-policy calibration. At three points during training (after steps 16, 40, and 64), they take the current proposer's samples, pick the ones the world model rates highest, actually run them, and append the measured gains to the world model's in-context records. The world model never changes weights — only its evidence changes. This costs a fixed number of real runs (200 by default) regardless of how many proposals RL samples. The same predicted records are also recycled to expand the pool of RL training states.

The main experiment trains Qwen3.8-27B as the proposer with DAPO for 96 steps, using GPT-6-sol as the world model and GPT-6-astra as the builder. Training data comes from 40 training tasks with 300 curated and executed proposals, 100 of which seed the world model; evaluation uses 60 AR and 18 diffusion held-out research states built only from real histories, with four proposals generated per state and executed for real.

Why This Matters

The paper reframes "research taste" as something that can be learned from feedback rather than only prompted, and shows a practical way to get scalable training signal when the ground-truth reward costs a full training run. The calibration mechanism is the distinctive idea: rather than making the surrogate more accurate everywhere, it spends a small, fixed execution budget on exactly the proposals RL is about to reinforce. The finding that a fixed-record world model's ranking accuracy eventually falls below chance (0.37 vs. 0.5) is a concrete warning for anyone using frozen LLM judges as reward models in a shifting policy regime.

Real-world applications:

• Automated hyperparameter and architecture search. Pointing a learned proposer at training-recipe decisions instead of exhaustive or random sweeps, using cheap predicted scores to triage candidates before spending GPU hours.
• Lower-cost LLM judge and reward-model deployment. The calibration result applies to any pipeline where a frozen model scores outputs for a policy that keeps changing, such as synthetic-data filtering or RLHF reward modeling.
• Research-agent tooling. Teams building agents that run experiments can use the world-model-plus-calibration pattern to rank candidate ideas before committing compute.
• Diffusion and non-AR model development. The method is demonstrated on both autoregressive and diffusion families (E2D2-SW, E2D2, MDM, BD3LM), suggesting portability across model classes.

Industry relevance: the reported cost accounting ($14.208 per B200 GPU-hour, $40.1k vs. $60.0k for comparable or better results) speaks directly to organizations deciding whether to invest in learned research agents versus simply stuffing more experimental history into a frontier model's context. The result that a 27B open-backbone model trained with LM2LM outperforms GPT-6-sol — even when GPT-6-sol is given 300 real records in context — is a strong argument for training over prompting in this setting.

Future Directions

• From proposals to campaigns. The authors explicitly note that the work targets proposal-level improvements rather than end-to-end research campaigns, and suggest evaluating the framework in longer, end-to-end settings.
• Broader research environments. Extending beyond pretraining-recipe choices to areas such as LM post-training is named as future work.
• Reducing residual real execution. Calibration still requires additional model-training runs; making the world model's predictions reliable enough to shrink that budget further, or to retrain rather than re-prompt it, remains open.
• Safety and oversight of self-improving loops. The ethics statement raises unreliable experimental conclusions, increased compute consumption, and misuse of automated model-improvement capabilities as risks, and calls for human oversight and independent evaluation as such systems are applied to more capable models.

Target Audience

Researchers and engineers working on LLM-based research agents, RL for LLMs, reward modeling and LLM-as-judge systems, and automated machine-learning engineering. It is also relevant to practitioners running architecture or training-recipe search who want to cut the number of full training runs, and to readers interested in the technical prerequisites for recursive self-improvement. The paper is not beginner-friendly: it assumes comfort with policy-gradient objectives, group-relative advantages, and the mechanics of language-model pretraining.

Authors’ abstract

Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.