The Pulse
Small Test Sets Made LLM Self-Improvement Look 13–20 Points Better
A pre-registered study found that LLM self-improvement loops using 16 evaluation items overstated final accuracy by 13 to 20 percentage points. The researchers also tested larger evaluation sets and acceptance rules, finding that neither pr
AI.info Team · Written with AI ·
With only 16 evaluation examples, LLM self-improvement loops overstated their final accuracy by 13 to 20 percentage points, according to a pre-registered study posted to arXiv on October 7. When researchers used 256 examples, the gap fell to 1 to 5 points. The results show how a system can appear to improve when it repeatedly chooses whichever prompt performs best on the same small test set.
The study examines loops that ask a model to revise instructions, score the revisions on a selection set and keep the best-scoring change. Litao Hu of Meta and Yutong Tang of Microsoft compared those selection scores with accuracy on separate, held-out examples. “Self-improving LLM systems propose changes to themselves and keep those that score better on a small evaluation set,” the authors write in the paper’s abstract.
“Self-improving LLM systems propose changes to themselves and keep those that score better on a small evaluation set.”
Litao Hu and Yutong Tang, authors of the paper
Sixteen examples amplified the winner’s score
The statistical problem is selection under noise: test scores vary, and a loop that keeps the highest-scoring candidate tends to select one that benefited from random fluctuations. Reusing the same examples compounds the issue because later edits are judged against an incumbent already chosen partly for scoring well on that set. The measured score can rise even when the revised instruction performs no better on new examples.
In the confirmatory experiments, the researchers tested Qwen2.5 models on TREC question classification, Banking77 intent classification and GSM8K math problems. They compared selection sets of 16, 64 or 256 examples with a separate 600-item held-out set. After 11 to 20 generations, the selection-set score exceeded held-out accuracy by an average of 13.6 points with 16 items, 7.2 points with 64 and 1.7 points with 256.
Larger tests helped on TREC, not every task
The larger selection set also improved held-out results on TREC. For the 1.5-billion-parameter Qwen2.5 model, average held-out gains over the starting instruction rose from 13.7 points with 16 examples to 24.2 points with 256. For the 7-billion-parameter model, they rose from 14.0 to 20.4 points.
That pattern did not hold on GSM8K: the researchers found no increase in held-out gains as the selection set grew, and no experimental setup gained more than 3.1 points. On Banking77, the 16-item greedy loop ended 0.6 points below its starting instruction on average, while its selection-set score rose 12.5 points. Four of eight runs lost held-out accuracy; the paper says an earlier exploratory result showing losses on that task did not replicate in the confirmatory study.
Stricter acceptance rules did not fix whole runs
The researchers also tested a pilot-calibrated Bayesian gate and a select-then-confirm rule, which checks a chosen candidate on examples not used to select it. Neither improved complete runs over the basic greedy approach when the same selection set was reused. Across 32 paired runs, the Bayesian gate finished 2.1 points below greedy acceptance on average, while select-then-confirm finished 1.6 points below it.
Additional tests extended the concern beyond the researchers’ own prompt-rewriting loop. In GEPA, a prompt optimizer, the validation score for a returned program based on 16 items suggested a 38.8-point TREC improvement, while the held-out gain was 19.2 points. The authors stress that this measures optimism in the validation score, not an error in GEPA’s published results, which use separate test sets.
The study leaves agent and code tasks untested
The experiments focus on instruction edits, Qwen models and tasks with deterministic scoring. The authors did not test code-writing or agent tasks, where outcomes can involve stochastic rollouts and additional measurement noise. They also caution that the acceptance rules they tested are not a general solution to repeated use of evaluation data.
For a practical check, the researchers scored the starting and final instructions on 64 examples never used for selection. That audit removed the average bias in reported gains and reduced the root-mean-square error from 15.5 to 5.7 percentage points, though the remaining error could still match a typical gain. Their recommendation is to report held-out accuracy and its standard error alongside the score used to select changes.