ParetoRSI-Gym: Can AI Agents Improve Language Models Across Multiple Domains?

AI summary

Overview

Research area: Machine learning — evaluation of LLM agents for automated post-training data curation and recursive self-improvement (RSI).

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, LoRA, benchmark contamination, and agent harnesses.

Scope: A benchmark (ParetoRSI-Gym) and empirical study of whether agentic systems can iteratively curate training data that improves a language model across five domains at once, measured on held-out generalization benchmarks the agents never see.

What This Paper Is About

Existing benchmarks test whether an AI agent can push up a single score on a single benchmark, often by picking from a fixed data pool or distilling a stronger model. Real post-training instead targets several capabilities at once, where a gain in one domain can cost you in another. The authors build ParetoRSI-Gym, a five-round loop in which an agent searches HuggingFace for data, curates a training pool, picks training hyperparameters, reads back per-domain scores, and revises its written principles — with the starting model, the five domains, the toolset, and the SFT pipeline all held fixed — and then ask whether the resulting models actually generalize beyond the feedback the agent optimized against.

Key Contributions

1. Multi-domain RSI benchmark. ParetoRSI-Gym evaluates iterative, feedback-driven post-training data curation across five domains (math, code, finance, medicine, science) under bounded per-round budgets, with persistent cross-round state, dynamic HuggingFace data sources, feedback-driven revision, and a multi-domain objective.
2. Generalization-aware evaluation. The authors separate proxy benchmarks used as agent feedback (MATH500 ID, GPQA ID, MedBullets ID, SuperGPQA Finance ID, MBPP+) from held-out generalization benchmarks (MATH500 OOD, GPQA OOD, MedBullets OOD, SuperGPQA Finance OOD, HumanEval+), and add rejection criteria that screen training entries for overlap with evaluation data.
3. Systematic empirical study. Nine main curation runs across three agent systems (Claude Code with Opus-5, Codex with GPT-5.6 Sol, CheetahClaw with deepseek-v4-flash) and three base models (Qwen3-4B-Base, Ministral-3-3B-Base-2512, Olmo-3-1025-7B), reporting both the final-round checkpoint and the proxy-best checkpoint.
4. Diagnostic analyses. A fixed-backbone harness comparison, a round-by-round trend analysis of proxy versus generalization gains, and a dataset-level analysis of what each agent selects, how much runs overlap, and when exploration stops.

Main Findings

• The strongest system gives a modest but real gain. At its proxy-best checkpoint, Claude Code with Opus-5 scores 9.2% higher in relative terms (+2.3 absolute points) than the starting model, averaged with equal weight over three base models and five domains.
• Ranking across agents. Proxy-best average gains are +2.3 points for Claude Code, +0.6 for Codex, and −0.3 for CheetahClaw; the corresponding relative figures, averaged over the fifteen base-model/domain values, are +9.2%, +3.1%, and +2.9%. The authors note the relative figure weights low-baseline cells heavily, so they report both.
• Large single-domain gains can be partly formatting. The largest relative gain is 63% on medicine with Ministral-3-3B (23.65 → 38.46); the authors caution that this base model often does not end answers with an option letter, so part of the gain is learning the answer format rather than medical knowledge.
• Gains are uneven across domains. At the proxy-best round, the largest increase is 14.8 absolute points, concentrated in medical and finance; averaged over runs the agents improve medical (4.3 absolute points) and code (3.2 absolute points), while math tends to degrade.
• Base-model dependence. Agents improve Ministral-3-3B more than Qwen3-4B-Base and Olmo-3-1025-7B, indicating a strong model-dependent effect.
• Agents peak before they stop. In seven of the nine configurations the proxy-best round is not the last round, and 24 of 45 domain settings show a drop relative to the round with the highest proxy gain — later exploration can undo earlier gains.
• One checkpoint for five domains is costly. In 43 of 45 model–domain cells the best round for a domain is not the round the agent selects, at an average cost of 2.0 points and up to 6.9.
• The harness matters as much as the model. Holding the backbone fixed at DeepSeek V4.1 Flash Max and varying only the harness, results range from +4.3 points (Codex) to +2.6 (Claude Code) to +0.2 (CheetahClaw); coding shows the largest harness-driven gain, 13.2 points with Codex.
• Proxy and generalization signals move together. In the round-by-round study of Qwen3-4B-Base with Codex and GPT-5.6 Sol, proxy gains move in the same direction as held-out gains, though not by the same amount: on code the proxy gains stay within 3 points of the starting model while generalization gains reach 11 points.
• Each agent has a stable data "taste." Two runs of the same agent overlap at 0.24 Jaccard on their dataset lists versus 0.07 for different agents; overlap does not rise when the runs also share a base model (0.23 within a base model versus 0.25 across base models), suggesting selection strategy is set by the agent rather than adapted to the target model.
• Exploration decays over the run. Fewer and fewer datasets enter the pool for the first time in later rounds; individual runs can be extremely lopsided, with one Codex run devoting its entire final pool to code and one CheetahClaw run giving medical over 80% of two rounds.
• Distinct restart strategies. Claude Code and Codex continue from an earlier checkpoint in 64% and 53% of their rounds, while CheetahClaw starts from scratch in 67% of rounds and never branches back.

Methodology in Plain English

The setup is a five-round loop with everything except the data held constant. Starting from a fixed base model, the agent reads its current per-domain proxy scores and the item-level evaluation trace (including its own wrong answers), writes or revises a set of written "principles" that persist across rounds, and then uses three operations on the HuggingFace Hub — search, select, pull — to build a training pool. It chooses which checkpoint in the model pool to start from, sets training hyperparameters (learning rate, LoRA rank, batch size, sequence length, warmup) within fixed ranges, and submits the pool for LoRA SFT with AdamW for one epoch. Each round is capped at 15M training tokens and 40k entries. The harness then trains and scores the new model on the five proxy benchmarks and returns the full per-domain vector plus per-item results.

Two safeguards keep the evaluation honest. First, training entries that overlap the evaluation data are rejected at registration, and searches containing proxy text are logged as violations. Second, the generalization suite is never present in the agent's sandbox — network access is restricted to HuggingFace — and is scored only once, after the run. The authors report two checkpoints per run: the final-round model and the proxy-best model (the round with the highest average proxy score), plus an oracle "domain best" row that takes each domain's best round and is explicitly not a deployable model. Claude Code and Codex configurations are repeated with three training seeds and averaged; CheetahClaw is run once per base model.

Why This Matters

Impact on research. The paper argues that single-benchmark RSI evaluations overstate progress, and offers a protocol that separates genuine capability gains from fitting the feedback signal. Its finding that harness design can outweigh backbone choice, and that agents stop exploring new data sources as the horizon lengthens, gives concrete failure modes for future agentic post-training work.

Real-world applications:

• Automated domain adaptation of open base models for regulated fields such as medicine and finance, where multiple capabilities must be preserved simultaneously.
• Reducing the human data-curation burden in model releases, where teams report per-capability results rather than one aggregate score.
• Auditing agentic data pipelines for contamination and reward hacking before deployment.
• Comparing agent scaffolds (harnesses) for ML engineering tasks, independent of which model drives them.

Industry relevance. Companies that ship models trained on multi-domain mixtures care about cross-domain trade-offs, and the paper's results — small average gains, uneven per-domain outcomes, and suboptimal returned checkpoints — set a realistic bar for what autonomous post-training can currently deliver. The code is released at github.com/haojinw0027/ParetoRSI-GYM.

Future Directions

• Scaling the study. The authors state they experiment with only three agent systems and three base models; larger models and more harnesses remain untested.
• Making results durable over time. The three agent systems are "moving targets," so the reported numbers are a snapshot of the harness and backbone versions available at the time of writing.
• Broadening domains and languages. The benchmark covers five domains and only English text; other languages and domains are left open.
• Fixing the exploration and checkpoint-selection failures. Since agents peak before their final round and rarely pick the best round per domain, a natural next question is how to give agents the ability to hold gains while exploring, and to return the right checkpoint.

Target Audience

Researchers and engineers working on agentic systems, automated ML engineering, data curation, and recursive self-improvement; practitioners who post-train open models on multi-domain mixtures and need per-capability evaluation; and benchmark designers interested in separating proxy feedback from held-out generalization. Readers should be comfortable with SFT, LoRA, and standard LLM evaluation terminology.

Authors’ abstract

Recursive self-improvement (RSI) requires an agentic system to design, train, and evaluate its own successor. Data-driven post-training is a central part of that loop, yet existing evaluations measure it on a single benchmark, where an agent selects from a static pool or distills a stronger model to push one score up in one run. Real post-training is multi-domain, and a gain in one domain has to be weighed against a regression in another. We propose ParetoRSI-Gym, in which an agent searches for data, curates it, trains, reads back per-domain results, and repeats, with everything but the data held fixed: the initial LLM, the five domains (math, code, finance, medicine, science), the toolset, and the SFT pipeline. Generalization benchmarks in each domain separate capability gains from overfitting to the scores the agent optimizes against. Across three agent systems, three base models, five rounds and multiple seeds, the strongest system, Claude Code with Opus-5, raises the generalization scores of its proxy-best checkpoint by 9.2% relative to the starting model (+2.3 absolute points), averaged with equal weight over three base models and five domains, with a single-domain relative gain of up to 63% (medicine on Ministral-3-3B). Further analysis shows that the harness matters as much as the model it drives, and that each agent keeps a unique data selection strategy. We see our work as a step towards evaluation of RSI systems in real-world machine learning engineering.