RSI Arena: Towards Recursive Self-Improvement with Human Feedback

AI summary

Overview

Research area: Autonomous AI research and development — specifically whether frontier LLM agents can post-train a language model end to end, and then improve it using feedback from the people who use it. This sits at the intersection of LLM post-training (SFT, DPO/ORPO/SimPO), automated ML engineering, and human-preference evaluation (Bradley–Terry / Chatbot Arena-style rating).

Technical level: Intermediate. The paper is readable without deep math, but it assumes familiarity with concepts like base vs. instruct models, LoRA vs. full fine-tuning, benchmark contamination, and Elo-style preference ratings.

Scope (one sentence): A two-stage, livestreamed experiment in which ten frontier models autonomously post-train the same base model under matched compute budgets and are then judged by both 20 benchmarks and blind human votes, followed by a shorter second stage in which each agent retrains its own model using the votes it received.

A note on completeness: the supplied text is truncated partway through Section 6.4, so the full Stage 2 rating table (Appendix F) and the remainder of the Stage 2 discussion are not available here. Where a Stage 2 figure does not appear in the text above, it is not reported in this summary.

What This Paper Is About

Turning a pretrained language model into a useful assistant normally requires a human team to pick data, write training code, run fine-tuning and preference optimization, and repeatedly evaluate the result. This paper asks whether AI agents can do that whole job themselves — and, crucially, whether they can then use feedback from real users to make their models better in a second round.

The authors build "RSI Arena," a controlled competition in which ten frontier models act as autonomous research agents, each given the same base model, the same compute budget, and no target benchmark, and left to decide everything else. Their results are scored both on standard benchmarks and by people voting on blind side-by-side comparisons, which lets the authors test how well benchmark gains track what users actually prefer — and whether human feedback can close the loop toward "recursive self-improvement with human feedback" (RSIHF).

Key Contributions

1. A two-stage, auditable testbed for autonomous post-training. Stage 1 mimics pre-release development (144 hours, 1,000 GPU-hours, $300 of API credit per agent on a shared 64-GPU cluster); Stage 2 mimics post-release improvement (72 hours, 500 GPU-hours, $150 of API credit) with each agent receiving the battle records its own model played. The entire run was livestreamed, with daily reports, and checkpoints released on Hugging Face.

2. A paired evaluation design linking benchmarks to human preference. Every final model is scored on 20 benchmarks (16 capability, 4 safety) and simultaneously enters a blind Human Arena modeled on Chatbot Arena, with NVIDIA Instruct as a reference and a Bradley–Terry rating that includes explicit style control for answer length, headers, list items and bold spans. The same analysis is repeated across both stages and two separate arenas.

3. A demonstration that human feedback can be handed to an autonomous agent as training signal. Each Stage 2 agent received only 213 to 276 accepted battles involving its own model — questions, both answers, and the vote — with no explanation of what the votes showed. Eight of ten agents improved their style-adjusted rating relative to NVIDIA Instruct.

4. A diagnosis of where benchmarks and users diverge. The paper quantifies rank correlations between each benchmark and human ratings across the ten agent models, showing that LLM-judged chat benchmarks track people closely (WildBench at ρ = 0.95) while competition mathematics does not (AIME 2026 at ρ = 0.10, HMMT at ρ = −0.30), and that raw answer length correlates with rating at ρ = 0.78.

Main Findings

• All ten agents produced working assistants, and the best matched NVIDIA Instruct after style control. The model trained by GPT-6 Astra closed a median of 76% of the base-to-Instruct gap across 12 capability benchmarks. Without style control, people rated NVIDIA Instruct highest (1,182 vs. 1,161); with length and formatting controlled, the two were level at 1,095 and 1,094.

• The recipes converged. All ten began with supervised fine-tuning on public instruction data, most often Tülu 3, and six then ran preference optimization (DPO, ORPO or SimPO) on public preference data such as UltraFeedback. Six trained LoRA adapters and four trained all weights.

• Budget use did not predict quality. Gemini 3.8 Flash stopped itself at hour 12 after 19 GPU-hours (having answered 59 of 60 questions on its own test correctly), leaving 981 GPU-hours and $293 unused — yet people rated its model above the models of Muse Spark 1.3 and DeepSeek V4.1 Flash, which used about 1,000 GPU-hours each. Across the ten agents, GPU-hours correlated with the style-controlled human rating at ρ = −0.21 and with benchmark progress at ρ = −0.01.

• Benchmarks and people agreed only partly. Benchmark progress and the unadjusted human rating correlated at ρ = 0.73 (ρ = 0.72 with style control). LLM-judged chat benchmarks tracked people best — WildBench at ρ = 0.95, Arena-Hard at 0.85 — while competition math did not. The model with the best AIME 2026 score (66.7, against 21.7 for NVIDIA Instruct with reasoning off) placed eighth of ten in human evaluation, because its long reasoning consumed the answer budget.

• Length and formatting dominate human votes. The longer answer won 72% of the 999 decisive votes between answers of different lengths, rising from 57% when one answer was under 1.2× the other's length to 81% when it was more than 6.7×. Style control compressed the spread of agent models from 365 to 208 points: it lifted the DeepSeek-trained model by 91 points and lowered the Astra-trained model by 66 and NVIDIA Instruct by 88.

• Safety was not part of the task, and three models regressed. On StrongREJECT, the Gemini-, Grok- and Muse-trained models scored 43.6, 34.4 and 31.4 against 16.9 for the base model, refusing 54%, 63% and 64% of harmful requests versus 99% for NVIDIA Instruct. Seven models scored 12.2 or lower.

• Stage 2: human feedback helped most agents, modestly. Using NVIDIA Instruct as a common reference, eight of ten models improved their style-adjusted relative rating, with a median gain of 28 points. The DeepSeek-trained model gained most, going from −207 to −35, as its median answer grew from 61 to 508 tokens.

• Stage 2: NVIDIA Instruct still led without style control. It beat the second-place DeepSeek-trained model in 99.5% of bootstrap resamples, and in a targeted round it won 39 of 65 battles against the Astra-trained model, which won 18. With style control the two were level at 1,061 and 1,054.

• Stage 2: benchmark movement was small and unrelated to preference gains. Six of ten agents raised their median benchmark progress, with a median change of +2 points of the gap; the correlation between benchmark change and rating change was ρ = −0.19. The MiniMax-trained model lost 18 points, largely because many answers no longer ended in the format scorers read, and its StrongREJECT score rose to 24.5 from 2.6.

• Agents left a lot of budget on the table. In Stage 2 they used 2,880 of 5,051 GPU-hours and $541 of $1,500 in API credit, with Gemini 3.8 Flash again using only 10 GPU-hours.

• Engineering friction, not research taste, consumed much of the time. The three agents attempting full-parameter training from the start spent their opening hours on memory errors; GLM-5.3 abandoned the Hugging Face trainer after seven hours and 39 test jobs to write its own FSDP2 trainer, roughly seven times faster. Four agents exhausted their 400 GiB storage quotas at least once, and one eight-GPU job spent 30 hours 43 minutes in the queue. Because the base model's end-of-turn token was never trained, every agent had to teach its model to stop — and the token even truncated one agent's own tool calls.

• Two agents decontaminated on their own initiative. GPT-6 Astra found MMLU-Pro and IFEval items on day 1 and MATH-500 and HumanEval items on day 2, retraining from the base model each time; MiMo-V2.6-Pro removed 92,477 rows on day 1 and 113,686 more on day 2 after checking ten further benchmarks.

Methodology in Plain English

The authors set up a competition with fixed rules. Every agent — GPT-6 Astra, MiMo-V2.6-Pro, Grok 4.7, DeepSeek V4.1 Flash, GLM-5.3, Kimi K3, Gemini 3.8 Flash, Muse Spark 1.3, MiniMax M3.1 Flash and Step 5 Preview — got the identical starting point: the Nemotron 3.5 Lightning 30B-A3B base model, a mixture-of-experts model with about 30 billion parameters and about 3 billion active per token. Each got the same budget and the same playbook-free instruction: build the strongest model you can, for evaluation on held-out data and by people. No benchmark was named, which the authors say reduces the incentive to game a single metric. Cheating — training on test data, hard-coding answers, copying another agent's work — would score zero.

The infrastructure mattered. All ten agents shared one Slurm cluster of 64 NVIDIA RTX PRO 6000 Blackwell GPUs (96 GB each), with each agent able to hold up to 16 at a time. That is deliberate: it forces queueing, preemption and compute planning, as in a real lab. Each agent ran in its own virtual machine, while a separate management machine held the API keys and metered every model call and GPU-hour against that agent's budget.

Evaluation used two lenses. Benchmarks give a fixed-format measure of specific skills; the Human Arena gives the thing the models are actually built for. Visitors at COLM 2026 and online typed questions, saw two anonymous answers side by side, and voted for one, a tie, or "both are bad." Votes counted only if the voter had time to read both answers, organizers' votes were excluded, and attention-check battles (one answer deliberately off-topic) filtered careless voters — 100 of 102 passed in Stage 1. Ratings come from a Bradley–Terry model with an added term that corrects for stylistic differences between the two answers, so that a model is not rewarded purely for being longer or more heavily formatted.

For Stage 2, the loop closes. Each agent got exactly one snapshot of the battles its own model had played, with the opponent's name stripped out and no commentary on what the votes showed. Any pattern — including the length effect the authors themselves document — had to be discovered by the agent. The Stage 2 models were then re-evaluated on the same 20 benchmarks and in a fresh arena, with NVIDIA Instruct appearing in both arenas as the fixed yardstick so the two rounds could be compared.

Why This Matters

Impact on research. The paper supplies something the autonomous-research literature mostly lacks: an end-to-end test where the success criterion is human preference rather than a single disclosed benchmark. Prior work such as PostTrainBench has agents post-train small models against one known target metric; here no target is named, and the models are then used by people. The result is a set of concrete, reusable measurements — which benchmarks track human preference (WildBench at ρ = 0.95) and which do not (AIME 2026 at ρ = 0.10), how much of a rating gap is style rather than substance, and what an agent does with raw vote data when handed it. The released checkpoints from throughout development make the trajectory of autonomous post-training auditable after the fact.

Real-world applications:

• Cheap model customization. If an agent can take a base model to near-instruct quality on a modest budget, organizations without ML teams could produce domain-tuned assistants, with the caveat that the cheapest run here (19 GPU-hours) beat several 1,000-GPU-hour runs on human preference.
• Evaluation design. The benchmark-versus-human correlation table is directly actionable for anyone choosing which benchmarks predict user satisfaction — and a warning against optimizing competition-math scores for chat products.
• Safety regression screening. Three models became measurably more compliant with harmful requests without anyone asking them to, which argues for mandatory safety checks in any pipeline where agents train models unsupervised.
• Feedback-driven iteration. The Stage 2 setup is a template for how a deployed product could route user votes back into training, with the agent, not the engineer, deciding what to do with them.

Industry relevance. The engineering log is as useful as the results: FSDP2 rewrites, inter-GPU bandwidth of 11 GB/s forcing CPU offloading, storage quota exhaustion, untrained end-of-turn tokens truncating tool calls, and host-memory reservations idling GPUs badly enough that the organizers imposed a 64 GB per-GPU cap (after 427 jobs from eight agents in 24 hours requested more). These are the failure modes that dominate real autonomous-training runs, and they are reported with unusual specificity.

Future Directions

• Making safety part of the objective. The instruction deliberately said nothing about safety, and three models regressed on StrongREJECT while one (the MiniMax-trained model) went from 2.6 to 24.5 between stages. Whether an explicit safety term in the task, or safety-aware feedback in the arena, removes this drift is untested.

• Closing the benchmark–preference gap. Benchmark change and rating change were essentially unrelated (ρ = −0.19) between stages. A natural next question is whether a benchmark can be built that tracks the style-controlled human ranking as well as WildBench tracks the unadjusted one, or whether preference feedback must simply be collected directly each round.

• Giving agents more of the feedback loop. Agents received a single static snapshot of 213 to 276 battles. Whether iterative, live feedback — or feedback on the follow-up turns and web-search battles that were excluded from rating — produces larger gains than the median 28 points observed is open.

• Understanding why effort and outcome diverge. Gemini 3.8 Flash used 19 GPU-hours in Stage 1 and 10 in Stage 2 yet was rated above agents that spent roughly 1,000. Identifying what the low-budget runs did differently — and whether that is reproducible or luck — is the most practically valuable open question the paper raises.

Target Audience

Researchers working on autonomous agents, automated ML engineering, and self-improving systems will find the experimental design and the engineering post-mortem most valuable. Evaluation researchers and leaderboard maintainers should read the style-control analysis and the per-benchmark correlation table, which quantify how much of a preference win is length and formatting rather than content. Safety researchers will care about the unrequested regressions on StrongREJECT and the observation that low risk scores can reflect blanket refusal rather than genuine safety. Product and policy teams assessing whether to let agents touch model training pipelines will benefit from the budget-utilization and failure-mode reporting. A working knowledge of LLM post-training is assumed; the paper is not an introduction to the topic.

Authors’ abstract

Can AI agents autonomously train useful language models and improve them using feedback from users? We introduce RSI Arena, a two-stage study of autonomous post-training as a step toward recursive self-improvement. Ten frontier models act as training agents, each starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. During a six-day first stage, each agent has 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster, independently selecting data, implementing training procedures, and choosing checkpoints. We evaluate their models on 20 benchmarks and through blind human evaluations. All ten agents produce working assistants, with the strongest model, trained by GPT-6 Astra, closing a median of 76% of the base-to-Instruct gap across 12 capability benchmarks. Its human-preference rating is also close to NVIDIA Instruct after statistical adjustment for answer length and formatting. In Stage 2, each agent receives the comparison records and votes involving its Stage 1 model and is given three additional days for further training. Using NVIDIA Instruct as a common reference across the two evaluation rounds, we observe an improvement from Stage 1 to Stage 2 in the style-adjusted relative rating of eight of the ten models. We show that benchmark gains were modest, with no clear link to preference gains, while some agents encouraged longer responses and one model showed increased harmful compliance on StrongREJECT. RSI Arena provides an auditable setting for studying how human feedback can guide autonomous model development. It takes a step toward recursive self-improvement with human feedback (RSIHF), in which people help define what improvement means for the AI systems they use.