RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environments

AI summary

Overview

Research area: Autonomous digital agents / agent-level recursive self-improvement and memory construction, with evaluations on computer-use and game-development benchmarks.

Technical level: Advanced. The paper assumes familiarity with multi-agent harnesses, code-as-policy action spaces, benchmark scoring (partial score and binary accuracy), and context/memory management.

Scope: A training-free, three-role multi-agent framework (curriculum, actor, verifier) that autonomously explores a new software environment through a broad-then-deep strategy, freezes the resulting memory, and reuses it for downstream tasks without updating model parameters.

What This Paper Is About

Digital agents often must work in new environments whose interfaces, tools, conventions, and failure modes are not fully captured by pretrained knowledge, and existing adaptation approaches usually require collecting additional interaction data, often with human assistance. RSIAgent instead lets an agent autonomously explore a new environment, verify outcomes against environment feedback, and consolidate reusable causal knowledge into a persistent memory that is frozen and reused for downstream tasks. The goal is recursive self-improvement at the agent level — narrowing or reversing the gap between open- and closed-source foundation models without changing model weights.

Key Contributions

1. RSIAgent framework: A general multi-agent framework for recursive self-improvement that autonomously acquires, verifies, and reuses knowledge in new environments without updating model parameters.
2. Broad-then-deep exploration: Two complementary strategies — Broad Recursive Self-exploration (BRS) for diverse environment knowledge and Deep Recursive Self-exploration (DRS) for hard cases, hidden constraints, and boundary conditions.
3. Empirical demonstration of agent-level self-improvement: RSIAgent enables Kimi-K3 and GLM-5.3 to outperform frontier closed-source models on OSWorld 2.0 (0808 offline) and Agents' Last Exam (Near-term).
4. Verification-grounded memory design: Separate role contexts, canonical memory ownership, and parallel work with sequential memory updates, plus a documented analysis of failure modes.

Main Findings

• RSI improves the existing harness on both benchmarks. On OSWorld 2.0 (0808 offline, 82 tasks), partial score rises from 71.97 (RSIAgent w/o RSI) to 78.98, and binary accuracy from 37.80 to 42.68. On Agents' Last Exam (Near-term, 67 tasks), partial rises from 83.75 to 84.82 and binary from 49.25 to 50.75.
• Highest reported partial-credit scores in the compared table. RSIAgent reaches 78.98 on OSWorld 2.0 and 84.82 on ALE, exceeding GPT-6 Astra's reported scores by 6.38 and 2.56 percentage points respectively. GPT-6 Astra is listed at 72.60 partial (OSWorld) and 82.26 partial / 52.24 binary (ALE); its OSWorld binary is not reported. Claude Opus 5 is listed at 70.19 partial / 34.72 binary (OSWorld) and 79.54 / 46.27 (ALE), so RSIAgent scores above it on both benchmarks.
• Comparison context for open-source baselines. Kimi-K3 is listed at 58.30 partial on OSWorld 2.0 (binary not reported) and 71.60 / 40.30 on ALE; GPT-5.6 Sol at 64.13 / 28.10 and 78.82 / 47.76; Kimi-K2.6 at 22.10 / 4.60 and 21.70 / 9.20. Other listed entries (MiMo-V2.5, DeepSeek V4 Pro, Qwen3.8-Max, Claude Fable 5, Gemini-3.8-Flash, Muse Spark 1.3, Claude Opus 4.8) are reported on a subset of the two benchmarks.
• Breakthroughs are discrete, not smooth. On three OSWorld 2.0 tasks — T044 (video editing), T049 (presentation repair), T065 (railway booking) — scores are tracked across steps 0–8 with RSIAgent (w/o RSI) as step 0. By step 8 they reach 100%, 80%, and 100% respectively, with discrete jumps when accumulated memory covers a remaining bottleneck.
• Both exploration stages are needed. On four OSWorld 2.0 tasks (T080 WPS spreadsheet repair, T085 REAPER audio editing, T089 browser-based presentation repair, T106 3D Slicer liver segmentation), full RSI averages 74.54% versus 65.52% for broad-only and 56.50% for deep-only. Broad-only improves over baseline on every task; deep-only falls below baseline on T085 and T089.
• Generalization to game development. On 40 tasks sampled from GameCraft-Bench, RSIAgent improves overall quality for every base generator: Codex + GPT-5.5 (high) 52.77 baseline to 61.28; Kimi-K2.6 31.28 to 46.37; GLM-5.3-Flash 30.55 to 48.72; Qwen3.8-27B 41.30 to 57.46. Play2Code improves weaker generators but degrades the strongest base games (52.77 to 51.05 for Codex + GPT-5.5 (high)).
• Three recurring failure modes. Insufficiently targeted exploration leaves target-specific weaknesses unresolved; incomplete verification returns PASS despite unsupported field values or artifact discrepancies under the official rubric; unreliable memory consolidation turns inadequately verified decisions into reusable rules that later propagate.

Methodology in Plain English

The system splits work among three agents with separate contexts. A curriculum agent decides what to practice next, using the target query, past outcomes, and a disposable copy of memory; it cannot edit canonical memory. An actor agent executes code-based actions (Python or Bash programs), can request visual evidence with look, ask for missing information with ask, and submit done; it owns all durable memory updates. A verifier agent inspects the candidate against the original task requirements using reset-and-replay or checkpoint restoration, is isolated from the actor's private reasoning and memory, and returns a grounded PASS or FAIL (the target interface also supports UNVERIFIED).

Exploration runs in two stages. BRS generates multiple independent projects per wave, runs up to four concurrently in isolated environments against the same immutable pre-wave memory snapshot, then resumes the actor contexts one at a time to distill and reconcile each verified experience into the current canonical memory; the next wave is chosen only after those commits finish. DRS then works sequentially on the target: attempt the target, consolidate, let the curriculum decide whether more practice is worthwhile, run each practice project through actor-execution, verification, memory update, and curriculum review, then retry the target in a reset environment. Stopping is governed by a declared policy (curriculum_review by default, or verifier_pass); a STALLED decision allows one final target attempt before recording the actual verdict. Unresolved verification or infrastructure errors suspend advancement rather than being counted as task verdicts.

Before evaluation, the memory is frozen — its file-tree hash is recorded, curriculum decisions and host memory writeback are disabled — and an actor–verifier loop runs the target after an environment reset, with an external evaluator scoring the candidate outside all agent contexts.

Configuration details: GLM-5.3 is the default actor agent, while Kimi-K3 serves as both verifier agent and curriculum agent in a separate context. BRS uses a nominal budget of eight exploration projects with up to four concurrent; the budget is checked between completed waves without interrupting an ongoing wave. DRS proceeds sequentially until the curriculum agent judges no further useful practice is needed, and a successful practice does not automatically end exploration.

Why This Matters

Impact on research: The results argue that agent-level self-improvement — memory construction and verification, not weight updates — can measurably shift where open-source models stand relative to frontier closed-source models. The paper also contributes a concrete, documented failure-mode taxonomy (exploration targeting, verification completeness, memory consolidation reliability) that points at where recursive self-improvement pipelines break.

Real-world applications:

• Adapting agents to private or continuously changing enterprise software where collecting training data with human help is costly or infeasible.
• Desktop office workflows, as studied in the paper's own tasks: spreadsheet repair, presentation repair, audio and video editing, and booking interfaces.
• Autonomous game development, where agents repeatedly play, diagnose, and modify executable games, evaluated on GameCraft-Bench.
• Scientific software workflows, which the authors name as a future target domain requiring specialized tools, workflows, and procedures.

Industry relevance: Because the method is training-free and reusable memory is frozen for direct deployment, the cost profile is test-time exploration plus inference rather than model training. Companies building computer-use agents on open-source backbones could apply the approach without fine-tuning, though the authors note the additional computation cost and dependence on exploration budgets and stopping policies.

Future Directions

• Extending beyond computer-use environments: The authors plan to move recursive self-improvement to broader interactive domains, including AI for Science (specialized tools, workflows, scientific procedures) and games as long-horizon, continuously evolving testbeds.
• Cost and budget control: Recursive self-improvement requires additional test-time exploration and practice; how to bound this computation while retaining gains is left open.
• More reliable verification: A model-based verifier can produce incorrect judgments that propagate into later exploration and memory updates, so stronger grounding for verification decisions remains a problem.
• Better stopping policies and memory quality: Performance depends on finite exploration budgets, stopping policies, and the quality of the learned memory, and the authors state that current experiments do not fully isolate the contribution of every component.

Target Audience

Researchers and engineers working on LLM/VLM-based digital agents, computer-use agents, agent memory systems, and agent self-improvement, as well as practitioners evaluating whether open-source backbones can match frontier closed-source models for software automation. Readers need enough background to interpret multi-agent harness design, code-as-policy action spaces, and benchmark scoring, so the paper is best suited to those already familiar with the digital-agent literature rather than newcomers to the area.

Authors’ abstract

Digital agents must often adapt to new environments whose interfaces, tools, and failure modes are not fully captured by pretrained models. We introduce \textbf{RSIAgent}, a training-free multi-agent framework for recursive self-improvement through autonomous memory construction. RSIAgent coordinates curriculum, actor, and verifier agents to continually explore the environment, validate outcomes, and retain environment-specific knowledge, including reusable causal relationships between actions, conditions, and consequences. It further adopts a \textbf{broad-then-deep} exploration strategy, combining parallel broad recursive self-exploration for discovering diverse environment structures with focused deep self-exploration for uncovering hard cases, hidden constraints, boundary conditions, and previously unknown causal dependencies. The resulting memory is frozen and can be directly reused for downstream tasks without updating model parameters. Experiments on OSWorld-v2 and Agent's Last Exam show that RSIAgent substantially improves strong open-source models, enabling Kimi-K3 and GLM-5.3 to outperform frontier closed-source models including GPT-6.