Dream-RSI: Recursive Self-Improvement through Evolving Worlds

AI summary

Overview

Research area: Autonomous AI agents and recursive self-improvement, specifically meta-level optimization of exploration policies for long-horizon agentic discovery. The paper sits at the intersection of natural language processing (LLM-driven coding agents), LLM-based scientific/algorithmic discovery, and model-based reinforcement learning.

Technical level: Advanced. It formalizes discovery as a stochastic tree-building process, defines a replay simulator over recorded discovery trees, and evaluates on GPU kernel engineering, Lasso solver design, and open mathematical optimization problems.

Scope in one sentence: Dream-RSI converts completed agent discovery histories into reusable replay simulators so that exploration policies can be cheaply evaluated "off-policy" offline and then redeployed online, closing a recursive self-improvement loop at the exploration layer.

What This Paper Is About

Long-horizon agentic discovery — where an LLM coding agent proposes, evaluates and refines candidate solutions over thousands of proposal–evaluation cycles — is bottlenecked by its exploration strategy. Hand-designed exploration policies stay fixed and never learn from accumulated experience, while trying to optimize exploration online is extremely costly because meta-level feedback is delayed (a policy can only be judged after many cycles) and the meta-policy space is vast.

Dream-RSI's goal is to make exploration policies recursively self-improving at low cost by reusing something that discovery already produces: the structured tree of past exploration decisions and their realized execution outcomes, which the authors treat as a replay simulator rather than as static textual context or fine-tuning data.

Key Contributions

1. History as Replay Simulator. The paper conceptualizes completed discovery histories as replay simulators (also called "worlds"). Because every node's outcome is pre-stored, a single expensive online run supports many zero-execution-cost off-policy evaluations of alternative exploration strategies, making delayed meta-level feedback reusable.
2. The Dream-RSI meta-layer RSI loop. A three-stage recursive loop: (1) Online Exploration, where the current policy guides a coding agent and logs traces; (2) Simulator Construction, where the resulting discovery tree is converted into a reusable simulator pool; (3) Dreaming-based Policy Improvement, where candidate policies are evaluated on the replay worlds and the best version is redeployed online, expanding the pool.
3. A programmable orchestration layer. Exploration is made explicit and executable through a lightweight orchestration layer controlling branching, parallel exploration and stopping, while the underlying coding agent, evaluator, models and execution interfaces remain fixed. Only the exploration-policy code changes across rounds.
4. Empirical validation across 8 scientific discovery tasks in three domains — algorithm engineering, mathematical optimization and GPU kernel engineering — compared against a controlled Recursive Fixed Exploration baseline and a broad set of domain baselines.

Main Findings

• Lasso regularization path (algorithm engineering). With Gemini-3.1 Pro, Dream-RSI reduces average runtime across six held-out datasets from 3587.1 ms (fixed exploration) to 2931.0 ms while using 317 discovery-agent calls instead of 550. With Gemini-3.7-Flash, it reduces average runtime from 2516.7 ms to 2350.6 ms using 1879 calls instead of 3200. The discovered solvers outperform sklearn (average 44180.3 ms) and glmnet (average 13767.5 ms) on all six held-out datasets. Against SimpleTES, which uses 51,200 generations, Dream-RSI achieves lower average downstream runtime with roughly two orders of magnitude fewer discovery-agent calls (reported as up to 162× fewer calls over SimpleTES and 1.7× over fixed-exploration baselines).
• Mathematical optimization. Using Gemini-3.1 Pro over 10 rounds, Dream-RSI reaches a Sum–Difference score of 1.145427 (vs 1.144047 for Recursive Fixed Exploration and 1.143975 for SimpleTES), a Circle Packing score of 2.635983 (matching the strongest reported result among compared methods), and an Autocorrelation score of 1.456375 (vs 1.456001 for Recursive Fixed Exploration; AlphaEvolve reports 1.455700, ThetaEvolve 1.493000, EvoX 1.458900, ShinkaEvolve 1.457800, OpenEvolve 1.460000, SimpleTES 1.453675). The paper reports matching or surpassing strong baselines within 1k generations, over 50× budget savings compared with SimpleTES.
• GPU kernel engineering (KernelBench). On VGG16 and LayerNorm, Dream-RSI reaches comparable final performance with 2.43× and 1.79× fewer generations respectively. On ConvDiv and ConvMax, under comparable discovery budgets, it achieves 2.09× and 1.44× higher performance, where performance is inverse runtime (1/ms).
• Guidance from history hurts. On ConvDiv, injecting explicit semantic directional guidance distilled from past trajectories into the prompt consistently underperformed the unguided counterpart for both Recursive Fixed Exploration and Dream-RSI, suggesting strong semantic inductive biases over-constrain parallel search.
• Learned exploration adapts effort. On ConvDiv, the round-best performance rises from 0.427 at round E0 to 1.898 at E8. The number of evaluated attempts drops from 110 to 50 while performance improves, then increases again (92, 80, 91, 86) as progress plateaus, coinciding with further gains.
• Discovered solver structure. Unlike SimpleTES, which switches between LARS and coordinate descent by problem dimension, the discovered Lasso solver adds adaptivity inside active-set optimization: strong-rule screening with Cauchy–Schwarz-based KKT pruning, selective recomputation of exact gradients only when the bound cannot certify a feature, fallback to a full refresh, plus lazy Gram-matrix construction and hardware-aware implementation.
• Per-round budget parity. Gemini-3.1 Pro runs 10 parallel workspaces with up to 11 refinement steps (10 × 11 = 110 discovery-agent calls per round); Gemini-3.7-Flash runs 32 parallel workspaces with up to 20 refinement steps (32 × 20 = 640 calls). Dream-RSI and the baseline share these budgets and are identical in Round 1.

Methodology in Plain English

The system keeps the coding agent and the evaluator frozen and only changes the code that decides where to explore. A discovery run is recorded as a tree: the root is the initial workspace state, and each non-root node records one generation–evaluation attempt with the parent workspace it resumed, its filesystem snapshot, generated artifact, diagnostics and score.

At any moment the policy may pick only from the root plus the current leaves — a set of eligible nodes — and chooses a batch of at most W of them, where W is the number of parallel workers. Online, each selected node spawns a real new child. Offline, the same policy is replayed against a stored tree: when it selects a node, replay just reveals the children that were already recorded (deterministically, in recorded parent–child order), with no new agent calls or evaluations. Replay stops when the policy picks an empty batch, hits a round cap K2, or the whole recorded tree has been revealed.

Each replay trajectory is scored with a three-part objective: the best solution quality found along the revealed subtree, minus a penalty proportional to the number of revealed non-root nodes (execution cost), plus a bonus for the average number of attempts per decision round (parallelism). A policy version's overall score is its average across all historical trees. Each outer iteration evaluates M policy versions, starting from the current online policy, and a fixed LLM-based policy-development agent reads the replay trajectories and scores and rewrites the executable policy code between revisions. The version with the highest average replay score is selected for the next online rollout. Because the current policy is always in the candidate set, the selected policy is guaranteed to be no worse than the current one on the fixed history — a monotone improvement guarantee at the meta level.

Why This Matters

This work reframes accumulated agent experience from passive context into an active, queryable model of the search space, making meta-level optimization of exploration tractable where feedback is otherwise delayed and expensive. It provides a concrete, off-policy way to close a self-improvement loop without training weights or rerunning discovery.

Real-world applications:

• Scientific and mathematical discovery pipelines, where search over constructions or proofs spans thousands of proposal–evaluation cycles and compute is the binding constraint.
• High-dimensional statistics and genomics/finance modeling, where Lasso regularization paths and similar primitives are run repeatedly and faster discovered solvers translate directly into wall-clock savings.
• Compilers, inference engines and ML systems, via automatically discovered GPU kernels with better performance per generation of search effort.
• General agent infrastructure, where the same replay-simulator trick could govern scheduling, branching and stopping policies in any long-horizon tool-using agent.

Industry relevance: the framework leaves the underlying coding agent and evaluator unchanged and only swaps an orchestration layer, so it can be layered onto existing LLM-based discovery/optimization stacks (the authors evaluate via the Gemini CLI with Gemini-3.1 Pro and Gemini-3.7-Flash) to cut discovery compute budgets.

Future Directions

• Whether replay simulators can be built from partial or noisy histories, and how closely offline replay scores predict real online outcomes as the history grows and the world distribution shifts.
• Extending the loop beyond exploration policy code to other meta-level components (evaluation rubrics, agent harnesses, concurrency limits) that the paper deliberately keeps fixed.
• Understanding why explicit semantic guidance degrades search, and designing guidance mechanisms that preserve diversity in parallel exploration.
• Generalizing beyond the three studied domains — the paper's evidence covers algorithm engineering, mathematical optimization and KernelBench kernel engineering, but not other discovery settings.

Target Audience

Researchers and engineers working on autonomous LLM agents, recursive self-improvement, agentic scientific and algorithmic discovery, and meta-learning or model-based RL for search control. It is most useful to readers already comfortable with reinforcement-learning framing (policies, off-policy evaluation, world models) and with LLM-driven optimization loops; the formal tree-construction and replay definitions in Section 3 make it less accessible to a purely applied audience.

Authors’ abstract

Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts. We introduce \textsc{Dream-RSI}, a framework for scalable and recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. By performing dreaming in the replay simulator constructed from historical discovery trees, \textsc{Dream-RSI} secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations. The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, \textsc{Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.