RSIHub: A Composable Framework for Agent-Evolution Methods

AI summary

Overview

Research area: AI agent engineering — specifically infrastructure for automating and comparing "agent-evolution" methods (systems that use execution feedback to rewrite an agent's prompts, skills, tools, or orchestration).

Technical level: Intermediate. The paper is an infrastructure/systems contribution with an accompanying empirical study; it assumes familiarity with language-model agents, benchmarks, and pass@1 evaluation, but the core ideas are explained conceptually.

Scope: RSIHub is an open-source, Apache-2.0 framework that separates evolution policy (a composable "recipe" of lifecycle operators) from shared machinery (execution, evaluation, recovery, lineage), demonstrated by re-implementing AHE, Hyperagents, A-Evolve, and GEPA and comparing them across two seed agents and two benchmarks.

What This Paper Is About

Agent-evolution research keeps rebuilding the same experimental scaffolding — isolated candidate workspaces, task execution, evaluation, artifact storage, crash recovery, and lineage tracking — bundled together with each method's own policy. That makes new methods expensive to implement and makes comparing methods hard, because runners, configs, and evaluator interfaces differ. RSIHub's goal is to make evolution policy the only thing a researcher writes, while one protected, shared runtime handles evaluation and record-keeping so that reported gains can be attributed to the method rather than to changes in measurement conditions.

Key Contributions

1. Composable recipes. A recipe abstraction that binds one operator implementation to each evolution-lifecycle slot (Select, Rollout, Analyze, Mutate, Guardrail, Evaluate, Absorb), declares the mutable surface (what a candidate may change), and configures the target, evaluator, runtime, and data splits. AHE, Hyperagents, A-Evolve, and GEPA are all expressed as recipes over the same five required and four optional operator interfaces.

2. Shared experimental infrastructure. A common lifecycle for candidate execution, workspace construction, snapshots, artifact retention, recovery from interrupted runs, certified evaluation records, and lineage — implemented once on top of Harbor, which supplies container execution, trajectory capture, and per-task verification.

3. Protected evaluation and provenance. Evaluation and its archive live outside the declared mutable surface and under framework control; a certified record binds the committed candidate, its parent, the active recipe snapshot, evaluator and task-set fingerprints, trials, score, costs, and artifact references, so a change in method policy can be distinguished from a change in measurement conditions.

4. Evaluation on common ground. A comparison of the four recipes across two seed agents (MiniSWE, Codex) and two benchmarks (Terminal Bench 2, τ³ Banking) under fixed model and evaluator configurations, plus tests of the framework itself: capability preservation, interruption recovery, boundary enforcement, runtime overhead, and integration effort.

Main Findings

• Largest gain is 13.5 percentage points. On Terminal Bench 2 with MiniSWE, Hyperagents and A-Evolve both raise overall pass@1 from 56.2 to 69.7 (+13.5); AHE reaches 65.2 (+9.0) and GEPA 64.0 (+7.8). The paper states this 13.5-point figure as the largest observed overall pass@1 gain.

• Gains shrink on the stronger seed agent. On Terminal Bench 2 with Codex (seed 69.7), AHE reaches 71.9 (+2.2), Hyperagents and A-Evolve each reach 70.8 (+1.1), and GEPA stays at 69.7 (+0.0).

• Rankings flip across settings. On τ³ Banking with MiniSWE, Hyperagents is best (33.0, +5.2 from 27.8), with GEPA at 29.9 (+2.1) and AHE at 28.9 (+1.1). On τ³ Banking with Codex, AHE is best (34.0, +9.3 from 24.7), with GEPA at 33.0 (+8.3). No method dominates across benchmarks, seed agents, and splits.

• Held-out transfer is uneven. With MiniSWE on Terminal Bench 2, all methods improve the train and sealed splits but only Hyperagents improves the gate split. With Codex, AHE and Hyperagents improve train and sealed while dropping gate performance by 3.4 and 2.2 points. On τ³ Banking with MiniSWE, Hyperagents posts the largest overall gain despite a 2.1-point sealed regression, and AHE and GEPA lose 4.1 and 2.1 points on sealed tasks.

• Capability preservation is close but not exact. Across three repetitions, RSIHub's generation-zero execution of an unchanged seed agent differs from direct execution by −2.6 points on Terminal Bench 2 for both MiniSWE and Codex, by −2.6 for MiniSWE on τ³ Banking, and by +3.4 for Codex on τ³ Banking. D–R agreement ranges from 80.1% to 84.6%, and direct executions themselves disagree on 12.0–18.9% of tasks. Codex's RSIHub Avg@3 of 61.4 falls inside the 60.7–65.2 range across the six direct executions.

• All integrity and recovery trials passed. All 420 fault-injection and boundary-enforcement trials (30 per scenario, seven scenarios, two benchmark campaigns) reached the required outcome without admitting corrupted state — covering termination during mutation, during evaluation, and after a receipt was written, plus evaluator modification, out-of-surface writes, protected-evidence access, and forged archive outcomes.

• Framework overhead is small. Framework-exclusive spans account for a median of 0.02% (P95 0.06%) of end-to-end generation time on Terminal Bench 2 and 0.019% on τ³ Banking; model, mutation, and benchmark execution dominate runtime.

• Low integration cost. The four adaptations used 47–61 recipe lines each, reused two operator bindings each, and required no framework-source changes; method-specific operator code ranged from 216 to 1,087 lines.

Methodology in Plain English

The researchers fixed the stages of the evolution loop and left the contents of each stage open. A stage is a slot (e.g., select a parent, mutate a candidate); an operator is one implementation of that slot; a recipe names which operator goes in each slot and configures it, along with the target agent, evaluator, runtime, and data splits. Two recipe declarations control the experiment: the surface block says which files a candidate may change, and the mutate block says how the editing agent runs and what context it sees.

Every iteration follows seven phases. The framework selects an eligible parent from the archive and forks its exact recorded commit; the parent rolls out on training tasks; an optional analysis step distills failures into bounded evidence; a mutation agent proposes edits; the framework checks the actual tree diff against the declared surface, with optional validation and novelty filters; then the framework — not the method — commits the tree, scores a clean checkout on gate tasks, and certifies the result; finally a separate Gate operator decides whether the child becomes a parent, while Record and optional Reflect add annotations that cannot touch certified fields.

Benchmarks were split into train, gate, and sealed portions: Terminal Bench 2 has 89 tasks (50 train, 19 gate, 20 sealed) and τ³ Banking has 97 tasks (50 train, 20 gate, 27 sealed). Train trajectories feed evolution, gate tasks govern admission, and sealed tasks are excluded from mutation feedback and used only for held-out reporting. All experiments used GPT-5.4 with high reasoning effort as the target model and a Codex mutation agent backed by GPT-5.4 with xhigh reasoning effort; runs proposed one child per generation for at most 10 generations, with one trial per task and pass@1 reporting. For the framework's own fidelity, the authors ran three repetitions of a fixed order — Direct-A, RSIHub generation zero, Direct-B — and compared task-level agreement within and across modes.

Why This Matters

Impact on research. Agent-evolution papers currently report numbers produced by bespoke pipelines, which makes cross-method comparison unreliable and makes it hard to tell whether a gain came from better policy or from a changed evaluator. RSIHub turns the evolution lifecycle into a stable interface with protected measurement, inspectable candidate histories, and recorded evaluation conditions — the kind of shared protocol that lets a field accumulate results instead of anecdotes.

Real-world applications:

• Building agents for terminal-based software and system-administration tasks, where the paper shows the largest gains.
• Customer-service and knowledge-grounded dialogue agents, represented by the τ³ Banking benchmark. (The paper cautions that banking-benchmark success does not establish suitability for financial decisions.)
• Internal platform teams that want to improve a deployed agent's prompts and skills from production feedback while keeping evaluation outside the reach of the optimizer.
• Auditable model-improvement pipelines in regulated settings, where rollback, lineage, and certified evidence of what changed and under what conditions are requirements rather than niceties.

Industry relevance. The framework addresses an engineering cost that labs and product teams both pay: every new evolution idea currently requires rebuilding workspaces, execution, recovery, and evaluation. RSIHub reports 47–61 recipe lines and 216–1,087 operator lines per method with no framework-core changes, and framework overhead at roughly 0.02% of generation time — numbers that matter to teams deciding whether to adopt shared infrastructure.

Future Directions

• Evaluate the agent-driven mode. RSIHub supports an outer agent that revises recipe bindings and policy settings between iterations, but the reported results characterize fixed, recipe-driven runs only. Whether adaptive recipe revision beats fixed recipes is untested here.

• Establish stability of close differences. The authors note that the descriptive point estimates do not establish the stability of close rankings without repeated runs, and that the capability-preservation controls do not establish execution-mode equivalence or statistical significance for the single-run evolution gains.

• Strengthen trust assumptions. The evidence guarantees rely on trusted framework services, host and filesystem protections, Git and digest operations, and valid evaluators; runtime fingerprints do not replace host or service attestation, and RSIHub does not contain hostile code, so stronger deployments need independent attestation, isolation, and containment.

• Broaden the method and workload space. With four adapted methods, two seed agents, and two benchmarks, the setting-dependent rankings argue for adding more recipes, operators, and environments — including the optional Analyze, Validate, Novelty, and Reflect slots that the current adaptations leave largely unused.

Target Audience

Researchers and engineers working on self-improving agents, prompt and program optimization, or automated agent-harness design who want to implement a new evolution policy without rebuilding execution and evaluation; benchmark and evaluation-infrastructure developers concerned with attribution, provenance, and measurement validity; and platform teams assessing whether to standardize agent-improvement experiments on shared tooling. Readers seeking a purely algorithmic or theoretical contribution will find this is a systems and empirical-methodology paper instead.

Authors’ abstract

Agent-evolution methods use execution feedback to improve prompts, skills, tools, and orchestration, but their implementations often rebuild the same experimental machinery. We present RSIHub, an open-source framework for building, comparing, and extending these methods on shared infrastructure. Recipes compose reusable operators and declare mutable resources; RSIHub supplies execution, evaluation, recovery, and lineage. Researchers can focus on evolution policy while evaluation stays outside the mutable surface and results remain tied to candidates and evaluation conditions. We instantiate AHE, Hyperagents, A-Evolve, and GEPA without modifying the framework core, keeping method-specific logic in recipes and operators. We evaluate them with MiniSWE and Codex on Terminal Bench 2 and $τ^3$ Banking under fixed model and evaluator configurations. The largest observed overall pass@1 gain is 13.5 percentage points, while method rankings vary across settings and optimization gains do not always transfer to held-out tasks. RSIHub provides a common protocol for comparing evolution policies, inspectable candidate histories, and reusable components for developing new methods.