PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

AI summary

Overview

Research area: Natural Language Processing / AI agent evaluation — specifically benchmarks for persistent personal agents and the foundations of recursive self-improvement.

Technical level: Advanced. The paper assumes familiarity with agent loops, tool calling, memory/skill persistence layers, and paired-ablation experimental design.

Scope: The paper introduces PAST-Bench, a trajectory-level benchmark that measures whether a personal agent's retained experience actually improves its later performance, plus Hermes+, a modified agent runtime that targets five diagnosed failure points.

What This Paper Is About

Personal AI agents now persist across sessions, accumulating memories, skills, files, and histories, and it is widely assumed this accumulated experience makes them better over time. The problem is that a later-session score gain might come from the retained experience — or from the base model, the runtime, prompt shortcuts, task difficulty, or scoring noise, and existing benchmarks cannot separate these. The paper builds a benchmark that turns persistence on and off under matched conditions so any gain can be attributed to retained experience, and checks whether the gain travels through the intended save-retrieve-apply-update pathway rather than just showing up in the score.

Key Contributions

1. PAST-Bench, a trajectory-level performance-attribution benchmark for self-evolving personal agents. It spans 26 scenarios and 204 episodes across four capabilities — memory (5 families / 41 episodes), procedural reuse (8/64), information gathering (6/48), and update (7/51) — with matched persistence-on/persistence-off controls and trace-level evidence for per-stage diagnosis.

2. A diagnostic study of self-evolution failure. Across seven base models and four agent frameworks, persistence gaps vary sharply by capability, and agents that tie on task-score gap can differ substantially in mechanism evidence — the paper's example is Hermes versus nanobot, both at Δ = +0.13 but with mechanism scores of 0.64 versus 0.57, a discrepancy invisible to one-shot scoring.

3. Hermes+, a new agent framework baseline that extends Hermes with five runtime mechanisms, one per stage of the agent loop (Plan, Render, Route, Gate, Close). It raises reported means on both axes (Δ: +0.13 → +0.15, Mech: 0.64 → 0.73), with super-additive composition on Update (Δ = +0.24, above closeout alone at +0.16 and retrieval gate alone at +0.06). The paper notes the Overall Δ difference is smaller than run-to-run variation.

4. A methodology comparison (Table 1) situating PAST-Bench against GAIA, AgentBench, VisualWebArena, WorkArena, OSWorld, LongMemEval, LoCoMo, SkillsBench, and AgentBoard along four axes — cross-session retained experience, fixed-framework model comparison, fixed-model framework comparison, and trajectory-level diagnostics. PAST-Bench is claimed as the first to support all four jointly.

Main Findings

• Persistence helps every model tested, but by uneven amounts. With the framework fixed to Hermes, every base model gains: GLM-5.1 +0.20 (0.52 → 0.71), Kimi K2.6 +0.17 (0.57 → 0.73), DeepSeek-V4-Pro +0.17 (0.50 → 0.67), MiniMax-M2.7 +0.13 (0.53 → 0.66), GPT-5.4 +0.24 (0.51 → 0.75), Claude Sonnet 4.6 +0.20 (0.50 → 0.71), Claude Opus 4.6 +0.19 (0.49 → 0.67). Overall Δ ranges from +0.13 to +0.24.

• Each model concentrates its gain differently. GPT-5.4 spreads movement across Memory (38%) and Update (35%); GLM-5.1 places 46% on Update; Kimi K2.6 places 49% on Memory. The paper argues a single Overall Δ tells the wrong story and that the four-capability decomposition is what reveals model-specific strengths.

• Same headline gap, different mechanisms. Holding MiniMax-M2.7 fixed, nanobot and Hermes both reach Overall Δ = +0.13, but nanobot earns it from a single capability with no consistent write-then-read trace, giving Mech 0.57 against Hermes's 0.64. ZeroClaw reaches Δ = +0.12 with Mech 0.55; Agent-Zero regresses overall (Δ = −0.08, Mech 0.39), worsening on three of four capabilities.

• Hermes is the strongest controlled baseline for intervention. It is the only non-Hermes+ framework with positive Δ on all four capabilities, ties nanobot for best baseline Overall Δ (+0.13) while showing stronger mechanism evidence, and is the only framework in the set that exposes the agent loop without a pre-instantiated persistence stack — the property required for clean mechanism-by-mechanism ablation. Other frameworks are reported as off-the-shelf baselines.

• Five disjoint failure categories drove Hermes+. Plans were drafted without first consulting saved state; saved facts appeared in the wrong form; learned procedures stayed outside the skill library; stored evidence was skipped before action; and corrected state failed to reach the next fresh session. Each was assigned to one loop stage so the mechanisms could be ablated independently.

• Single mechanisms hit their target diagnoses. E2 (Render) gives the highest Memory persistence-on score (0.80), E3 (Route) the largest single-mechanism Procedural Δ (+0.10), E4 (Gate) the largest Information Gathering Δ (+0.17), and E5 (Close) the strongest single-mechanism Update Δ (+0.16).

• Mechanism interaction can reverse component effects. In a focused Procedural diagnosis, removing E2 raises Δ from +0.085 to +0.108, while removing E3 or E5 lowers it to +0.062 or +0.042. The routing rows link the E2 effect to more consistent task-specific skill creation and reuse (no task-specific skill before evaluation: 2/6 full versus 1/6 without E2; no task-specific skill read during evaluation: 3/6 versus 2/6).

• Cross-model transfer is positive but not uniform. Swapping Hermes+ (tuned on MiniMax-M2.7) onto five base models matches or improves the Hermes baseline on three of five configurations: MiniMax-M2.7 (+0.13 → +0.15), Claude Sonnet 4.6 (+0.20 → +0.22), and GPT-5.4 (flat at +0.24). DeepSeek-V4-Pro and Claude Opus 4.6 regress slightly. Hermes+ with GPT-5.4 ties the benchmark's highest configuration (Δ = +0.24, Mech 0.80).

• The full system preserves overall performance with its clearest gain on Update. Hermes+ (full) ties Base Hermes on Overall persistence-on score (0.66), raises Overall Δ from +0.13 to +0.15, and reaches the best Update score (0.74) and gap (+0.24), while Procedural declines slightly (−0.02).

• Not reported in the available content: per-episode token and wall-time costs are cited as appearing in Table 12 of Appendix D.6 but are not included in the truncated text, and the full per-capability breakdown of Table 6 is cut off after GPT-5.4's Update column.

Methodology in Plain English

The core design move is to change the unit of evaluation from a single task to a task family: an ordered sequence of fresh sessions that share a latent rule, a reusable artifact, a correction, or a pre-seeded reference. The framework's volatile context is wiped between episodes, so the agent cannot carry anything forward through prompt overlap. Any improvement in a later episode must therefore travel through the persistent substrate — a memory store, skill file, playbook, or edited rule.

Each family contains episodes in four roles. Cold episodes measure first-contact behavior. Learn episodes (and an Update episode in Update families) deposit the target into the persistence substrate. Evaluation episodes probe reuse in a later fresh session with the trigger wording removed. Control episodes check that the gain cannot be explained by shortcut, surface-memorization, stale reuse, or writing to the wrong substrate — the benchmark uses no-retention, distractor, stale, and wrong-mechanism controls.

The head-to-head comparison is a matched ablation. The w/o-evolve condition denies the runtime any access to family-produced state; the w/-evolve condition permits it. Both runs share the same prompt, grader, tool stack, and seed, so the gap is plausibly attributable to the persistence layer. The authors describe this as a strong design control rather than causal proof. Cold scores calibrate headroom but are explicitly not the persistence-off baseline.

The primary metric is the self-evolution gap Δ_f = S(w/-evolve) − S(w/o-evolve), where S is the within-family mean of the per-episode task score on a 0-to-1 scale; the capability-level Δ is the macro-average over families. A gap is only accepted as evidence of self-evolution when it clears the upper bound set by that family's control episodes. Alongside the gap, the benchmark reports a mechanism-evidence score built from memory writes and reads, skill creation and patching, session-search calls, artifact diffs, and update-correctness signals — so an agent that writes artifacts but never reads them, or succeeds through the wrong substrate, does not earn credit on Δ alone.

For the second half, the authors used the benchmark as a diagnostic: they held the model fixed and varied frameworks, held the framework fixed and varied models, and then inspected low-gain and uneven-gain traces. The failures fell into five categories, each mapped to one loop stage, and Hermes+ adds one independent drop-in mechanism per stage. Ablations are run one mechanism at a time on top of Base Hermes (non-cumulative rows), then all five together, then full-minus-one for the Procedural interaction, then cross-model transfer.

Why This Matters

Impact on research. The paper reframes self-evolution from a score question to a performance-attribution question, and argues that cross-session retention, a well-formed tool trajectory, or a self-generated artifact alone is not evidence of beneficial cross-episode reuse. It also speaks to a contamination concern raised in recent critiques — that later-task gains can be real score gains yet not caused by the mechanism a framework claims credit for — by separating outcome gains from trace-level mechanism diagnosis under matched controls. The finding that two agents can tie at Δ = +0.13 while differing on mechanism evidence (0.64 versus 0.57) is a concrete argument for changing how persistent agents are evaluated.

Real-world applications (mapped from the four capabilities the benchmark targets):

• Personal assistants that retain durable user preferences, constraints, one-line policies, and exceptions across sessions without being restated.
• Technical workflow agents that save a solved SOP, playbook, build/deploy pipeline, or incident-triage flow and re-execute it later in the correct order with the right tool composition.
• Knowledge-heavy work where the answer already sits in a memory store, registered skill, indexed transcript, or home-state fixture and the agent must proactively retrieve it instead of answering from a plausible-but-wrong generic default.
• Long-lived deployments where facts, rules, and procedures change — fact correction, rule migration, temporary-exception expiry, SOP patching — and the agent must use the new state without leaking the old.

Industry relevance. Agent frameworks such as Hermes and OpenClaw already treat persistent workspaces, memories, skills, and tool execution as first-class runtime components, with memory layers such as Mem0 and LangGraph supplying editable memory, interaction-derived facts, temporal knowledge graphs, and procedural skill files. The paper provides a way to check whether those persistence stacks are actually delivering value, and a reference baseline (Hermes+) that other frameworks can be compared against. The caveat that the Overall Δ difference is smaller than run-to-run variation is a useful signal that headline gains in this space are easy to over-read.

Future Directions

• Strengthening the causal claim. The authors describe matched ablation as a strong design control rather than causal proof and lean on mechanism evidence as a complementary substrate-level check; a tighter causal identification of which persistence decision produced a gain is left open.

• Explaining and fixing the Procedural regression. Hermes+ shows a small Procedural dip (−0.02), and the full-minus-one diagnosis shows removing E2 raises the Procedural gap to +0.108 from +0.085. Understanding why E2 interacts negatively with Procedural performance is flagged as a focused diagnostic rather than a full-benchmark estimate.

• Making transfer uniform. Hermes+ was tuned on MiniMax-M2.7. It matched or improved the Hermes baseline on three of five base models but regressed slightly on DeepSeek-V4-Pro and Claude Opus 4.6, so capability- and model-dependent effects remain unresolved.

• Broadening the mechanism set and stage coverage. The five interventions cover Plan, Render, Route, Gate, and Close, and

Authors’ abstract

Recursive self-improvement requires agents to turn accumulated experience into better future behavior. Personal AI agents offer a concrete setting for studying this capability because they retain preferences, task histories, tool routines, and learned skills across sessions. Yet whether retained experience actually improves them over time has not been systematically tested. We introduce PAST-Bench, a benchmark designed to isolate this question. Each agent runs through ordered sequences of fresh-session tasks under matched conditions that turn retained experience on and off. It spans 26 scenarios and 204 episodes across memory, procedural reuse, information gathering, and update. We report both later-task gains and whether those gains follow the intended save, retrieve, and update pathway. Across seven base models and four agent frameworks, improvement is real but uneven across capabilities. Agents with the same headline gain can differ markedly in whether that gain is supported by evidence of the intended pathway. Guided by these findings, we develop Hermes+, which extends Hermes with five targeted interventions across stages of the agent loop. Hermes+ raises the average gain from retained experience and provides clearer pathway evidence, with its strongest improvement on tasks requiring outdated state to be replaced, although the effect remains capability- and model-dependent. Together, PAST-Bench and Hermes+ provide an evaluation and diagnostic foundation for studying how persistent agents can progress from retaining experience to systematically improving through it. Code: https://github.com/Gen-Verse/PAST-Bench