Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents

AI summary

Overview

Research area: Evaluation of large language model (LLM) agents, specifically continual learning — whether an agent improves across a long sequence of related tasks by retaining and reusing experience.

Technical level: Intermediate. The paper is readable without deep mathematics, but it formalizes the benchmark as a partially observable Markov decision process and assumes familiarity with agent harnesses, memory systems, and rubric-based grading.

Scope: The paper introduces and reports results for the Continual Learning Exam (CLE), a 600-level benchmark built by 50+ domain experts to measure experience-driven improvement in realistic professional workflows across 17 harness–model configurations.

What This Paper Is About

Most existing agent benchmarks score each task in isolation, so they measure how well an agent solves a problem but not whether it gets better over time from accumulated experience. CLE closes that gap by organizing 120 human-authored professional tasks into five-level ladders, where later levels build on knowledge, artifacts, and user context established in earlier ones and agents must pass a level to unlock the next. The goal is to test whether frontier LLM agents can acquire, retain, and reuse experience — and to show how far current systems fall short.

Key Contributions

1. A laddered continual-learning benchmark. CLE contains 120 five-level task ladders (600 levels total) spanning 89 professional roles across 12 occupational domains, written by 50+ domain experts. Levels are linked by five forms of transferable knowledge: Tools and Skills, Data Dependency, Strategy, User Context, and Domain Knowledge.

2. An executable, stateful environment. The benchmark ships 23,981 real-world input files covering 77 distinct file extensions, 19 stateful mock applications exposing 1,024 APIs, and 16,125 human-authored and human-verified rubric criteria. Mock application databases persist across the levels of a task, so actions taken early carry into later assignments.

3. An agentic, rubric-based evaluator. A separate multi-turn evaluation agent runs in an isolated Docker container per grading request, using tools to extract evidence from submitted artifacts and final application state, and checks it against hidden reference materials. A submission passes only when every checklist item is satisfied.

4. A controlled feedback protocol and a large empirical sweep. The paper compares a single-submission setting (K=1) with one that returns rubric-derived diagnostic feedback after a failure and allows one revision (K=2), and evaluates 11 frontier backbone model variants in 17 configurations across 4 agent harnesses.

Main Findings

• Feedback helps, but modestly. Diagnostic feedback plus one revision produces a median completion gain of 5.3 percentage points across the 17 harness–model configurations, measured as the change in the percentage of all 600 levels passed between Trial 1 and Trial 2.

• Best system still fails more than half the levels. The top-performing configuration, Claude Code with Opus 5, clears only 47.67% of levels and completes all five levels in just 23.3% of tasks. The abstract rounds this headline figure to 47.7%.

• A large gap separates the leaders from other frontier models. GPT-5.6-Sol and GPT-5.6-Terra each reach 33.17% under Codex.

• Sustained learning, not single-task skill, is the bottleneck. Because later levels depend on earlier ones and agents must pass each level to advance, failures cascade: unreached levels count as failed, so weak retention of experience compounds into low ladder-completion rates.

• The benchmark is positioned as training data as well as a test. The authors state that CLE can be used to post-train LLMs to strengthen continual learning capabilities, not only to evaluate them.

• Conceptual scaffolding is offered alongside the data. The paper defines continual learning via an update operator that modifies model parameters, persistent harness state, or both, and organizes capabilities into five cumulative tiers (T1 guided, T2 autonomous within a task family, T3 cross-domain, T4 discovery-driven, T5 self-driven). It notes that current systems such as Claude Cowork and Codex instantiate T1 through predefined memory and skill workflows, with emerging T2 autonomy. No results are reported for T3–T5.

• Not reported in the provided content: human expert baselines, per-domain or per-knowledge-type breakdowns of results, cost or wall-clock statistics, and variance across repeated runs.

Methodology in Plain English

The researchers recruited more than 50 professionals to write realistic work assignments for their own occupations. Each task is a five-step ladder: an agent must complete step one before it can attempt step two, and so on, with later steps deliberately requiring information or habits from earlier ones — a tool procedure learned in level one, a file created in level two, a colleague's scheduling preference stated in level three.

The agent works inside a simulated office environment. Rather than real software, it uses 19 mock applications with 1,024 APIs that remember changes, so a record edited in one level is still edited in the next. Each level ships with the real files a professional would start from.

Grading is done by a separate AI evaluator, not by the agent itself. Every level has a hidden rubric written by the task author, listing concrete criteria with a single unambiguous expected outcome and the evidence needed to verify it. The evaluator reads the submitted files and the final state of the applications, then checks each criterion. Anything short of all criteria counts as a failure.

Two conditions are compared. In the first, the agent gets one submission per level and no hints. In the second, a failed first attempt returns feedback that names the relevant rubric category while keeping the individual checklist items and reference answers hidden; the agent can then revise and submit once more.

To measure improvement over a long horizon, the 120 ladders are run as one continuous stream in a fixed order, with model weights frozen but the agent's surrounding scaffolding — its memory, skills, notes, and context — carried forward between tasks. The framework also supports "blocks," contiguous slices of a ladder that can start at any level with the necessary predecessor files preloaded, which allows custom or repeated evaluation.

Why This Matters

Continual learning is the difference between an agent that can be deployed for weeks and one that must be reset after every task. This paper supplies the first large-scale benchmark, as far as the authors frame it, that measures improvement through accumulated experience rather than performance on isolated problems — and its results suggest current frontier systems are far from the T3–T5 tiers of autonomous, cross-domain, or self-directed learning.

Real-world applications:

• Software maintenance and verification. The paper's worked example is maintaining and verifying an Ibex RISC-V core, where each level changes objectives and constraints while reusing prior knowledge of the codebase and toolchain.
• Collaborative engineering workflows. Integrating collaborators' changes in Git repositories and synchronising project updates across multiple applications.
• Scheduling and coordination. Arranging meeting times around participants' availability and preferences that the agent learned in earlier interactions.
• Multi-application office work. Carrying artifacts and decisions across 19 stateful business applications, mimicking how real staff move between tools.

Industry relevance: Any organization deploying agents over long horizons — coding assistants with persistent memory, enterprise automation, customer-facing coworkers — needs to know whether retained experience actually improves future work or merely accumulates. The finding that one round of feedback and revision buys a median 5.3 percentage points, while the best system still completes only 23.3% of full ladders, is a direct signal about how much scaffolding and harness engineering still matter, and the dataset offers a possible post-training corpus for closing that gap.

Future Directions

1. Post-training for continual learning. The authors explicitly propose using CLE to fine-tune or reinforce models on experience-driven improvement, rather than only measuring it — an open question is whether parameter updates can beat frozen-model harness updates on these ladders.

2. Moving beyond T1 and T2. The capability hierarchy defines five tiers, but only the rule-governed and emerging-autonomy tiers are observed in current systems. Evaluating agents on cross-domain adaptation (T3), investigation design (T4), and self-selected research problems (T5) remains unaddressed.

3. Improving harness design and experience retention. Since model weights are frozen and only harness state carries over, the benchmark should reveal which memory, reflection, and skill-reuse mechanisms actually transfer across levels and tasks — the paper does not report a per-mechanism ablation in the provided content.

4. Separating retention from revision. With only K=1 and K=2 compared, it is unclear whether more revision rounds, richer feedback, or feedback at the checklist-item level would change outcomes; the block-based evaluation facility also opens the question of how results vary when ladders start mid-stream rather than at level one.

Target Audience

Researchers and engineers building LLM agent systems, particularly those working on agent memory, harness design, reflection and skill-reuse mechanisms, and long-horizon evaluation. It is also relevant to practitioners deciding how much scaffolding to invest in before deployment, and to benchmark designers interested in laddered, stateful, rubric-graded task structures. Readers seeking a rigorous treatment of forgetting in neural networks will find the formalization lighter than the empirical contribution.

Authors’ abstract

Continual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience. We introduce Continual Learning Exam (CLE), a large-scale benchmark developed by 50+ domain experts to measure experience-driven improvement in realistic professional workflows. CLE comprises 120 human-authored tasks, each organised as a five-level ladder, yielding 600 levels spanning 89 professional roles across 12 occupational domains. Agents must pass each level to unlock the next, with later levels building on experience accumulated during earlier assignments. An evaluation agent assesses each submission against human-authored rubrics. We compare a single-submission setting with one that provides diagnostic feedback after failure and allows one revision. Across 17 harness-model configurations using frontier LLMs, feedback and revision yield a median completion gain of 5.3 percentage points. Yet the best-performing system clears only 47.7% of levels and completes all five levels in just 23.3% of tasks, highlighting the difficulty of sustained learning from experience for existing frontier LLMs. As such, the dataset can also be used to post-train LLMs to strengthen their continual learning capabilities.