ClawLoop: Self-Improving Agents via Harness-Native Training Loop

AI summary

Overview

Research area: Post-training of large language model (LLM) agents that operate inside "harness" environments (sandboxed Docker containers with a shell, file system, package manager and network access), plus automated training-data synthesis.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, reinforcement learning for LLMs (specifically GRPO), agent rollouts, and benchmark evaluation. The prose is accessible, but the framework is engineering-heavy.

Scope in one sentence: The paper introduces ClawLoop, a closed-loop, teacher-free framework that lets a harness-based agent synthesize its own training tasks, train on its own successful trajectories, diagnose its own failures, and feed that diagnosis back into the next round of task synthesis.

What This Paper Is About

Existing post-training pipelines for harness-based agents are one-shot and depend on a stronger external "teacher" model to write tasks and collect trajectories, which caps the student at the teacher's capability and leaves no mechanism for the agent to improve itself. ClawLoop replaces that teacher with a three-phase cycle — task synthesis, training, and diagnosis/feedback — run entirely inside the same harness the agent is being trained for. The goal is to show that an agent can identify its own weaknesses, generate targeted practice for them, and measurably improve on held-out benchmarks without any external supervision.

Key Contributions

1. A closed-loop, teacher-free training framework for general agents. Unlike prior pipelines that rely on an external strong model and operate in a single pass, ClawLoop has the agent synthesize its own tasks, collect its own trajectories, diagnose its own weaknesses, and iterate — all within the same harness environment used for evaluation.

2. A multi-stage task synthesis mechanism. It combines two complementary routines (capability-driven synthesis and domain-grounded synthesis), a diagnosis-conditioned difficulty-growth step, and a two-stage validation (agent-free static checks plus agent-based filtering), designed so that tasks are both code-verifiable and complex enough to require deep reasoning.

3. Empirical demonstration on Qwen3.6-27B inside the loop, with consistent gains over both the base model and the ClawGym pipeline on SkillsBench, Claw-Eval-Live and ClawGym-Bench.

4. Transfer of the synthesized tasks to smaller models (Qwen3.5-4B and 9B), showing gains across all three benchmarks and thereby validating data quality independently of the loop model, plus ablations isolating each synthesis routine, the difficulty-growth schedule, and the iteration count.

Main Findings

• ClawLoop beats both the base model and ClawGym on Qwen3.6-27B. SkillsBench (Avg@5): base 46.3, ClawGym 52.6, ClawLoop 57.2. Claw-Eval-Live (Avg Score): base 69.3, ClawGym 74.6, ClawLoop 76.9. ClawGym-Bench (Avg Score): base 67.6, ClawGym 61.9, ClawLoop 68.4. Notably, the ClawGym pipeline falls below the base model on ClawGym-Bench.

• In-distribution gains are comparable to out-of-distribution gains. The paper argues that because improvements on ClawGym-Bench are similar in size to those on SkillsBench and Claw-Eval-Live, the loop produces genuine capability improvement rather than overfitting to its own task distribution.

• Synthesized tasks transfer to much smaller models and outperform teacher-dependent data. Qwen3.5-4B: base 5.8 / 32.4 / 33.0; ClawGym (with MiniMax M2.5 + GLM-5.1 teachers, SFT) 6.2 / 36.8 / 34.6; ClawLoop (no teacher, RL) 8.4 / 48.0 / 44.1. Qwen3.5-9B: base 6.4 / 58.3 / 32.0; ClawGym 6.7 / 64.3 / 35.3; ClawLoop 10.8 / 71.7 / 43.6. Despite ClawGym using stronger external teachers, ClawLoop data matches or exceeds it at both scales.

• Both synthesis routines matter, and they affect different benchmarks. Removing capability-driven synthesis: 54.5 / 73.2 / 67.2. Removing domain-grounded synthesis: 55.9 / 70.1 / 67.8. Full ClawLoop: 57.2 / 76.9 / 68.4. Dropping the capability-driven routine hurts Claw-Eval-Live most, while dropping the domain-grounded routine hurts SkillsBench and ClawGym-Bench most.

• Difficulty growth helps up to a point. Harder tasks visibly lengthen trajectories (more tool calls, reasoning steps and recovery cycles), but longer trajectories do not guarantee better training signal. A moderate schedule — raising difficulty by at most one level per iteration — gives the best balance between challenge and success rate.

• Iteration count saturates unevenly across benchmarks. Performance rises steadily in early iterations. SkillsBench keeps improving with more iterations, whereas Claw-Eval-Live and ClawGym-Bench plateau around K = 3, which is why the main experiments use K = 3. The authors attribute SkillsBench's continued gains to its broader domain coverage.

• Rollout collection, not gradient updates, dominates compute. Per iteration on 32 H800 GPUs: validation rollouts 30.76 h (984 GPU-hours, roughly 41% of the total), GRPO 18.52 h (593), task synthesis 15.98 h (511), SFT 8.03 h (257), diagnosis rollouts 0.94 h (30); total 74.23 h (2,375 GPU-hours).

Methodology in Plain English

The agent starts as a pretrained model placed in an OpenClaw sandbox. Each of the K iterations has three phases.

Phase 1 — Task synthesis. Two routines build seed tasks from opposite directions. Capability-driven synthesis starts from a workflow pattern (reconciliation, cross-source audit, multi-document merge, domain calculation) and instantiates it across many domains, deliberately including long-tail ones such as beekeeping hive inventories or amateur radio logs, so the agent learns the workflow rather than surface cues. Crucially, the synthesis model never writes task data directly; it writes a runnable generator that the TaskForge system executes to produce the workspace, the answer key and an "answer contract." Execution grounds the answer in computation instead of model belief, and the contract spells out hidden decisions (filtering criteria, rounding, tie-breaks, join keys) without revealing answer values. Domain-grounded synthesis starts from an "executable domain" — tools with schemas plus a database — and uses AST inspection plus LLM analysis to build a tripartite graph of tools, parameters/return fields and database tables, which is clustered into skills. An abstract task graph is sampled by structural features (long-range dependencies, branching, convergence, error recovery, multiple sub-goals), then grounded in a domain under an information barrier that prevents leaking the gold trajectory.

Difficulty growth. Multi-hop draws force three-source joins (an index file with no values, per-entity detail files, and a cross-cutting reference table); dirtiness draws inject distractors, missing fields, schema inconsistencies and unit mismatches while keeping the ground truth well-defined.

Validation. Agent-free checks require the oracle to pass its own verifier and audit for answer leakage and trivial solutions. Agent-based validation runs N = 8 trials per task and keeps only tasks passing between N_l = 1 and N_h = 5 trials — filtering out tasks that are always solved (too easy) or never solved (too hard or inconsistent).

Phase 2 — Training. Self-distilled SFT on successful trajectories only, with loss masked on environment observation tokens so that long tool outputs do not dominate; trajectories are packed to the maximum length rather than truncated, to preserve verification turns. GRPO then refines the policy using outcome rewards, with advantages computed relative to the group mean over the same N rollouts.

Phase 3 — Diagnosis and feedback. The updated agent runs on the held-out synthesized split, and the agent itself writes structured summaries of each failed trajectory: failure point, root cause, and missing capability. These are clustered by skill category and root cause into failure patterns (for example, "modifies the target file but fails to update import statements in dependent modules"), and the aggregated report conditions task synthesis in the next iteration.

Evaluation uses SkillsBench (86 tasks, 11 professional domains, Avg@5), Claw-Eval-Live (105 tasks, real-world workflows, Avg Score) and ClawGym-Bench (200 tasks, Avg Score). Decoding uses temperature 0.7 and top-p 0.95. None of the three benchmarks is used for training, task selection or difficulty scheduling. Rollouts are capped at 24 hours wall-clock and 300 agent turns, with 32 concurrent containers.

Why This Matters

The paper attacks a structural limitation rather than a benchmark number: if every training pipeline needs a stronger external model, then agent capability is permanently bounded by whatever teacher is available, and the loop must be re-run by hand each time. ClawLoop's claim is that the harness itself already contains everything needed to close the loop — workspace instantiation, execution and code-based verification — so the missing ingredient is a mechanism, not infrastructure. The transfer result is the strongest evidence: tasks produced without any teacher produced larger gains on 4B and 9B models than tasks produced by MiniMax M2.5 and GLM-5.1 teachers.

Real-world applications:

• Automated post-training of coding and systems-administration agents that operate on real file systems and shells.
• Continual improvement of deployed agents on their own failure logs, without collecting human-annotated demonstrations.
• Lowering the cost barrier for organizations that lack access to frontier teacher models but can afford sandbox compute.
• Generating domain-specific agent training data (database administration, workload planning, document reconciliation) from existing tool ecosystems and MCP servers.

Industry relevance: The compute breakdown matters commercially — validation rollouts alone consume 984 of 2,375 GPU-hours per iteration, more than SFT and GRPO combined. Any team deploying this must budget for sandbox execution rather than gradient updates, which shifts the infrastructure conversation toward container orchestration and rollout throughput.

Future Directions

• Longer training horizons with adaptive stopping. The authors explicitly leave this to future work; the current study fixes K = 3 because Claw-Eval-Live and ClawGym-Bench plateau there, while SkillsBench was still improving, so the right stopping rule is unresolved.

• Diagnosis quality as the binding constraint. Feedback is generated by the agent summarizing its own failures. The paper reports that this is easier than solving the task, but does not measure how accurate or how actionable those self-diagnoses are, nor what happens when the agent misdiagnoses itself.

• Reusing one rollout set for three purposes. Every task gets N = 8 rollouts that simultaneously serve as validation signal, SFT data and the GRPO group. The paper defers examination of the consequences of this reuse to an appendix section, leaving the trade-off between compute savings and signal quality open.

• Extending beyond the evaluated harnesses. All training and OpenClaw-specific evaluation happen on one harness; only SkillsBench is harness-agnostic. Whether the loop transfers across different harness substrates is not established here.

Target Audience

Researchers and engineers working on agent post-training, synthetic data generation, and self-improving or self-play LLM systems. It is most useful to practitioners who already run sandboxed agent rollouts at scale and want a recipe for turning evaluation failures into the next round of training data without a teacher model. Readers seeking a self-contained introduction to agent training will find the framework description clear but will need prior background in SFT, GRPO and harness-based evaluation to follow the experimental setup.

Authors’ abstract

The rapid advancement of Large Language Models (LLMs) has driven their deployment toward general-purpose harness environments, where LLM-based agents operate in sandbox workspaces and interact directly with file systems and shells. While numerous post-training pipelines exist for tool-use agents, their counterparts for harness-based agents remain scarce and are predominantly one-shot and teacher-dependent: a stronger external model synthesizes tasks and collects trajectories, the target agent is trained once, and evaluation results are consumed by human researchers rather than fed back into the loop. Such pipelines are inherently bounded by the capability of the external model and the scope of human prior, leaving no room for self-improvement. To address these limitations, we propose ClawLoop, a framework that enables self-improving without any external teacher model and with minimal prior dependencies. ClawLoop composes three phases into a self-sustaining cycle: Task Synthesis, which generates a diverse pool of training tasks via two complementary construction routines, applies diagnosis-conditioned difficulty growth followed by both agent-free and agent-based task validation; Training, which updates the agent via SFT and/or RL; and Diagnosis and Feedback, which analyzes the agent's rollout behaviors to produce a capability analysis report that will be used as feedback for task synthesis in the next iteration. Using Qwen3.6-27B as the loop model, ClawLoop yields consistent improvements on SkillsBench, Claw-Eval-Live, and ClawGym-Bench. The iteratively refined tasks are further transferred to smaller-scale Qwen3.5-4B and 9B models with gains across all benchmarks, validating data quality independently of the loop model.