Harness-Model Collaboration for Long-Horizon Continual Learning

AI summary

Overview

• Research area: Continual learning for LLM-based long-horizon agents, combining external experience memory with parameter-efficient model adaptation.
• Technical level: Intermediate — assumes familiarity with LLM agents, LoRA-style adapters, replay-based continual learning, and retrieval, but the core idea is explained in plain terms.
• Scope: The paper proposes HaMo, a framework that pairs a persistent "harness" memory of distilled strategy cards with experience-conditioned continual fine-tuning of a single LoRA adapter, and evaluates it on ScienceWorld, ALFWorld, and WebShop under within-domain and cross-domain task streams.

What This Paper Is About

Long-horizon agents (especially Web agents) repeatedly perform related multi-step workflows, so they generate reusable procedural knowledge — but most systems either store that knowledge externally and never improve the model, or fine-tune the model and risk forgetting earlier skills. HaMo's goal is to get both benefits: instant reuse of newly learned strategies through external memory, plus gradual internalization of the ability to interpret and execute those strategies through continual model updates. The authors test whether coordinating these two timescales beats either mechanism alone.

Key Contributions

1. A two-level collaboration framework (HaMo). Each trajectory is distilled into a compact "experience card" — the task description, the initial observable conditions, and a strategy-level solution outline — stored in persistent non-parametric memory, while the model is continually updated at a slower cadence.
2. Experience-conditioned adaptation. Instead of learning only a direct context-to-action mapping, the model is trained to condition on retrieved strategy-level experience, so it learns to interpret, ground, and execute a retrieved strategy under different objects, observations, states, and interaction histories. The same input format (request, initial observation, injected experience) is used at training and inference.
3. Retrieval-coverage-based anchor selection. A greedy procedure embeds experience cards, does leave-one-out top-3 retrieval, and selects cards by marginal retrieval coverage — favoring broadly reusable strategies while still covering rare strategy contexts. A knee point on the coverage-saturation curve sets each stage's training budget; unselected cards stay in memory for retrieval.
4. A single, continuously updated LoRA adapter with historical replay. The adapter is initialized from the previous stage rather than trained from scratch, the backbone stays frozen, and no adapter merging or growing collection of task-specific adapters is used, so the trainable parameter budget stays fixed across the stream.

Main Findings

• Within-domain gains grow over stages (ScienceWorld). With Qwen3-8B, HaMo rises from 38.13 to 60.49, averaging 52.22 over four stages; with Qwen3-14B it rises from 38.61 to 61.15, averaging 53.09. These averages exceed Base+Harness by 13.88 and 12.44 points respectively, and final-stage gains over the unadapted backbones are 22.94 (8B) and 21.20 (14B) points.
• The advantage is not from a stronger start. At Stage 1 HaMo scores slightly below Base+Harness for both backbones, then leads all listed methods from Stage 2 onward; Base+Harness stays close to the backbone reference, suggesting external experience alone gives limited improvement here.
• Cross-domain results (S → A → W, Qwen3-8B). After the final stage HaMo reaches 52.43 on ScienceWorld, 66.79 on ALFWorld, and 59.64 on WebShop, beating every listed baseline on each domain — gains of 14.88, 22.26, and 13.96 points over Base, and 13.02, 19.71, and 5.16 points over the strongest non-HaMo baseline per domain.
• Retention vs. the trajectory-training baselines. AgentTuning's ScienceWorld score falls from 26.24 to 3.13 and ETO's from 20.16 to 3.20, while both improve over Base on the final WebShop task but finish below Base on the two earlier domains. HaMo's ScienceWorld score goes 52.43 → 54.19 → 52.43 across stages, and its ALFWorld score declines moderately from 70.08 to 66.79.
• Harness-only plateaus. In the component ablation, "Harness only" (i.e., Base+Harness) produces relatively small changes from Base across the stream.
• Model-only adapts fast but forgets. "Model only" reaches 53.98 on ScienceWorld after stage 1 and 73.72 on ALFWorld after stage 2, but its ScienceWorld score drops to 22.18 after ALFWorld training and finishes at 24.57 — a 29.41-point decline. HaMo finishes at 52.43, exceeding Model only by 27.86 points and matching its own first-stage score.
• The retention benefit comes with a tradeoff. Model only beats HaMo on final ALFWorld (69.34 vs 66.79), while HaMo is slightly better on WebShop (59.64 vs 58.74); HaMo does not dominate parameter-only adaptation on every domain.
• Training order matters. Final-stage totals (sum of the three domain scores, a descriptive aggregate rather than a pooled success rate) are 178.86 for S → A → W, 188.41 for A → W → S, and 181.94 for W → S → A. W → S → A gives the highest final ALFWorld and WebShop scores (74.09 and 60.43), and in that order ALFWorld improves from 66.79 to 73.72 and WebShop from 54.99 to 60.43 — but ScienceWorld falls from 55.19 to 47.42.
• Benefits hold across model scales. HaMo remains effective with Qwen3 backbones of 1.7B, 4B, and 8B parameters on the S → A → W stream; larger backbones generally show stronger acquisition of newly introduced tasks.
• Case study (WebShop). For a request of chocolate-coated, caramel-flavored wafers, 4.41 ounce, under $60, Base bought a hazelnut product and scored 25 in 3 actions; HaMo noticed the candidate's "Triple Dark" title did not establish a caramel match, checked the available flavor options, verified the current $60 budget (rather than the reference card's $70 limit), and scored 100 in 5 actions. The authors caution that the two option clicks (click[caramel], click[4.41 ounce]) produced no visible observation change, so the case does not isolate what caused the reward gain.

Methodology in Plain English

1. Turn each successful trajectory into a short strategy note. A large language model distills an expert trajectory into a solution outline; that outline is packaged with the task description and the initial observation into an "experience card." Cards are added to a persistent memory that accumulates across stages, and each card maps back to its source trajectory.
2. Pick a small, non-redundant set of trajectories to train on. Because many trajectories encode the same strategy, the authors embed all cards from the new stage and, for each card, retrieve its three nearest neighbors while excluding itself. Reversing that relation shows which cards are retrieved often (broadly reusable). They then greedily add cards that cover queries not yet covered, stop when marginal coverage hits zero, and use the knee of the resulting coverage curve to set the training budget for that stage. Only the selected cards' source trajectories become training "anchors"; everything stays retrievable.
3. Train the model to use the retrieved strategy. The training set mixes new anchors with replayed trajectories from earlier stages. Each example is formatted exactly as it will be at test time — request, initial observation, injected retrieved experience — and the model is trained by experience-conditioned behavioral cloning on expert actions given the evolving interaction history.
4. Keep the model cheap to update. Only a single LoRA adapter is trained (backbone frozen, adapter carried over from the previous stage), so parameter cost stays fixed. Training used 3 epochs at a learning rate of 2×10⁻⁴ on NVIDIA A800 GPUs with 80 GB memory; inference uses a decoding temperature of 0.7 and caps episodes at 50 agent–environment turns.
5. Evaluate under two streams. Within-domain: the ScienceWorld training set is split into four equally sized, sample-disjoint subsets that each cover all task types, with the full test set evaluated after every stage. Cross-domain: sequential training on ScienceWorld, ALFWorld, and WebShop, evaluating all domains seen so far after each stage. Metrics are each environment's native score on a 0–100 scale.
6. A feedback option exists but was not used. The design allows releasing memory entries for trajectories already internalized; this was disabled in all reported experiments, so all cards remained available across stages.

Why This Matters

• Research impact: The paper reframes agent continual learning as a division of labor across timescales rather than a choice between memory and weights, and it introduces retrieval coverage (not just similarity or frequency) as the criterion for choosing what deserves a parameter update. Its component ablation also documents a concrete tradeoff — parameter-only adaptation wins on the most recently learned domain while losing heavily on earlier ones.
• Web agents: Shopping, booking, form-filling, and information-seeking workflows repeat with different products and constraints; the WebShop case study is exactly this pattern, where a stored strategy reminds the agent to verify options and budget before purchasing.
• Household and embodied assistants: ALFWorld-style tasks (locating, moving, cleaning, heating, cooling objects) are naturally repetitive, and the paper's mechanism lets a successful procedure inform a new home layout without retraining from scratch.
• Scientific and procedural workbenches: ScienceWorld tasks require manipulating objects and running experiments; accumulated strategy cards could shorten ramp-up on similar protocols while fine-tuning gradually builds execution skill.
• Industry relevance: Because only one LoRA adapter is trained and the backbone stays frozen, the approach sidesteps maintaining a growing library of per-task checkpoints; the harness is cheap to extend and the internal training set is deliberately compact, which matters for teams running agents in production where retraining cost and regression on old skills are both costly.

Future Directions

• Adaptive experience use: deciding when to retrieve a strategy, when to reject an unsuitable card, and when to refresh guidance mid-execution — using retrieval confidence, observed task progress, and explicit checks of strategy preconditions, so misleading retrieved experience can be resisted.
• Isolating which component does the work: the authors note their ablation compares full HaMo against "Model only" (which removes both the harness and replay) and therefore does not separately attribute gains to anchor selection, replay, or experience conditioning; controlled ablations of these, plus decoding temperature, are called for.
• Stronger evaluation: repeated runs and longer task streams to measure variability and persistent forgetting, plus end-to-end cost measurements to expose the performance-versus-memory-maintenance tradeoff — none of which are reported in this paper.
• Beyond text-only benchmarks: extending to multimodal agents and changing web environments to test whether experience-conditioned execution transfers outside the text-based ScienceWorld, ALFWorld, and WebShop settings.

Target Audience

Researchers and practitioners working on LLM agents, agentic memory, and continual/lifelong learning — particularly those building multi-step Web or embodied agents who need experience reuse without catastrophic forgetting. It is also useful for engineers deciding between retrieval-based memory and fine-tuning, since the component ablation quantifies the tradeoff between acquisition and retention, and for readers interested in data-efficient continual fine-tuning (single LoRA adapter, compact anchor selection, replay).

Authors’ abstract

Long-horizon agents continuously encounter reusable procedural knowledge, yet existing systems typically rely either on model adaptation or external memory, trading persistent learning for rapid reuse. This limitation is especially important for Web agents that repeatedly execute related multi-step workflows. We propose HaMo, a Harness-Model collaborative framework that combines fast non-parametric experience accumulation with slow parametric adaptation. HaMo distills trajectories into strategy-level experience cards for immediate retrieval, while an experience-conditioned continual learning objective trains the model to interpret, ground, and execute retrieved strategies across changing contexts. A retrieval-coverage-based selection mechanism further identifies compact and complementary trajectories for parameter updates. Experiments show that HaMo outperforms harness-only and model-only adaptation under within-domain and cross-domain shifts, improving both retention and adaptation. These results highlight the importance of jointly preserving procedural experience and internalizing the competence to transfer and execute it across tasks.