EMHO: EMbodied Agent Harness Optimization via Experience Traces

AI summary

Overview

• Research area: Embodied AI / autonomous LLM-agent self-improvement, specifically the design of the "harness" (planning logic, context and memory, tool orchestration, output validation, failure recovery) that surrounds a vision-language model acting in physical or simulated environments.
• Technical level: Advanced. The core idea is easy to state, but the paper assumes familiarity with vision-language agents, benchmark search/held-out splits, and iterative prompt/code optimization loops.
• Scope (one sentence): The paper proposes EMHO and EMHO-Merge, two methods for automatically revising an embodied agent's harness from execution traces while keeping the underlying model frozen, evaluated on EmbodiedBench navigation and manipulation tasks with Qwen3.5-9B and Qwen3.8-27B-FP8.

What This Paper Is About

Most work on embodied agents tries to improve the underlying vision-language model through training. This paper instead treats the surrounding software "harness" — the logic that decides how the agent plans, what context and memory it sees, when it calls vision tools like depth or segmentation, and how it reacts to failures — as the thing to be optimized, even though the model itself stays frozen. The core problem is that embodied environments give only sparse, weakly diagnostic feedback (an action fails, an episode ends unsuccessfully), so it is hard to tell which earlier decision caused a failure and how the harness should be fixed; EMHO addresses this by mining full execution traces plus the history of past harness edits, and EMHO-Merge extends this to a single shared harness covering several subtasks with conflicting needs.

Key Contributions

1. EMHO, a self-evolving harness optimizer. The paper frames harness optimization as searching over a harness space to maximize expected episode success, and implements an inner loop that collects experience traces (observations, plans, tool interactions, actions, environmental feedback, outcomes) plus an outer loop that proposes targeted harness edits using both trace history and the record of previously accepted and rejected harnesses.
2. Optimization beyond skills and recovery prompts. Unlike prior embodied self-improvement work that targets reusable skills, task-specific policies, or recovery mechanisms, EMHO edits the broader interaction procedure: progress monitoring, context/memory construction, vision-tool selection, visual grounding, output validation, and failure recovery.
3. EMHO-Merge for shared harnesses across subtasks. It compares each candidate harness against its parent, labels episodes that flip from failure to success as gains and episodes that flip from success to failure as losses, then inspects paired trajectories to find observable conditions separating help from harm. Each original revision is allowed at most one evaluated repair, proposed in the next iteration.
4. Empirical and qualitative evidence of behavior change. Results on EmbodiedBench navigation and manipulation for two model sizes, including comparisons against frontier-model one-shot harness revisions and skill-only optimization, plus a qualitative breakdown of recovery edits versus non-recovery edits (progress monitoring, tool orchestration, perception and grounding).

Main Findings

• Gains over the initial harness on both splits and both model sizes. For Qwen3.5-9B navigation, average success rises from 50.0 (initial, search) to 63.0 with EMHO and from 48.7 to 56.7 on held-out. For Qwen3.8-27B navigation, search goes from 54.7 to 67.3 and held-out from 66.0 to 75.3.
• Manipulation improves substantially. For Qwen3.5-9B manipulation, search rises from 9.7 to 18.3 and held-out from 10.7 to 21.0; for Qwen3.8-27B, search rises from 23.9 to 44.2 and held-out from 26.4 to 39.4.
• Headline improvement figures. The introduction reports gains of up to 23.3% in navigation and 33.3% in manipulation over the initial harness (the 33.3% figure corresponds to the manipulation Base subtask in the skill-only comparison table).
• Gains extend past the easiest subtasks. Improvements appear across Base, Common, Complex, Visual, Long (navigation) and Base, Common, Complex, Visual, Spatial (manipulation), not only on Base. The evolved 9B navigation harness reaches performance competitive with larger published models on the search set (EMHO 9B search average 63.0 versus Claude-4.6-Sonnet 62.2 and GPT-5 56.8).
• Beats frontier-model one-shot harness revision in the paper's account. One-shot revisions authored by Claude Opus-5 and GPT-5.6 Sol improved the initial harness but, the authors report, EMHO performs better across both domains. The strongest one-shot baseline for the 27B model scores 67.7 navigation and 27.8 manipulation on average, compared with EMHO's 67.3 search / 75.3 held-out navigation and 44.2 search / 39.4 held-out manipulation.
• Larger average gains than skill-only optimization. On navigation, EMHO gains +16.7, +16.7, +13.3, +13.3, +3.3 across Base/Common/Complex/Visual/Long versus skill-only's +10.0, +10.0, 0.0, +10.0, +3.3. On manipulation, EMHO's gains are +33.3, +29.2, +8.3, +5.6, +25.0 versus skill-only's +16.7, +8.3, +8.3, +11.1, +8.3 (skill-only is higher on the Visual manipulation subtask, +11.1 versus +5.6).
• The harness changes more than recovery behavior. EMHO modifies how the agent detects that valid-but-ineffective action sequences are stalling: sequences of at least five consecutive rotations become less frequent in navigation, identical replans drop from approximately 21% to 18%, and repeated grasps at the same coordinates drop from approximately 30% to 26% in manipulation.
• Tool orchestration shifts measurably. After blocked moves in navigation, depth-estimation use rises from approximately 70% to nearly 100%, and the fraction of valid subsequent actions rises from approximately 44% to 68%.
• Grounding improves. The fraction of scene descriptions listing at least three objects rises from approximately 8% to 22%, and incorrect object selection in initial plans falls from 50% to 34% in spatial-relation tasks; replanning traces more frequently identify the current execution phase, reaching approximately 21% in some subtasks.
• EMHO-Merge mitigates cross-subtask trade-offs. On the navigation search sets of three pooled subtasks, EMHO-Merge averages 67.8 versus EMHO (Combined) 61.1, EMHO (Separate) 65.6, and Initial 54.4; on manipulation search, EMHO-Merge reaches 39.8 versus Combined 34.7 and Initial 25.9, while Separate remains highest at 47.2. On held-out manipulation, EMHO-Merge (33.8) trails Separate (40.7) and the initial harness on one subtask, and on navigation held-out it matches Separate at 75.6.
• Edit types evolve over iterations. Early proposals are dominated by recovery edits; non-recovery edits gain share only after the harness has explicit failure responses. Recovery rules also become more conditional over generations, progressing from "do not repeat the blocked action" to direction-, heading-, and clearance-dependent responses.
• Overfitting is an acknowledged risk. In several runs, later iterations maintain or improve search performance while held-out performance plateaus or declines, so the best search harness is not always the best held-out harness.

Methodology in Plain English

The researchers treat the harness as a set of editable files — for manipulation these cover planning, vision and tool use, memory and context, and validation — and leave the vision-language model untouched.

• Inner loop (collect experience). The current harness is run on a search split of the benchmark. Each episode is recorded as a trace containing the task, a sequence of observations, generated plans, optional tool interactions, executed actions, environmental feedback, and the final success/failure outcome.
• Outer loop (revise the harness). An optimizer receives the current parent harness, all traces gathered so far (including from rejected candidates), and the full history of past harness edits with their scores. It proposes a candidate revision, which replaces the parent only if it scores strictly higher on the search set. Proposals must be general and non-episode-specific; the optimizer instructions forbid using held-out data, hidden simulator state, shortest paths, or hard-coded episode solutions, and ask the optimizer to cite search evidence.
• EMHO-Merge (shared harness). When one harness must serve several subtasks, EMHO-Merge contrasts the newest candidate with its parent to identify gained episodes (failure to success) and lost episodes (success to failure). If both exist and the revision's one repair attempt is unused, the optimizer is asked to inspect at least one gained and one lost trajectory, find a visible condition that separates benefit from harm, and either repair when/how the behavior triggers or propose a genuinely new mechanism instead.
• Setup. Evaluation uses EmbodiedBench navigation (Base, Common, Complex, Visual, Long) and manipulation (Base, Common, Complex, Visual, Spatial). Models are Qwen3.5-9B and Qwen3.8-27B-FP8, with Depth Anything 3 for depth estimation, SAM 3 for segmentation, and zoom tools. The harness is initialized from the original EmbodiedBench implementation, the benchmark is split evenly into disjoint search and held-out sets, and 10 optimization iterations are run per task. Baselines include GPT-5, Claude-4.6-Sonnet, Gemini-2.0-flash, InternVL3-78B, gemma-3-27b-it, one-shot harness revisions from Claude Opus-5 and GPT-5.6 Sol, and skill-only optimization.

Why This Matters

The paper argues that a better embodied agent does not always require a better model: substantial behavioral gains can come from changing the surrounding system, which is attractive when retraining or fine-tuning is expensive or infeasible. It also moves agent self-improvement out of coding and command-line settings, where failures are explicit, into embodied settings where feedback is sparse and delayed, and it offers a concrete procedure — paired gain/loss trajectory analysis with a single repair per revision — for handling conflicts when one policy must serve many tasks.

Real-world applications suggested by the paper's framing:

• Robotics manipulation pipelines where a frozen vision-language model drives grasping, stacking, and container placement, and the harness must decide when to query depth for height-sensitive steps.
• Navigation and mobile agents in indoor or outdoor environments where blocked movement and repeated rotations are the dominant failure modes and recovery logic matters more than raw perception quality.
• Assistive or service robots that need to keep operating when individual actions are valid but collectively unproductive, using progress monitoring rather than waiting for an explicit error.
• Multi-task deployed agents where a single shared operating policy must cover several task families, which is exactly the setting EMHO-Merge targets.

Industry relevance: the approach requires no model retraining and no gradient access, so it can be applied on top of hosted or quantized models; the paper's own use of an FP8 27B model and off-the-shelf tools (Depth Anything 3, SAM 3, zoom) reflects a deployable configuration rather than a research-only one. It also complements the emerging line of coding-agent harness optimization work (Meta-Harness, Self-Harness) by extending harness-level self-improvement to embodied domains.

Future Directions

• Generalization-aware harness selection. The authors identify search/held-out divergence — search performance improving while held-out plateaus or declines — and call for considering generalization when choosing among harness iterations. A validation-based selection rule or explicit regularization of edits is a natural next step.
• Stronger cross-subtask merging. EMHO-Merge beats the pooled-experience baseline but still trails separately optimized harnesses on some held-out splits (for example, manipulation held-out 33.8 versus 40.7 for separate optimization), so the trade-off between a single shared harness and per-subtask harnesses is not fully resolved.
• Richer diagnostics from sparse feedback. The paper notes that embodied feedback is weakly diagnostic compared with failed tests or exceptions in coding environments; better automatic failure attribution from traces would likely improve both the recovery and non-recovery edits the optimizer can propose.
• Extending optimization scope. The current work modifies instructions and runtime components across four functional areas; whether the same trace-guided loop scales to more harness components, to more benchmarks, or to longer optimization budgets is left open. The paper reports 10 iterations per task and does not report computational cost or wall-clock time for optimization.

Target Audience

Researchers and practitioners working on LLM-based agents, embodied AI, and robot learning who want to improve agent behavior without retraining the underlying model; engineers building harnesses or scaffolding around frozen vision-language models for navigation and manipulation; and readers interested in automated, trace-driven self-improvement of agent systems who want an embodied counterpart to coding-agent harness optimization. Readers without background in agent architectures or vision-language models will find the qualitative sections accessible but the optimization procedure and benchmark tables demanding.

Authors’ abstract

Improving embodied agents often focuses on optimizing the underlying model through training, while the surrounding agent harness that controls planning, context, and tool use is typically engineered. We ask whether this harness can instead improve itself directly from experience traces under sparse environmental feedback. We propose EMbodied Agent Harness Optimization (EMHO), a self-evolving framework that keeps the embodied model frozen and iteratively revises its harness by analyzing execution trajectories and prior harness history. EMHO optimizes beyond skills or recovery prompts, modifying how the agent monitors progress, uses vision tools, grounds observations, and responds to failures. To support multiple subtasks with a single harness, we introduce EMHO-Merge, which addresses trade-offs in jointly optimizing a single shared harness across subtasks by using episode-level gains and losses to guide evidence-supported refinement of when and how revised behaviors are applied. We evaluate EMHO on EmbodiedBench across navigation and manipulation tasks, and EMHO consistently improves task success for both Qwen 9B and 27B models. Qualitative analysis shows that EMHO goes beyond recovering from failures and unproductive actions to reshape how the embodied agent interprets and interacts with its environment.