Auriga Technical Report: A Task-Agnostic Agent Harness for Zero-Shot Embodied Manipulation

AI summary

Overview

Research area: Robotics (cs.RO) — embodied manipulation with general-purpose vision-language models, agent harness design, and zero-shot evaluation on simulated manipulation benchmarks (RoboDojo, RoboCasa365).

Technical level: Advanced. The paper assumes familiarity with robot manipulation benchmarks, vision-language models, end-effector control, and agent tool-use loops, and uses formal notation for its decision process. The core argument, however, is conceptually simple and readable by a motivated non-specialist.

One-sentence scope: The paper introduces Auriga, a task-agnostic software harness that lets a frozen general-purpose model plan, preview, execute, and recover from robot manipulation tasks within a single episode and with no target-task practice, and reports its results on RoboDojo and RoboCasa365.

What This Paper Is About

Vision-language models can look at a scene, understand an instruction, and decide what to do, but when they are wired directly into a robot arm they fail at the physical part: grasps miss by millimeters, objects slip, and the arm clips container rims. On RoboDojo, the paper measures this "planning–execution gap" directly — GPT-6 Astra succeeds in 76.2% of episodes when objects are moved for it without the arm, but only 22.5% when it has to drive the arm through the benchmark's reference interface.

The goal is to close that gap without giving the model any task-specific help. Auriga is the harness between model and robot: it measures the scene in 3D, compiles each requested motion into a trajectory the model can inspect before anything moves, monitors the motion while it runs, and stops the robot with a quantitative report when reality diverges from the plan — leaving diagnosis and recovery to the model, inside the same episode, with no resets.

Key Contributions

1. A task-agnostic harness with an explicit task belief. Auriga contains no task-specific code, prompts, heuristics, or privileged simulator state. A single persistent model session ("Cortex") maintains a compact belief B_t — a plan, an attention list of open uncertainties, and a ledger of premises marked pending, supported, contradicted, or unknown — while full interaction logs live off-context in a retrievable evidence record E_t. No physical action is admitted until a belief has been submitted.

2. A single admission gateway with preview–execute–verify. Every motion, whether a direct tool call or generated inside a model-written Python program, passes through one gateway. With simulation paused, the gateway compiles a multi-stage dual-arm sequence into a trajectory preview showing gripper paths, per-arm step cost, and clearance warnings. On acceptance the robot replays exactly that trajectory (no replanning), and per-stage tolerances plus gripper checks and model-placed checkpoints halt execution on divergence.

3. Quantitative 3D perception and finger-referenced action interfaces. Camera calibration, VGGT-Omega depth, and SAM 3.1 segmentation supply 3D object estimates, ray–plane intersection with height sensitivity, and two-view triangulation, plus in-hand slip measurements and warnings when a motion re-enters a past failure region. On the output side the model sets goals at the finger contact region rather than the wrist, chooses tolerances per stage, and coordinates both arms on a shared timeline.

4. A measured separation of cross-task tool development from single-episode execution. Manipulation failures across 13 development iterations (#0 to #12) were converted into shared, reusable capabilities rather than per-scene fixes; the harness was then frozen and run under a strict single-episode zero-shot protocol (no target-task exploration, no demonstrations, one execution per case, no resets) on 6,300 RoboDojo episodes and 250 RoboCasa365 episodes.

Main Findings

• The bottleneck is execution, not decision-making. On a 210-episode diagnostic (five episodes for each of the 42 RoboDojo tasks), a privileged action that moves objects along model-chosen waypoints without the arm yields 76.2% SR, versus 22.5% SR for the reference end-effector harness RoboProbe and 59.2% SR for Auriga — a 68.3% closure of the planning–execution gap in SR (70.6% in Score).

• First place on the RoboDojo leaderboard. Across all 42 tasks and five dimensions (6,300 episodes over three seeds), Auriga with GPT-6 Astra reaches 64.0 Score and 57.7% SR, more than doubling the GPT-6 Astra reference harness (29.0 Score, 22.5% SR), ranking first overall among learned policies and agent methods, and leading all 12 columns of the agent block.

• Per-dimension diagnostic detail. Auriga's diagnostic averages were Generalization 43.3% SR / 46.2 Score, Precision 50.0% / 55.6, Long-Horizon 45.0% / 59.5, Memory 90.0% / 90.5 (over 29 episodes, since one episode has no counterpart in the reference results), and Open 67.5% / 71.2. The same diagnostic shows the reference harness at 2.5% SR on Precision and 7.5% on Long-Horizon.

• Development history is documented step by step. The harness evolved from 21.5 Score (4.0% SR on the 50-case diagnostic set) at the initial base-model loop (#0) to 45.5 Score (36.2% SR across all 42 tasks) at #4 and 64.0 Score (57.7% SR) at the frozen #12, with gains of +16.8 Score in Generalization, +5.6 in Precision, +29.5 in Long-Horizon, +30.8 in Memory, and +9.8 in Open from #4 to the final harness. Named milestones include motion preview (#2, 26.2 Score), unified dual-arm actions (#3, 30.0), verbatim task specifications (#4, 45.5), structured history retrieval (#6, 46.6), ROI zoom (#7, 49.9), stage tolerances and obstacle checks (#9, 53.4), explicit motion control (#11, 58.8), and VGGT-Omega / SAM 3.1 perception (#12, 64.0).

• Robustness under scene randomization. Across the 12 Generalization tasks, learned policies drop by up to 33.9% in SR on randomized layouts, while Auriga ranks first on both standard scenes (52.3% SR) and random scenes (45.6% SR) and achieves the highest generalization score (G = 46.3).

• Transfer to a second benchmark without retraining. On the 50 Target50 mobile manipulation tasks in unseen RoboCasa365 kitchens (250 episodes), the same frozen harness reaches 73.2% Overall SR — 92.2% Atomic-Seen, 67.5% Composite-Seen, 57.5% Composite-Unseen — beating VLA-assisted agents RPent (59.2%) and PyRUA-Lean (56.4%) by 19.4% and 23.1% on composite tasks, with no learned policy and no cross-episode memory.

• Why common paradigms leave the gap open. Step-wise or chunked action interfaces never show the route of a motion before the model commits to it, so collisions appear only after the arm has moved. Code-as-policy hands judgment to a script that cannot see the scene, so normal program exit does not mean the intended contact succeeded. Auriga keeps judgment with the model by describing a complete action sequence per call, previewing it, and checking measured outcomes afterward.

Methodology in Plain English

The setup. A general-purpose model (GPT-6 Astra) is given a task description and access to a fixed set of tools. The model cannot see the simulator's hidden state — only calibrated RGB images, the measured robot state, and the public camera calibration and robot model. The benchmark evaluator, not the model, decides success.

Three layers. A Cortex session makes every task decision for the whole episode. A capability layer provides general tools grouped as Measure (observations, object estimates, geometry queries), Act (preview and execute multi-stage dual-arm sequences, or let time pass), Compute (run programs), and Recall (revise the belief, retrieve earlier interactions, record operation lessons). An admission gateway is the only path that moves the robot.

Preview before acting. Instead of emitting one small movement at a time, the model describes a whole sequence — approach, close gripper, lift, transport, release. While the simulation is paused, the gateway plans the full trajectory and returns a preview drawn on the camera image: gripper paths, per-arm timing, and estimated clearance against declared obstacles. If any part cannot be planned, nothing executes and the preview says which part failed. If accepted, the previewed trajectory is replayed without replanning, so the motion that reaches the robot is exactly the motion shown.

Verify while acting. Each stage has a target and a tolerance the model sets (defaults: 10 mm position, 0.15 rad orientation). If the reached state deviates beyond tolerance — typically because of contact the planner does not model — the robot stops in place and returns a receipt listing which stages completed, by how much the stage missed, which stages did not run, plus fresh camera views and images of the measured versus previewed route. Completed stages are kept, not undone. The model may also insert checkpoints at moments it considers critical, for example after a lift when the measured gripper opening suggests nothing was grasped.

Three nested loops. A stage loop inside the gateway stops an unsafe or off-target motion. An action loop has the model compare the receipt with its intent and form a cause hypothesis before acting again. A belief loop updates the premises and plan across the episode; if repeated evidence contradicts a premise, the model switches mechanism rather than restating the same hypothesis.

How the harness was improved. Rather than tuning per task, the developers used failures across 13 iterations to diagnose missing shared capabilities — 3D localization, contact checking, in-hand state tracking — and upgraded the tools accordingly. The paper frames this in a self-improvement taxonomy: L0–L2 is within-task trial search (which the authors attribute to HexaShell, EmbodiedEvo, and SEAR), L3–L4 is cross-task experience and tool evolution, and L5 is meta-improvement. Auriga targets L3–L5 during development, then freezes the harness entirely for evaluation.

Why This Matters

Research impact. The paper reframes the embodied-agent problem: instead of asking how to make the model a better policy, it asks what interface and verification layer a general model needs to act competently. The 76.2% versus 22.5% measurement gives the field a concrete, reusable way to quantify a planning–execution gap, and the frozen-harness, single-episode protocol is a stricter yardstick than multi-trial tuning on a fixed layout. The separation of cross-task development (L3–L5) from within-episode adaptation is a clean conceptual contribution.

Real-world applications:

• Warehouse and logistics picking — millimeter-scale grasp misses and dropped objects are the dominant failure mode; a preview-plus-verification harness addresses exactly that.
• Home and service robots — mobile manipulation in unfamiliar kitchens and cluttered rooms, where "unseen layout" is the normal condition rather than a test setting.
• Laboratory and pharmaceutical handling — precise placement into racks, tubes, and containers with tight clearances, where a stop-and-report failure is far safer than a silent miss.
• Manufacturing and assembly — insertion tasks (a jammed pin insertion is cited as a motivating failure), where per-stage tolerances and model-set checkpoints map onto quality gates.
• Field and inspection robots — operation in single-attempt situations where a reset is physically impossible.

Industry relevance. The result matters commercially because it requires no fine-tuning: a frozen general model plus a well-designed harness beats VLA-assisted agents on RoboCasa365 composite tasks by up to 23.1%. For companies building on frontier models, this suggests the integration layer — calibrated perception, admission control, quantitative error feedback — is a first-class engineering asset, not glue code. The emphasis on single-episode, no-reset evaluation also mirrors deployment economics better than trial-and-error benchmarking.

Future Directions

• Whether the harness transfers to other models and real hardware. All headline results use one model, GPT-6 Astra, in simulation; the paper does not report cross-model ablations or physical-robot results, so the degree to which the harness is model-agnostic or sim-to-real robust is an open question.

• Learning the tolerances instead of setting them. Tolerances are chosen per stage by the model (defaults 10 mm and 0.15 rad). Whether these could be predicted from context, task type, or observed contact history — and whether the harness could improve them across development iterations without touching the frozen evaluation version — is unexplored.

• Closing the remaining ~32% of the gap. Auriga recovers 68.3% of the planning–execution gap in SR; the residual, especially on Precision (50.0% SR) and Long-Horizon (45.0% SR) in the diagnostic, points to capabilities still missing — likely deformable-object contact, richer force sensing, and longer-horizon state tracking.

• Pushing beyond preview-based verification. Divergence detection works by comparing the measured path to the previewed one, which catches the consequence of unmodeled contact rather than predicting it. Incorporating contact or force models into the preview, and extending the belief ledger into a more formal mechanism for revising task-level explanations, are natural next steps.

Target Audience

Robotics and embodied-AI researchers working on manipulation, agent architectures, and tool-use with foundation models; benchmark designers interested in zero-shot and single-episode evaluation protocols; and engineers at robotics or automation companies who are integrating frontier vision-language models with physical hardware and need to know where the failures actually come from. Readers mainly interested in model training will find little here — the model is fixed and the contribution is the interface, measurement, and verification layer around it.

Authors’ abstract

Vision-language models can interpret instructions and plan manipulation tasks, yet struggle when acting directly as robot policies. On RoboDojo, a large planning-execution gap separates high-level task decisions from physical execution: removing arm execution raises GPT-6 Astra's success rate from 22.5% under the reference control interface to 76.2%. We present Auriga, a task-agnostic agent harness that closes 68.3% of this gap in zero-shot manipulation without target-task…