The Last Mile to the Real World: Agent-Native Primitives as Reusable Building Blocks for Solving Real-World Tasks

AI summary

Overview

Research area: LLM agents, tool use, and harness engineering (Natural Language Processing / agent systems). Technical level: Advanced — the paper gives formal definitions, equations, and reports results across more than twenty benchmarks and eight "primitive families." One-sentence scope: The paper proposes agent-native primitives (reusable resources each equipped with a resident LLM) and HEELO, a harness engineering framework that composes, verifies, revises, and validates such primitives per task, evaluated on eight task families, more than twenty benchmarks, and 46 real-world tasks on the heelo.dev platform.

What This Paper Is About

Frontier models in their own harnesses still fail many real-world tasks: on Terminal-Bench 4.0 at maximum reasoning effort, GPT-5.6 Sol with Codex resolves 37.3% of tasks and Claude Opus 5 with Claude Code resolves 51.8%. The authors locate the failure in the "last mile to the real world" — the step from reaching a resource (a tool, code component, proof method, document) to applying it: grounding arguments, passing results to the steps that need them, and tracing a failure back to the work it implicates. Their goal is to make the competence to apply a resource travel with the resource itself, and to build each task's harness from a library of such units rather than rebuilding that competence per task.

Key Contributions

1. Agent-native primitives under one definition. A primitive pairs a reusable resource (tool schema and function, code component, proof method, source model, multi-agent pattern, repository files) with a resident LLM that serves as its natural-language interface, applies the resource in context, and sends messages of requirements or task-relevant information to other primitives. The authors build libraries of 25,519 Tool Primitives (16,464 wrapping live APIs), 1,424 validated Code Primitives, and 82 Math Primitives whose Lean artifacts compile.
2. HEELO, one framework unifying eight primitive families. Three mechanisms build and run each task's harness: library selection and composition (build a task-specific graph of the few primitives the task needs), evidence-based execution and improvement (verify against task-environment evidence and revise only the primitives a failure implicates, within a round budget), and validation and publishing (admit new or improved primitives to the library only after validation).
3. Eight instantiated families evaluated on more than twenty benchmarks: Tool, Code, Math, Knowledge, Agent, Dev-, Navigation, and Harness Primitives, each built as the solution to one kind of real-world task, with a new benchmark of 522 repository-reconstruction tasks drawn from real Python projects in seven domains.
4. heelo.dev, a platform of real-world actions (from flights, calendars, email, and browser automation to publishing pages and reading invoices) exposed to agents such as Claude Code, Codex, and Cursor through an MCP server, with 46 real-world tasks covering 26 capabilities of four end-to-end loops.

Main Findings

• Dev-Primitives raise Terminal-Bench 4.0 resolution from 37.3% under Codex to 55.5% on the same backbone, an 18.2-point gain.
• Harness Primitives composed per task beat the mini-SWE-agent seed harness on Pass@1: 73.0% to 80.5% on 100 SWE-bench Verified tasks, and 60.0% to 72.2% on 45 Terminal-Bench 2.0 tasks, above the 79.0% and 56.7% of Codex. Composition cost is 2.7% to 4.3% of one actor attempt.
• Tool Primitives raise ToolBench pass rate from 55.8% to 75.1%. On Qwen3-8B they lead every reference model on τ²-Bench at up to 7.7× lower cost than GPT-5.4, and complete 2.6–3.8× as many of 50 curated real-world tasks as GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro calling the same tools.
• Code Primitives raise repository delivery from 0.3180 to 0.5134, surpass Claude Code on its backbone by 56.2%, and keep 95.1% of their delivery at 24.0% lower cost under GPT-OSS-20B adaptation and diagnosis.
• Agent Primitives raise mean accuracy over eight math, code, and QA benchmarks from 58.8% for a single agent to 75.3%; on Qwen3-8B they exceed the strongest prior multi-agent system by 7.5 points with roughly 3–4× fewer tokens than a text-based system.
• Navigation Primitives raise GAIA-Text-103 accuracy from 45.6% to 58.3%, a 12.7-point lead over the strongest baseline.
• Math Primitives place an annotated reference method in the top five for 95.0% of held-out Putnam problems, against 84.2% for the strongest reranker (10.8 points in Putnam hit@5).
• Knowledge Primitives with GPT-5.6 Luna gain 14.3 points over Hybrid RAG on average.
• The gain lies inside the primitive. Tool Primitives more than double the ToolBench pass rate of budget-matched self-refinement over raw schemas, and active Navigation Primitives grow 1.23× while distractors grow the information space 16× around the same evidence.
• The task decides which primitives run. Companion studies bound activation by the task's demand, however large the library, and identify when per-task composition beats the best fixed harness.
• Real-world results on the 46 heelo.dev tasks. At medium reasoning effort, Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex complete 9 tasks each on their own (mean task scores 28% and 25%); connected to HEELO through the MCP server, Claude Code completes 39 and Codex 35 (scores 90% and 83%). Per completed task, Claude Code with HEELO needs 3.4 minutes and $0.80 instead of 6.3 minutes and $1.24, and Codex with HEELO 3.5 minutes and $0.42 instead of 8.0 minutes and $0.65 — 1.9× and 2.3× less time and 1.55× lower cost.
• Baseline shortfalls that motivate the work: on the four stages of DevOps-Gym the two frontier pairs average 49.9% and 52.1% success, and in the airline domain of τ²-Bench Claude-4.6-Sonnet succeeds in all four trials on half of the tasks.

Methodology in Plain English

The authors start from the observation that today's agents can find tools, code, and documents but must figure out how to use them themselves, and that this know-how is rebuilt from scratch for every task. Their fix is to give each resource its own small LLM that reads the resource, answers requests in natural language, fills in the details, and tells other resources what it needs from them. Resources stop being passive and become participants that talk to each other.

On top of this, HEELO treats building a task's harness as a construction problem. Given a task, it looks through the library of primitives (by interface or by asking each candidate whether it applies), assembles the few the task needs into a graph, and runs them. After a round, it checks the result against real evidence from the task's environment — tests, compiler output, a proof checker, development runs — and if something fails, traces that failure to the specific primitives involved and redoes only that work, within a bounded number of rounds. Between tasks, a separate path admits new or improved primitives into the library only once they pass validation, so the library grows without contaminating tasks in progress.

Because different domains stress the "last mile" differently, the authors instantiate the same recipe eight times (tools, code repositories, formal proofs, knowledge sources, multi-agent patterns, software engineering, navigation, and harnesses themselves), each with its own composition and verification details, and evaluate each family on its domain's benchmarks plus a live platform of 46 real-world tasks.

Why This Matters

Impact on research. The paper reframes agent capability as a property of reusable, validated units rather than of a monolithic harness or prompt. It supplies one definition spanning tools, code, proofs, retrieval, and multi-agent patterns; it separates running a task from validating what enters the library; and it reports that per-task composition can beat the best fixed harness and the strong Codex/Claude Code baselines on the same backbones.

Real-world applications (as demonstrated or described in the paper):

• Software engineering and terminal work: repository construction, issue repair, and Terminal-Bench-style tasks, with delivery measured by the repository's own tests.
• Business operations on heelo.dev: publishing pages, sending outreach, reading invoices, scraping sources, booking flights, and managing calendars.
• Formal mathematics: selecting and applying proof methods whose Lean artifacts compile.
• Knowledge-intensive and large-information-space work: answering from an authoritative source, and GAIA-style navigation over large document collections.

Industry relevance. The heelo.dev results show a practical deployment pattern: agents such as Claude Code, Codex, and Cursor connect to a primitive platform through an MCP server and complete far more tasks (39 and 35 versus 9 each) at lower time and cost per completed task (1.9× and 2.3× less time, 1.55× lower cost). The composition overhead of 2.7% to 4.3% of one actor attempt suggests per-task harness assembly is affordable in production loops.

Future Directions

• Adding new primitive families. The authors state that "a new kind of task can join as a new primitive family," so extending the eight families to further domains is the natural next step; the paper does not report results for such additions.
• Scaling and governing the library. The validation-and-publishing path (Eq. 7) governs growth, but how libraries of the reported scale (25,519 Tool Primitives, 1,424 Code Primitives, 82 Math Primitives) should be curated, deduplicated, and kept current as sources change is left open.
• Formal guarantees for composition. The paper cites companion studies that bound activation by task demand and identify when per-task composition beats the best fixed harness; extending these results to all eight families and to real-world tasks with fixed pass criteria is an open direction.
• Broader evaluation of the real-world platform. The 46 heelo.dev tasks cover 26 capabilities; the paper does not report results beyond that set, leaving open how the approach behaves on larger and more varied real-world task suites.

Target Audience

Researchers and engineers working on LLM agents, tool use, and agent harnesses, especially those building production systems where agents must complete tasks end to end rather than merely call APIs. It is also relevant to practitioners evaluating agent platforms (MCP-based tool access, multi-agent systems, repository-level coding agents) and to readers interested in formal theorem proving or knowledge-intensive retrieval with LLMs. The formal definitions and benchmark density make it most useful to readers with intermediate-to-advanced background in agent systems.

Authors’ abstract

How can an agent handle a real-world task end to end? Frontier models with their own harnesses still fall short on real-world benchmarks: on Terminal-Bench 4.0, GPT-5.6 Sol with Codex resolves 37.3% of the tasks and Claude Opus 5 with Claude Code 51.8%. These tasks fail at the step from reaching a resource to applying it, which we call the last mile to the real world. Existing approaches put the resource within reach but leave the competence to apply it with the caller, so each harness rebuilds that competence. We introduce agent-native primitives, reusable units that equip a resource with a resident LLM that applies it in context and communicates with other primitives, and HEELO (Harness Engineering around Evolving Agent-Native Primitives for the Last Mile to Real-WOrld Tasks), a harness engineering framework that builds each task's harness from a library of such primitives. HEELO composes the primitives a task needs into a task-specific graph, verifies the result against evidence from the environment, revises the primitives a failure implicates, and admits new or improved primitives after validation. We instantiate HEELO with eight primitive families, each built for one kind of task, and evaluate them on more than twenty benchmarks. Dev-Primitives raise Terminal-Bench 4.0 resolution from 37.3% under Codex to 55.5% on the same backbone, Harness Primitives composed per task raise Terminal-Bench 2.0 Pass@1 from 60.0% for the seed harness to 72.2%, Tool Primitives raise ToolBench pass rate from 55.8% to 75.1%, and Navigation Primitives raise GAIA-Text-103 accuracy from 45.6% to 58.3%. On the 46 real-world tasks of our heelo.dev platform, Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex complete 9 each on their own and 39 and 35 when connected to HEELO through our MCP server, with 1.9$\times$ and 2.3$\times$ less time and 1.55$\times$ lower cost per completed task.