Future Horizons

The Future of AI Agents: When Software Can Act, Not Just Answer

Agents are software that acts, and the record since 2023 shows progress concentrating wherever a task can be graded mechanically: SWE-bench Verified is near 96-97%, while open-ended desktop work got a harder benchmark instead.

Gabriele Masetti · 2026-08-27

From Autocomplete to Actors

For most of the last decade, "AI" meant a system that answered. You typed a question, it returned text, and the transaction ended. The current transition is about software that acts: it perceives a state of the world (a screen, a file system, an API response), plans a next step, executes an action, observes the result, and loops until a goal is met.

That perceive-plan-act cycle, wrapped around a language model that can call external tools, is what the industry now means by an "agent." The distinction matters because it changes what can go wrong. A chatbot's worst failure is a bad sentence. An agent's worst failure is a wrong action taken in the world — a file deleted, a flight booked, a customer refunded twice.

The technical seed of this shift is older than the current hype cycle. In October 2022, Princeton and Google researchers published ReAct ("Synergizing Reasoning and Acting in Language Models"), which showed that interleaving explicit reasoning traces with tool-calling actions let a language model track a plan, update it when actions failed, and recover from errors far better than models that only reasoned or only acted.

On the ALFWorld and WebShop interactive benchmarks, ReAct beat imitation- and reinforcement-learning baselines by 34 and 10 percentage points of absolute success rate, using only one or two in-context examples. A few months later, Meta AI's Toolformer (February 2023) showed a complementary idea: a model can learn, in a self-supervised way, when to call a calculator, a search engine, or a translator, simply by checking whether the API call reduces its own prediction error on the surrounding text.

Toolformer's 6.7-billion-parameter model scored 29.4% on the SVAMP math benchmark, nearly triple GPT-3 175B's 10.0% — a small model with the right tool beating a much larger model without one. Between them, ReAct and Toolformer supplied the two ingredients every agent since has combined: a loop that reasons about what to do next, and a mechanism for reaching outside the model's own weights to do it.

The 2023 Hype Cycle and Its Correction

What the research papers proposed carefully, the open-source community shipped recklessly. AutoGPT appeared on GitHub on March 30, 2023, two weeks after GPT-4's release, with a simple pitch: give it a goal, walk away, come back to finished work. It crossed 30,000 stars within 13 days and passed 100,000 soon after — by most accounts the fastest-growing repository in GitHub's history at the time.

The promise was full autonomy. The reality was an agent that could write its own sub-goals but had no reliable way to tell whether it had actually achieved them, so it looped, contradicted its own earlier steps, burned through API budgets, and rarely finished the multi-step tasks it was assigned.

AutoGPT's later evolution away from "autonomous employee" and toward a more constrained workflow-automation tool was not a change of ambition; it was a concession to a problem that turned out to be the central one — an agent that cannot verify its own progress cannot be trusted to run unsupervised.

That correction reframed the field's research question. Instead of "can a model decide what to do," the harder question became "can a system of model-plus-tools be made reliable enough to run for more than a few steps without a human catching the drift." Everything that followed — standardized tool interfaces, sandboxed execution, structured task grading, narrower scoped domains — is a response to that reframing.

Giving Agents Hands: Computer Use, Operator, and MCP

Two infrastructure moves in late 2024 turned "agent" from a prompting technique into a product category. On October 22, 2024, Anthropic introduced a public beta of "computer use" for Claude 3.5 Sonnet: the model could look at a screenshot, decide where to move a virtual cursor, click, and type, letting it operate ordinary desktop software that was never built with an API.

Anthropic was explicit that the capability was rough — scrolling, dragging, and zooming were still unreliable — and it layered in guardrails around sensitive actions like election-related content or interacting with government sites. OpenAI followed with Operator on January 23, 2025 (reaching Pro-tier subscribers in research preview on February 1), built on a "Computer-Using Agent" model that combined GPT-4o's vision with reinforcement-learning-trained reasoning over raw pixels and a virtual mouse and keyboard.

Operator's own lifecycle is instructive: it was folded into the broader "ChatGPT agent" product and the standalone surface was shut down on August 31, 2025 — a little over seven months after launch. The pattern then repeated. OpenAI's no-code Agent Builder, launched in late 2025, was marked deprecated in June 2026 and shuts down on 30 November 2026, about a year after it appeared. Computer use as a category is still being reshaped in real time; nobody, including the labs building it, has settled on the right product surface yet.

The other infrastructure move was less visible in demos but arguably more consequential: the Model Context Protocol. Anthropic open-sourced MCP in November 2024 as a standard way for a model to discover and call tools — file systems, databases, SaaS products — without every integration being bespoke glue code. The pitch was structural: models were "trapped behind information silos," and every new data source needed its own custom connector.

MCP's significance is that it was adopted outside Anthropic, including by OpenAI and Google DeepMind, and then stopped belonging to Anthropic at all. On 9 December 2025 Anthropic donated the protocol to the Agentic AI Foundation, a directed fund under the Linux Foundation co-founded with Block and OpenAI, with support from Google, Microsoft, AWS, Cloudflare and Bloomberg. One vendor's internal plumbing became neutral infrastructure in about thirteen months. Standardizing how an agent connects to a tool doesn't make the agent more reliable, but it does mean reliability engineering can be built once, at the protocol layer, instead of once per integration — the same reason a common port matters more than any single device that uses it.

What the Benchmarks Actually Show

Enthusiasm is easy to generate; measured capability is not. Four benchmarks, read in sequence, tell a consistent story about where agents stand and why.

WebArena, introduced in 2023 to test agents on realistic web tasks (shopping, forums, code hosting, content management), found that the best GPT-4-based agent completed only 14.41% of tasks end-to-end, against a human success rate of 78.24% on the same tasks. That is not a benchmark an agent was close to saturating — it is a benchmark showing that autonomous web navigation was, at the time, mostly broken.

Tau-bench, published in mid-2024 to test agents holding realistic multi-turn conversations with simulated customers while following business policy (in retail and airline settings), is more revealing about why agents fail, not just how often. On its frozen leaderboard, GPT-4o reached 60.4% success on retail tasks and 42.0% on airline tasks at pass^1 (succeeding once); Claude 3.5 Sonnet (October 2024) improved on that to 69.2% and 46.0% respectively.

But the paper's more important number is what happens under pass^k, which requires an agent to succeed on all of k independent attempts at the same task rather than just once: GPT-4o's retail score falls from roughly 60% at pass^1 to around 25% by pass^8. The same agent, given the same task eight times, produces a materially different outcome each time. That instability — not a lack of raw capability — is the core unsolved problem in deploying agents anywhere a business needs a consistent answer.

Tau-bench has since been replaced by its successor, tau-squared-bench, and the headline numbers have travelled a long way. OpenRouter's tau-squared-bench Airline leaderboard, run on 19 September 2026, puts Gemini 3.7 Flash at 80.6%, Claude Fable 5 at 80.0% and Claude Opus 5 at 79.2%, against the 42.0% GPT-4o managed on the airline split in 2024. What has not travelled is what the leaderboard reports: single-run accuracy, with no pass^k column. The number everyone quotes is still a pass^1 number, and nothing that replaced tau-bench has retired its finding that pass^1 flatters an agent.

SWE-bench Verified, a 500-issue, human-validated subset of real GitHub bug-fix tasks introduced by OpenAI and Epoch AI in August 2024, is the benchmark where agentic progress has been most measurable, because "did the test suite pass" is an unambiguous, automatable grading signal — a rarity in agent evaluation. Anthropic's Claude Opus 4 and Claude Sonnet 4 (May 2025) scored 72.5% and 72.7% respectively; Claude Sonnet 4.5 (September 2025) reached 77.2% averaged over ten trials.

Those figures are now historical, and the benchmark is close to saturated. BenchLM's snapshot of 18 September 2026 puts Claude Opus 5 at 96%, Claude Mythos 5 at 95.5% and Claude Fable 5 at 95%; Vals.ai's 1 September 2026 run puts Opus 5 at 97% and reports seven of the 86 models it evaluated at 95% or better. The two runs disagree by a point about the leader, and neither leaves much room above it.

The climb is real, and it is concentrated specifically in the one domain where an agent's output can be checked mechanically rather than judged.

GAIA, a benchmark built to test general-assistant tasks requiring web browsing, multi-step reasoning, and file handling, put the gap in stark terms in its original 2023 paper: human respondents solved about 92% of its questions, while GPT-4 augmented with plugins solved roughly 15%. GAIA's questions are designed to be conceptually simple for a person and difficult for an agent precisely because they require chaining several unreliable steps — a search, a file read, an arithmetic check — where a single mistake anywhere in the chain invalidates the final answer.

The Compounding Error Problem

Put the four benchmarks together and a single mechanism explains most of the gap: errors compound multiplicatively across steps, and most valuable tasks require many steps. If an agent is 95% reliable at any individual action — a generous number for real tool use — the probability it completes a 20-step task without a single mistake is 0.95 raised to the 20th power, or about 36%.

At 90% per-step reliability, a 20-step task succeeds only about 12% of the time. This is exactly what tau-bench's pass^k results measure directly: not "can the agent do the task" but "can the agent do the task the same way every time," and the answer, even for frontier models in 2024–2025, was frequently no.

This is why the domains where agents have made the fastest real progress — coding, and narrowly scoped business workflows — are the domains where either the number of steps can be kept small, or a cheap, automatable check exists at the end (tests pass, a diff compiles, a structured field validates) that lets the system retry rather than silently propagate a bad intermediate result.

It is also why general "computer use" agents, which must complete open-ended, many-step tasks across unpredictable software with no formal success check, remain the hardest and least reliable category. The field's answer has been to raise the bar rather than announce it cleared: OSWorld, whose original paper recorded a best model at 12.24% against a 72.36% human baseline, shipped OSWorld 2.0 on 26 June 2026. Practitioners writing about production browser agents argue the benchmarks were never measuring what actually breaks — DOM selector drift, expired logins, cookie banners and modals, rate-limit cliffs, and actions with no rollback. As one 2026 write-up puts it, "public benchmarks measure happy-path completion; production measures recovery rate per failure mode." WebArena's 14-point baseline and Operator's short product lifespan are two symptoms of the same difficulty.

The compounding-error math also explains a pattern that looks paradoxical at first: agents frequently look impressive in a single demonstrated run and unreliable in production. A demo is, by construction, one successful trajectory shown after the failed ones were discarded; a pass^1 number captures exactly that survivorship. Production deployment, by contrast, needs pass^k-level consistency across thousands of unseen inputs, which is a fundamentally harder bar than looking good once.

Every lab's internal roadmap now has to reckon with that gap between a compelling demo and a dependable system, and it is the reason "reliability" rather than "raw intelligence" has become the metric that determines whether an agent ships.

Where This Goes From Here

The trajectory implied by the last three years is not "agents get uniformly smarter until they can do everything." It is narrower and more useful than that: progress concentrates wherever a verification signal exists, and stalls wherever it doesn't. The years since have run that way: the mechanically gradable benchmark is nearly saturated, while the open-ended desktop one was replaced with a harder version rather than beaten. Expect the next phase of agent development to look less like scaling up autonomy and more like scaling up checking — sandboxed execution environments, explicit approval gates on irreversible actions, graders that score an agent's output against a rubric before it's accepted, and retry loops that treat a single attempt as disposable rather than final.

None of that requires a smarter base model; it requires treating reliability as an engineering problem with its own primitives, the same way distributed systems treated network failure as a first-class concern rather than an embarrassment to be smoothed over in a demo.

Two further shifts follow from that reframing. First, single-agent architectures will keep giving ground to systems that split a long task across multiple narrower agents — one that plans, one that executes, one that checks the executor's work against the plan — because a coordinator that only has to verify a subordinate's output is solving a smaller, more checkable problem than one agent trying to both act and self-audit across twenty steps.

Second, the unit of progress will shift from "can the model answer this" to "how much budget — in tool calls, tokens, or wall-clock time — does it take to reach a verified answer," because when retries are cheap and verification is automatic, the economically relevant question is efficiency, not whether success is possible at all.

The honest forecast is that fully autonomous, general-purpose agents — the AutoGPT promise of 2023 — are further away than the marketing suggests, while narrow, verifiable, high-step-count-but-checkable agents — coding assistants, structured data pipelines, workflows with a clear pass/fail signal — are closer than skeptics assume and already shipping. The protocol layer that MCP opened will keep consolidating so that connecting a new tool stops being custom engineering.

The benchmark numbers will keep climbing on tasks with clean grading and keep lagging on tasks that require judgment under ambiguity. And the systems that win will not be the ones that first claim full autonomy — they will be the ones that make an agent's individual failures cheap, visible, and recoverable, because at realistic per-step error rates, that is the only way a twenty-step plan ever finishes the same way twice.