A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software Evolution

AI summary

Overview

Research area: Software engineering and human–agent interaction — specifically, how the design of desktop-application interfaces (GUIs, SDKs, CLIs) affects the performance of large language model (LLM) agents, and how to automatically synthesize better interfaces for those agents.

Technical level: Advanced. The paper assumes familiarity with LLM agent scaffolds, software SDKs, benchmark construction with artifact-based verifiers, and programmatic automation of desktop applications such as QGIS, Blender and FreeCAD.

Scope: The paper builds a 255-task cross-version benchmark (expanded to 368 tasks) to isolate interface design as a variable in agent performance, derives three interface properties from a decade of human-driven software evolution, and presents ANIS, a system that synthesizes an agent-native interface from an installed application's runtime capabilities.

What This Paper Is About

Desktop applications expose their functionality through interfaces built for people (visual GUIs) or for developers (low-level SDKs), so LLM agents must either interpret visual controls or discover and stitch together low-level operations. The paper asks what interface properties actually make desktop software easier for agents, using real historical software releases as a controlled source of interface variation rather than hand-crafted variants that would only encode the authors' own assumptions. It then builds a system, ANIS, that synthesizes an agent-native interface delivering those properties for any installed application, without waiting for the application itself to evolve.

Key Contributions

1. DeskIF-Bench, a benchmark of 255 artifact-verified tasks across six applications in five domains (office productivity, 2D graphics editing, 3D content creation, geospatial analysis, and computer-aided design), expanded to 368 tasks across eight applications to test generalization. 102 tasks are retained from OSWorld and OpenComputer after filtering; 153 are newly synthesized.
2. A controlled cross-version methodology that holds the task, agent scaffold, backbone model, seed files, tool permissions, time budget and verifier fixed while varying only the installed software release, isolating interface design as the experimental variable across four releases of six applications.
3. Three interface properties derived from trajectory analysis — discoverability, granularity and observability — identified by tracing how documented interface changes across historical releases affect agent task completion.
4. ANIS (Agent-Native Interface Synthesizer), a two-phase framework that builds validated atomic commands at build time and mines verified goal-level commands from actual agent usage at run time, raising success on the expanded benchmark from 68.5% with the native SDK to 74.5% while cutting tokens per task from 84.4K to 21.2K.

Main Findings

• Newer releases are consistently better for agents: the newest release outperforms the oldest in all 12 model–application series, across Claude Haiku 4.5 and DeepSeek V4 Flash, even though none of these releases were designed for LLM agents.
• The QGIS illustration: on the same task (a graduated GDP map with fixed-width intervals), QGIS 3.10 required the agent to derive eleven classes through a long chain of poorly documented low-level calls and failed after 33 turns and 662k input tokens; QGIS 3.44 offers a task-level QgsClassificationFixedInterval operation and the same agent succeeded in 11 turns with 36k input tokens.
• Magnitude of version gains: QGIS rises from 33% to 65% success (Haiku) and 41% to 69% (DeepSeek), close to 30 points for both; Impress rises from 33% to 52% (Haiku) and 40% to 58% (DeepSeek). Mean SDK operations per task on QGIS fall from 30.4 to 21.2 for Haiku.
• Discoverability improved through cohesion and descriptions: FreeCAD 1.0 introduces App::VarSet, making model parameters explicit properties rather than aliased spreadsheet cells; QGIS runtime function descriptions grow by 20% per function from 3.10 to 3.44.
• Granularity improved through higher-level operations: QGIS's Processing toolbox grows from 272 to 354, 373 and 440 operations across releases, directly supporting 22 of the 54 QGIS benchmark tasks; Inkscape's multi-page document object turns a multi-page export from 101 turns and 1.8M input tokens into 30 turns and 119k.
• Observability never closed the gap: an audit of the six study applications finds 581 trajectories where the agent reported completion but the saved artifact failed verification; of the 569 containing a detected check, 153 (26.9%) lack independent artifact readback and 416 (73.1%) have it. QGIS examples include QgsFillSymbol.setAngle with no matching angle() reader, and QgsFeatureRenderer.setVertexMarkerAppearance with no corresponding reader in any of the four evaluated releases.
• Discovery consumes a large share of the budget: agents spend 24.1–36.2% of input tokens on discovery, and shipping official documentation does not help — Kimi K2.5 gains 1.4 points, while Haiku, DeepSeek and Opus decline by 1.1, 1.7 and 1.9 points respectively.
• ANIS beats native SDK use: on the expanded 368-task benchmark with DeepSeek V4 Flash, atomic commands reach 72.3% versus 68.5% for the native SDK and 17.4% for CLI-Anything; goal-level commands raise this to 74.5%. With Claude Haiku 4.5 the abstract reports atomic interfaces improving over native SDK use by 3.5 points and goal-level commands reaching 67.4%.
• Token efficiency: tokens per task fall from 84.4K (native SDK) to 22.6K (ANIS without goal-level commands) and 21.2K (with goal-level commands), a 73.2% and 74.9% reduction respectively; CLI-Anything uses 67.0K, a 20.7% reduction.
• CLI-Anything's low score is a coverage problem, not a form-factor problem: 98.7% of its failed tasks are attributable to missing commands and only 1.3% to agent errors with the necessary commands available; ANIS exposes over 20× as many commands as CLI-Anything on shared applications.
• Two exceptions: ANIS remains below the native SDK on FreeCAD (77.4% vs 96.8%) and Blender (45.7% vs 54.3% without goal-level commands), because operations there depend on implicit preconditions such as linked sketches and modeling features or coordinated scene objects and editing modes; 28.6% of ANIS's FreeCAD failures lack a required command.
• Observation commands matter a lot: prohibiting observation commands by instruction, while leaving all editing commands available, drops overall success from 72.3% to 60.3%, a 12.0-point decline.
• Goal-level commands help monotonically: 10 goal-level commands were synthesized across the eight applications, and adding them never lowered success on any application, raising the overall rate from 72.3% to 74.5%. Example: five QGIS operations to create a print layout and add its map, legend, scale bar and title collapse into one parameterized call.

Methodology in Plain English

The authors' central trick is to treat software history as a natural experiment. Because an application's core functionality stays largely stable across releases while its interface is continually reshaped by developers responding to real usability pressure, they can hold everything else constant and change only the software version.

To do this they build a benchmark where every task describes an artifact-level outcome (a saved document, a rendered map, a 3D scene) rather than a specific sequence of clicks or calls, and every task ships with a deterministic script that inspects the saved artifact and returns pass or fail. Tasks come from two places: filtered tasks reused from prior datasets such as OSWorld and OpenComputer, and tasks synthesized by an "authoring agent" that has privileges the evaluated "solving agent" does not — application-specific skills and privileged libraries such as trimesh — so the benchmark is not capped at the authoring agent's own capability ceiling. Proposals are grounded in official tutorials and community discussions, must specify artifact outcomes, and are kept only if a reference artifact and working verifier can be built and the outcome is feasible in every evaluated release. Verifiers must accept the reference artifact and reject task-relevant perturbations.

For the cross-version study, the same 255 tasks are run against four releases of six applications, with the agent scaffold (OpenCode), seeds, permissions, time budget and verifier all pinned, using pass@3 across Claude Haiku 4.5 and DeepSeek V4 Flash. Trajectory audits of exploration turns and operations per task are restricted to model–task pairs that succeeded in all four releases, so that failed trajectories at arbitrary stages don't distort the comparison.

ANIS then turns the three identified properties into engineering requirements. At build time it discovers candidate capabilities from the running application (registered operations, object introspection, application menus), executes candidates to see whether they persistently modify the saved artifact, and releases only invocations the application has demonstrably accepted — packaging each as an atomic command with a uniform specification, a usage example, and, where a read accessor exists, a paired read command. Commands are grouped by artifact object so that thousands of them remain discoverable. At run time, ANIS watches the atomic-command sequences agents actually execute, and when a subsequence recurs in at least three requests and realizes a shared artifact-level subgoal, it packages that sequence into a named goal-level command after verifying reproducibility on a clean copy of the originating artifact. Calc and GIMP were held out from every stage of method development and their interfaces synthesized once, after the pipeline was frozen.

Why This Matters

Impact on research: The paper inverts the usual benchmark setup. OSWorld, Windows Agent Arena, Agent S and related work hold the interface fixed and vary the agent; this work holds the agent fixed and varies the interface, which makes interface design a measurable experimental variable rather than an unexamined constant. It also provides a reusable benchmark and a concrete set of three design properties that other interface-synthesis efforts can be evaluated against, and it shows that official documentation is not a substitute for interface quality.

Real-world applications:

• Geospatial and scientific tooling: QGIS users could get a synthesized interface that turns multi-step renderer and classification work into single verified calls, cutting both failure rates and token cost on tasks like graduated maps and print layouts.
• Creative and 3D content pipelines: Blender and Inkscape workflows (scene assembly, multi-page export) could be exposed to agents at the level of recurring goals instead of coordinated object state, though the paper shows implicit dependencies in these applications remain the hardest case.
• Office productivity automation: Writer, Impress and Calc tasks could be driven through discoverable, verifiable commands rather than GUI grounding or fragile macro scripts.
• CAD and engineering: FreeCAD's introduction of App::VarSet illustrates how making parameters explicit in the object model helps agents; ANIS could surface such properties uniformly across versions, including older ones.

Industry relevance: Software vendors now face a genuine question of whether to ship an agent-facing interface alongside their GUI and SDK. The paper argues they need not wait: ANIS generates an interface in 17.1–36.4 hours of wall-clock time per application, without modifying the application, and the result is transport-agnostic and could be exposed via CLI or the Model Context Protocol. The 73% token reduction translates directly into cost, and the finding that 98.7% of CLI-Anything failures stem from missing commands rather than the command-line form itself is a concrete lesson for anyone building agent tooling.

Future Directions

• Handling implicit preconditions: ANIS's validation rejects operations whose preconditions are implicit, which is exactly why FreeCAD and Blender remain below native SDK performance. Extending discovery and synthesis to reproduce linked-sketch, selection and editing-mode dependencies is the clearest open problem.
• Publishing and sharing goal-level commands: the paper explicitly leaves publication of the mined goal-level command sequences as future work, which raises questions about how such commands should be versioned, distributed and validated across users and artifacts.
• Scaling the run-time stream: goal-level commands were mined from simulated trajectory streams and a disjoint held-out split, not from live long-term deployment. Whether recurrence thresholds (three requests) and goal detection hold up under real, heterogeneous request traffic is untested here.
• Beyond the evaluated applications: the benchmark covers eight desktop applications; whether the three properties and the synthesis pipeline transfer to web applications, mobile software or applications with no usable programmatic surface is not reported.

Target Audience

Researchers and engineers working on LLM agents, computer-use agents and agent tooling; benchmark designers interested in controlled evaluation methodology; and developers at software vendors or platform teams who are deciding how to expose their applications to agents. It is also relevant to HCI and software engineering researchers studying how interface design shapes machine as well as human users. Readers without background in agent scaffolds, SDK introspection or artifact-based verification will find the paper dense, though the motivating QGIS example is accessible to a general technical audience.

Authors’ abstract

Large language model agents are emerging as an important new class of users of desktop applications. However, existing desktop interfaces were designed for human end users or application developers, leaving agents to interpret visual controls or discover and compose low-level software operations. This mismatch can hinder even capable agents, yet it remains unclear what interface properties make desktop software easier for agents to use. To study this question, we treat software evolution as a natural source of interface variation and introduce DeskIF-Bench, a benchmark of 255 artifact-verified tasks across six desktop applications. By holding the task, agent, and verifier fixed while varying the software release, we measure how interface changes affect agent performance. Across two models and six applications, the newest release consistently outperforms the oldest. Trajectory analysis identifies three relevant interface properties: discoverability, granularity, and observability. Guided by these properties, we develop ANIS, a framework that synthesizes agent-native interfaces from the capabilities of installed applications. On an expanded evaluation of 368 tasks across eight applications, atomic interfaces improve success over native SDK use by 3.8 and 3.5 percentage points with DeepSeek V4 Flash and Claude Haiku 4.5, respectively, while reducing online token usage by approximately 73%. Goal-level commands further raise success to 74.5% and 67.4%, respectively.