Matched Capabilities, Different Behaviors: A Controlled Study of MCP and CLI Interfaces for Language Agents

AI summary

Overview

Research area: Language agent systems — specifically the interface layer between a language model and the executable tools it uses (Model Context Protocol vs. command-line interfaces).

Technical level: Advanced. The paper assumes familiarity with agent architectures, tool-calling protocols, and benchmark evaluation, though the core question is stated plainly.

Scope: A controlled study that holds a model's executable capabilities fixed while varying only how those capabilities are presented, plus a repair method built on the findings.

What This Paper Is About

Language agents increasingly do their work by calling external tools, so the way a capability is presented to the model becomes part of the decision problem. MCP and CLI can expose exactly the same underlying operations through very different model-facing representations. This paper asks a narrow, deliberately controlled question: if the available actions are truly matched, do agents still behave differently — and if so, where does that difference come from and can it be fixed?

Key Contributions

1. A controlled comparison framework (MICA). Matched Interface Comparison and Analysis renders both MCP and CLI from a single shared semantic action contract and routes decoded actions through the same execution runtime, so capability sets are genuinely identical and only the representation differs.

2. Evidence that matched capabilities still produce divergent behavior. Across six models and three interactive environments, the matched interfaces yield substantial differences in task performance and interaction cost, and the authors argue these go beyond surface validity into semantic action selection, failure pathways, and recovery.

3. Localization and trajectory analysis. Matched-state probes isolate where agents are sensitive — capability presentation, realization, and feedback — and paired natural trajectories show structured divergence in 94.3% of 2,004 episodes, with early differences propagating through later interaction.

4. A repair method (DART). Decoupled Action Realization and Targeted Repair preserves validated semantic commitments while repairing interface-specific realization. The abstract reports benchmark score improvements in all six evaluated settings by up to 9.0 points, and MCP token cost reductions in all three environments by up to 27.1k tokens per task.

Main Findings

• Same actions, different agents. When MCP and CLI expose equivalent capabilities, agents still differ substantially in both how well they complete tasks and what the interaction costs.

• The divergence is structured, not noise. Effects appear in which semantic actions get selected, the paths by which agents fail, and their ability to recover — not merely in superficial formatting or parseability.

• Sensitivity is localizable. Matched-state probes attribute the differences to three distinct sources: how the capability is presented, how it is realized, and the feedback the agent receives.

• Differences are pervasive and sticky. 94.3% of 2,004 paired episodes show structured divergence, and early divergences propagate through subsequent interaction rather than washing out.

• Interface realization is an intervention point. Because the divergence traces to realization rather than to capability, it can be repaired: DART improves benchmark score in all six evaluated settings by up to 9.0 points.

• Repair also cuts cost. DART reduces MCP token cost in all three environments by up to 27.1k tokens per task.

• Framing result. The authors conclude that interface representation should be treated as part of the closed-loop decision environment that shapes agent behavior, not as a neutral wrapper.

Methodology in Plain English

The design rests on a single idea: make the two interfaces as comparable as possible so that any remaining behavioral difference can be attributed to representation.

1. Define actions once. The authors write a shared semantic action contract describing what operations exist, independent of any interface.
2. Render it two ways. That contract is automatically expressed as an MCP interface and as a CLI, giving two model-facing views of the identical action set.
3. Execute through one runtime. Whatever the model decodes as an action is routed through the same execution layer, so success and failure are not artifacts of different backends.
4. Run agents in matched conditions. Six models are run in three interactive environments, and the paired runs are compared on task performance and interaction cost.
5. Probe the mechanism. Matched-state probes hold the agent's situation constant while varying presentation, realization, or feedback, to see which factor flips behavior.
6. Compare trajectories side by side. Paired natural trajectories are analyzed for where runs diverge and whether early divergence persists.
7. Build and test a repair. DART then keeps the semantic commitments that were validated while fixing the interface-specific realization, and is evaluated on score and token cost.

The abstract does not name the six models, the three environments, or the specific benchmarks, so those details are not available here.

Why This Matters

Impact on research. Agent benchmarks typically compare models across a fixed tool setup, treating the interface as part of the environment rather than a variable. This work shows that interface realization can shift measured performance and cost on its own, which means prior cross-system comparisons that differ in tooling style may be confounded. It also reframes interface design as an experimental knob and a legitimate object of study, rather than plumbing.

Real-world applications.

• MCP server and tool developers get evidence that wrapping the same API as an MCP tool versus a CLI tool is not a stylistic choice — it can change agent success rates and token spend.
• Agent platform and orchestration teams can use the decoupling idea in DART to keep a stable semantic action layer while iterating on how tools are exposed.
• Benchmark and evaluation designers can adopt matched-interface comparison to separate model capability from interface artifacts.
• Agent debugging and reliability engineering benefits from the probe approach, which points to presentation, realization, or feedback as the source of a failure instead of treating a bad run as opaque.

Industry relevance. MCP has become a widely used way to connect models to tools, and CLI remains the default for many developer workflows. A result showing double-digit-point score swings and large token-cost differences from representation alone speaks directly to cost and reliability decisions in deployed agent products.

Future Directions

• Which specific representational properties drive the effect? The abstract identifies presentation, realization, and feedback as loci of sensitivity, but not which concrete design choices within each matter most — the natural next step is design guidance for interface authors.
• How far does DART generalize? Its reported gains cover the six evaluated settings; whether the repair approach transfers to other models, other environments, and other interface styles is left open.
• Does early divergence have a fix? Since early differences propagate through an episode, targeted intervention in the first few turns may be a cheaper remedy than repairing the interface wholesale.
• Should matched-interface comparison become standard practice? The work raises the question of whether agent evaluations ought to report interface-matched baselines, and how that would change published comparisons.

Target Audience

Researchers working on language agents, tool use, and agent evaluation; engineers who build MCP servers or CLI tooling for agents and want to know whether the choice matters; benchmark designers concerned with isolating model capability from harness effects; and advanced practitioners debugging why an agent succeeds with one tool interface and fails with another.

Authors’ abstract

Language agents increasingly act through external tools and interactive environments, making the interface between the model and its executable capabilities part of the decision process. Model Context Protocol (MCP) and command-line interfaces (CLI) can expose the same semantic operations through markedly different model-facing representations. We study whether these representational differences change agent behavior when executable capabilities are matched. We introduce Matched Interface Comparison and Analysis (MICA), which renders MCP and CLI from a shared semantic action contract and routes decoded actions through the same runtime. Across six models and three interactive environments, matched interfaces produce substantial differences in task performance and interaction cost. These effects extend beyond surface validity to semantic action selection, failure pathways, and recovery. Matched-state probes localize sensitivity to capability presentation, realization, and feedback. Paired natural trajectories reveal structured divergence in 94.3% of 2,004 episodes and show that early differences propagate through subsequent interaction. This analysis identifies interface realization as a direct intervention boundary. We therefore introduce Decoupled Action Realization and Targeted Repair (DART), which preserves validated semantic commitments while repairing interface-specific realization. DART improves benchmark score in all six evaluated settings by up to 9.0 points and reduces MCP token cost in all three environments by up to 27.1k tokens per task. Together, these results establish interface representation as part of the closed-loop decision environment shaping agent behavior.