ProtocolBench: Which LLM MultiAgent Protocol to Choose?

AI summary

Overview

Research area: LLM-based multi-agent systems, specifically the communication protocol layer that connects agents (A2A, ACP, ANP, Agora).

Technical level: Intermediate. Readers should have basic familiarity with LLM agents, client-server communication patterns, and metrics like latency and throughput. No deep protocol internals are required.

Scope: This paper introduces ProtocolBench, a controlled benchmark that compares four multi-agent communication protocols across four scenarios and four measurement axes, and ProtocolRouter, a constraint-aware system that selects a protocol per scenario or per module.

What This Paper Is About

Multi-agent LLM systems increasingly rely on communication protocols to coordinate work, but practitioners pick protocols largely by intuition because no standardized benchmark compares them fairly. This paper builds that benchmark, isolates the protocol layer from other factors like model choice and prompts, and shows that protocol selection meaningfully changes task quality, latency, failure recovery, and security coverage. It then proposes a lightweight router that picks protocols based on stated requirements and observed performance.

Key Contributions

1. ProtocolBench: A protocol-agnostic benchmark with four scenarios (GAIA Document QA, Safety Tech, Streaming Queue, Fail-Storm Recovery) that measures task success, end-to-end latency, message/byte overhead, and failure-time robustness using thin wrappers around each protocol's native SDK.
2. A controlled comparison of A2A, ACP, ANP, and Agora, showing that no single protocol dominates across all scenarios and that the "best" protocol is workload-dependent.
3. ProtocolRouter: A deterministic, constraint-aware router that selects one protocol per scenario or per module, respecting hard requirements (encryption, streaming, delivery semantics) before optimizing for performance.
4. ProtocolRouterBench: A 60-scenario, 180-module selection benchmark (difficulty levels L1–L5) that tests whether a router can pick the correct protocol for each module under explicit constraints, with both spec-only and spec+performance variants.

Main Findings

• GAIA favors A2A for task utility: A2A achieved the highest LLM-judge quality (2.51 vs. next-best 2.33, +7.7%) and success average (9.29 vs. 7.28, +27.6%). Against ACP specifically, A2A showed 10.57% higher quality and 76.95% higher success.
• Streaming Queue favors ACP for latency: ACP had the lowest mean end-to-end latency (9.663 s), smallest standard deviation (1.077 s), and most controlled maximum (14.235 s). Agora was slowest at 13.135 s, a gap of roughly 3.48 s. Total completion time varied up to 36.5% across protocols (40.28 vs. 54.97 minutes).
• Fail-Storm Recovery favors A2A for continuity: A2A preserved 98.85% of its pre-fault answer-discovery signal (14.57 vs. 14.74), compared to ACP (92.41%), ANP (86.96%), and Agora (81.29%). Recovery times clustered around 8.0 s for all protocols.
• Safety Tech favors ANP and Agora for security: Both covered all five evaluated dimensions (TLS transport, session hijack protection, E2E encryption, tunnel sniffing resistance, metadata leakage prevention). A2A and ACP lacked native TLS-misconfiguration and tunnel-sniffing protections.
• Adapter overhead is not the bottleneck: Per-message adapter latency ranged from 0.13 ms (ACP, 4 agents) to 33.60 ms (Agora, 32 agents). ACP scaled sublinearly with worker count (exponent ~0.160), while A2A, ANP, and Agora grew approximately linearly.
• Cross-protocol bridges are cheap: Stateless routing and bridging added a mean of 0.80 ms per event (P95 = 0.94 ms), becoming noticeable only around 10³ routed events.
• ProtocolRouter improves targeted metrics: It cut Fail-Storm recovery time by 18.1% (8.00 s to 6.55 s) and raised GAIA success from 9.29 to 9.90, while leaving some secondary metrics flat or slightly worse.
• Selection accuracy improves with performance priors: Spec-only routing reached 53.5% scenario and 71.2% module accuracy; adding scenario-agnostic performance priors raised these to 63.3% and 81.7%, with macro-F1 improving from 0.721 to 0.824. Gains concentrated at harder levels (L4–L5), mostly by resolving A2A ↔ ACP confusions.

Methodology in Plain English

The researchers built a common harness that runs the same agent workflow, prompts, model, and hardware across all four protocols, swapping only the communication layer at the boundary. This prevents non-protocol factors from contaminating results and preserves each protocol's native retry, streaming, and reconnection behavior rather than hiding them behind a shared abstraction.

Each of the four scenarios stresses a different property: GAIA tests hierarchical multi-hop document reasoning; Streaming Queue tests high-throughput serving with 1,000 MS MARCO entries across a coordinator and four workers; Fail-Storm Recovery repeatedly kills 3 of 8 agents in a Shard-QA ring every 120 s and measures how well the system recovers; Safety Tech injects concrete attacks (TLS downgrade, replay, tunnel sniffing, session hijacking) into a medical Q&A pipeline.

Latency, success, quality, and byte counts are logged with shared trace and session IDs. For the router, the authors first filter out protocols that violate hard constraints, then break ties by interaction preference and optional performance priors. Ground-truth protocol labels for the selection benchmark were assigned by human experts.

Why This Matters

Impact on research: This is the first systematic protocol-level benchmark for LLM multi-agent systems, moving protocol choice from folklore to measurable engineering. It gives the field a shared evaluation surface and a reproducible baseline for comparing future protocols.

Real-world applications:

• Enterprise agent platforms choosing between A2A and ACP for cross-team workflow orchestration.
• Healthcare or financial systems where ANP-style identity-first, end-to-end encrypted communication is mandatory.
• High-throughput API services (search, retrieval, batch inference) where ACP's latency stability matters more than raw feature richness.
• Failure-prone or edge deployments where stateless, idempotent protocols like A2A keep systems running through node churn.

Industry relevance: The protocols evaluated (A2A, ACP, ANP, Agora) are backed by major vendors and consortia. The paper's finding that no protocol wins universally, and that per-module routing can beat the best single protocol on targeted metrics, is directly actionable for teams building production multi-agent infrastructure.

Future Directions

1. Online and cold-start routing: The current router is an offline planner that relies on measured priors. Adapting to completely new protocols with unknown performance remains an open problem.
2. Adversarial and highly dynamic workloads: The authors explicitly scope out adversarial conditions; extending ProtocolBench to cover them would test robustness more aggressively.
3. Scaling beyond moderate agent counts: Bridge and adapter costs were measured at 4–32 workers; behavior at hundreds or thousands of agents, and under dense communication, is untested.
4. Richer composition semantics: Cross-protocol bridges currently preserve application content but cannot transfer security guarantees across domains, an important limitation for heterogeneous deployments spanning trust boundaries.

Target Audience

Systems engineers and researchers building or deploying LLM multi-agent infrastructure, benchmark designers interested in controlled evaluation methodology, and technical decision-makers who need evidence for protocol selection rather than vendor-driven intuition.

Authors’ abstract

As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability. Despite the existence of diverse protocols (A2A, ACP, ANP, Agora, etc.), selection is often intuition-driven and lacks standardized guidance. We introduce ProtocolBench, a benchmark that systematically compares agent protocols along four measurable axes: task success, end-to-end latency, message or byte overhead, and robustness under failures. On ProtocolBench, protocol choice significantly influences system behavior. In the Streaming Queue scenario, overall completion time varies by up to 36.5% across protocols, and mean end-to-end latency differs by 3.48 s. Under Fail-Storm Recovery, resilience also differs consistently across protocols. Beyond evaluation, we present ProtocolRouter, a learnable protocol router that selects per-scenario (or per-module) protocols from requirement and runtime signals. ProtocolRouter reduces Fail-Storm recovery time by up to 18.1% versus the best single-protocol baseline, and achieves scenario-specific gains such as higher success in GAIA. We also release ProtocolRouterBench to standardize protocol evaluation and improve reliability at scale.