MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads

AI summary

Overview

Research area: LLM systems and hardware architecture — specifically, benchmarking and workload characterization for multi-agent LLM serving (MLSys, GPU/NPU architecture evaluation).

Technical level: Advanced. The paper assumes familiarity with LLM serving internals (continuous batching, KV cache, prefill/decode), accelerator architecture (Tensor Cores, L2 cache, HBM, SIMD/SIMT), and multi-agent frameworks.

Scope (1 sentence): MAServeBench is a benchmark that makes multi-agent collaboration a configurable workload-generation dimension, so researchers can produce native multi-agent serving workloads, replay them at fixed work under legal release changes, and trace how collaboration-induced execution shapes translate into GPU/NPU costs and task-level SLO attainment.

What This Paper Is About

Multi-agent systems (MAS) such as AutoGen, MetaGPT, ChatDev, and Magentic-One turn a single user request into a web of interdependent model calls, tool calls, and context exchanges. Conventional serving benchmarks model requests as independent arrivals, so they cannot capture which calls exist because of collaboration rules, when those calls become ready, or how much context and KV state they hold. The paper builds a benchmark that separates the logical collaboration rules from the realized operations, so that both the work generated by collaboration and the legal timing at which that work is exposed to a serving backend can be varied and measured against hardware cost and task deadlines.

Key Contributions

1. Composable MAS workload construction. Reusable collaboration stages (Spawn, Fork–Join, Refinement Loop, Debate) capture who participates, how agents coordinate, and what context they receive, and compose into workflows whose collaboration rules vary independently of fixed task inputs, roles, and tools.

2. Controlled work generation and exposure. Native realization produces collaboration-dependent operations, while hierarchical Semantic Traces record realized operations and typed control/data/state dependencies. This enables fixed-work replay that changes only legal release constraints (e.g., stage-annotated barriers) or only system configuration (e.g., KV budget), plus workloads for architecture simulators.

3. Task-linked hardware evaluation. Execution keys or aligned timestamps trace task, stage, and operation IDs through serving requests to scheduler steps, shared batches, and capture windows, connecting operator and memory observations back to the tasks and stages that exposed them.

4. Validation across structural, functional, and replay levels. Twelve documented workflows, seven paired native comparisons, and native-to-replay fidelity checks are reported; all 204 paired replay roots preserve resolved payloads, token counts, operations, and dependencies exactly, with root-latency differences between −1.85% and +2.88%.

Main Findings

• Workload abstraction and coverage: Twelve documented workflows cover all four local templates; five compose multiple templates, and MetaGPT uses sequential handoff through the workflow-level composition operator. The executable suite exercises seven structures across 30 task bindings, yielding 210 native realizations, plus a separate 21-instance fixed bank and an eight-binding GPT Researcher comparison.

• Functional comparability on real tasks: Seven paired native comparisons span six frameworks and four task sets (SWE-bench Verified, GAIA, WebArena, DeepResearch Bench) with matched instances and Qwen3.5-9B settings, showing quality differences of 1.2–3.8 percentage points on success/accuracy and at most 0.019 on RACE. Across three paired evaluations, request-input medians differ by at most 2.3%, with the largest input P90 shift a 7.1% reduction for Magentic-One/WebArena.

• Collaboration changes both the amount and the timing of work: At 64 users, Spawn shows 1.70× ordinary serving's aggregate modeled arithmetic intensity, and its longer execution (580 versus 109 seconds) contributes to a 13.85× KV memory–time integral; Loop has comparable intensity but longer state residency.

• Local savings can hide larger total work: In a four-call control, delivering selected answer fields instead of full JSON artifacts reduces input work by 29% and KV memory–time by 25%, but root latency by only 5%. In a GPT Researcher HTTP binding, the same selection reduces first-writer input by 7.5% yet increases total input by 28.4%, because an extra section revision and writing/checking pass expand LLM calls from 25 to 30 and graph depth from 15 to 20.

• Release ordering alone costs latency at fixed work: Adding a barrier to the four-call workflow reduces scheduled width from 24 to 12 and raises mean root latency by 26% (28% in selected replays). For three frozen GPT Researcher Full traces (83 calls), ordering research branches preserves paired arrivals, reduces scheduled width from 9 to 3, and increases root latency by 40–54%.

• Execution shapes shift during a MAS run: At 64 root users, parallel workflows decode mostly above 64 requests while ordinary serving and Loop stay within 64; Spawn's modal matrix row count shifts from 64 early to 128 late (versus 64 for Loop), mixed-cohort attention's kernel-time share rises from 28.3% to 44.6%, and Debate accounts for 44.8% of one capture's matrix kernel time through late calls with more than 128 rows.

• GPU ranking reverses with weight locality: Across 28 probe cases, for a one-row 4096×4096 output projection PRO 5000 is 1.78× faster with weight reuse, whereas A800 is 1.50× faster under eight-weight rotation. Independent counters record no external reads on PRO 5000 versus 25.6 MiB on A800 for one reused 32 MiB weight; rotation expands the working set to 256 MiB, exceeding both caches, with roughly 32 MiB read per call.

• NPU results depend on execution path, not just shape: Three matrix reference families cover 54.9% of sampled 910C hybrid-model matrix time. At 32 rows and 4096×4096, 950DT's native Memory pass is 1.49× faster with one reused weight and 2.32× faster with eight rotating weights. Across 12 Gather implementations on 950DT, binary-verified SIMT outperforms SIMD by 6.47–6.90×, while the Hybrid variant emits the same SIMD binary.

• Attention cost tracks delivered context: In native Qwen3-8B/910C captures, median attention duration rises from 296 to 769 μs between early and late execution; halving context vectors at fixed memory backing yields full/half kernel-time ratios of 1.91–2.01×, and 950DT is 3.08–3.17× faster than 910C at matched batch-32 geometry. The 1k–4k context range covers 95.8–98.7% of time-weighted pure-decode context observations in the application controls.

• Preemption turns memory pressure into repeated prefill: Across three native Qwen3-8B/910C mixed-cohort blocks, 1,372 preemptions amplify processed prefill to 1.76–1.84× source input, with all preemptions falling in time bins at 95% or more mean KV allocation; Loop-only blocks show neither effect. At fixed work, expanding KV from 16 to 40 GiB reduces the processed/source prefill factor by 52% and root P95 by 17.5%.

• Co-batching reshapes per-call latency: Reassigning the same 55 background traces to interleaved versus clustered arrivals raises target co-batching with all three background classes from 31.5% to 96.3% over 2–110 seconds. All 27 targets have shorter first-call but longer second-call scheduled residence, producing +2.17% (GSM8K) and +2.14% (PubMedQA) but −3.75% (Science) second-call-dominated root-latency changes, for a +0.743% overall median ratio.

• Hardware efficiency does not imply task SLOs: In a 2×2 DAG/Barrier × 16/40 GiB study against a shared 361.7-second deadline, the first three arrival rates incur no KV preemption, yet at the second rate DAG reaches 100% median deadline attainment versus Barrier's 4% under either KV budget. At the highest rate, 40 GiB removes preemption seen at 16 GiB in two blocks per release rule, cutting DAG root P95 by 15.5% and raising output TPS by 7.7%, but deadline goodput stays near zero. Raising the ordinary-serving request limit from 64 to 128 raises output TPS by 14.6% while lowering the timely-task rate by 22.5%.

Methodology in Plain English

The authors start from the observation that multi-agent systems are defined by rules (who talks to whom, in what order, and what history each agent sees), while serving benchmarks only see requests. They build a three-layer abstraction. First, a logical workload describes collaboration-stage slots, required artifacts and dependencies, workflow-level composition (sequence, choice, parallel composition and join, bounded repetition, handoff), and logical memory scopes. Second, each stage specifies participants, execution control (spawn, fork, join, repeat, finish), and context construction (inherited versus fresh history, which artifacts are delivered, in what order). Third, a native run executes those rules with a real model and real tools to produce a concrete operation graph whose nodes keep their task, stage-instance, and session identity.

Each run is recorded as a versioned Semantic Trace containing resolved inputs, output contracts, generation settings, optional native token IDs and output-length targets, tool results with measured elapsed times, and controlled external delays, with payloads in a content-addressed store. Replay can then hold this work fixed: tool calls return recorded results after their recorded durations, external services are not invoked, and dependencies are preserved, while the target backend is free to schedule, batch, cache, and place however it likes.

This trace/scenario/system-configuration split is what enables controlled comparison. A trace fixes work; a scenario binds it to arrivals, release rules, initial cache state, and task SLOs; a system configuration selects backend, hardware, and policies. A legal, stage-annotated barrier can serialize independent operations as long as it preserves required dependencies and creates no cycle, isolating the effect of exposure. Changing which artifact field is delivered instead changes resolved inputs, so it requires a new native realization. Measurements on the target side map task, stage, and operation IDs onto scheduler steps, shared batches, and kernel captures, and separate pre-submission, backend waiting, and scheduled residence on the completion path.

The evaluation runs on A800, PRO 5000, Ascend 910C, and Ascend 950DT using vLLM and vLLM-Ascend in BF16 with tensor parallelism of one, mixing application-level native runs (32/64/128 users, three blocks) with fixed-work replays and matched operator probes derived from the shapes observed in native traces.

Why This Matters

Impact on research. The paper argues that request-level traces record realized work but not the collaboration rules that generate it, so two tasks with similar request counts and token lengths can differ in readiness, batching opportunity, and memory demand. By making collaboration a workload variable with replayable, dependency-preserving traces, it gives systems and architecture researchers a common object for comparing schedulers, KV policies, and hardware without confounding changes in generated work. It also supports architecture simulators directly.

Real-world applications:

• Serving-stack tuning for agent products: choosing batching, KV capacity, and scheduling policies when agents fork, join, debate, and revise rather than issuing independent queries.
• Accelerator procurement and co-design: using workload-derived shapes (batch geometry, weight locality, context length, gather patterns) to decide which GPU/NPU features actually matter for agentic traffic.
• SLO planning for agent workflows: setting deadlines and admission policies using completion-path accounting rather than steady-state throughput alone.
• Framework design: showing that context-delivery choices (full artifact versus selected fields) can reduce local input while increasing total realized work and call count.

Industry relevance. The results directly challenge the common practice of judging serving improvements by aggregate throughput or per-request efficiency: raising the request limit from 64 to 128 raised output TPS by 14.6% but lowered the timely-task rate by 22.5%, and eliminating preemption at high load improved P95 and TPS without moving deadline goodput off near zero. For operators running multi-agent products on shared clusters, that means configuration choices must be evaluated on task completion, not on serving metrics alone.

Future Directions

• Reuse-aware compute and tiered KV memory. The GPU locality reversal motivates SRAM/L2 retention for reusable weights and cache bypass with HBM streaming for rotating weights; programmable SIMT with access-aware path selection for gather-heavy activity; and a page-managed HBM pool that asynchronously extends KV to DRAM when migration and remote-access cost is below saved prefill. High-Bandwidth Flash for cold, read-mostly weights and 3D near-memory attention for long contexts are proposed as untested hypotheses.

• Multi-replica and disaggregated serving. Prefix affinity can save prefill but concentrate demand on a contended replica, so routing and load balancing interact with complete-task latency even at fixed dependencies. The authors propose an API-facing replay adapter that preserves recorded work and arrivals while exposing per-replica routing, queueing, and pre-submission delay.

• Phase-level specialization. Systems that separate prefill and decode must offset KV/state transfer costs and cross-pool queueing, especially for long contexts and hybrid-model recurrent state. Replay extensions that associate phase placement and state transfers with each operation would let these designs be judged by complete-task SLOs.

• Whether hardware gains reach deadlines. The evaluation shows execution savings that do not translate into timely completion; an open question is which hardware or policy changes actually move complete-task latency distributions, and how far the twelve-workflow, seven-structure, thirty-binding coverage generalizes to other frameworks, tasks, and deployment topologies.

Target Audience

Researchers and engineers working on LLM serving systems, MLSys benchmarking, and accelerator architecture; performance engineers tuning multi-agent deployments; framework developers deciding how agents share context and artifacts; and graduate students studying workload characterization, since the paper's trace/scenario/system-configuration separation is a reusable template for controlled experiments beyond multi-agent systems.

Authors’ abstract

Large language model (LLM)-based multi-agent systems (MAS) coordinate specialized agents to solve complex tasks, turning each user request into interdependent model and tool operations. Unlike independent requests, these operations become ready at different times and consume evolving context as agents coordinate and exchange results. This creates changing patterns of concurrency, computation, and memory demand that can expose different bottlenecks in LLM serving hardware. Yet benchmarking these effects requires capturing the collaboration rules that generate serving workloads, beyond the requests observed during execution. We present MAServeBench, a benchmark that treats multi-agent collaboration as a configurable dimension of serving workloads for LLM serving and hardware evaluation. Its composable stages capture how agents coordinate and exchange context in documented MAS workflows, enabling native executions to generate different workloads even when task inputs and tools remain unchanged. Hierarchical Semantic Traces preserve realized operations and dependencies for reproducible fixed-work replay and as inputs to architecture simulators. Task-linked measurements connect these workloads to serving execution and GPU/NPU hardware behavior, revealing why serving efficiency does not always translate into timely task completion. The benchmark code is available at https://github.com/ShengxuanQiu/MAServeBench.