Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads

AI summary

Overview

Research area: LLM inference serving systems — specifically, capacity provisioning for Attention–FFN disaggregated (AFD) decoding architectures, analyzed with probability theory (renewal-reward processes, central limit theorems, Gaussian order statistics).

Technical level: Advanced. The paper's core is a closed-form provisioning derivation requiring comfort with stochastic processes and order-statistic asymptotics, though the system motivation is accessible.

Scope: The paper derives and simulates an analytical rule for choosing the Attention-to-FFN instance ratio in a disaggregated LLM decoding cluster under stochastic request workloads.

What This Paper Is About

In AFD serving, stateful Attention layers (dominated by KV-cache reads) and stateless, compute-intensive FFN layers run on separate hardware and communicate every decode step. Because these resources scale independently, an operator must choose how many Attention instances feed each FFN instance — a ratio the paper calls r — and a wrong choice causes step-level blocking and device idle time in either direction. The paper's goal is to replace empirical search or naive deterministic approximations with a closed-form, workload-aware rule for the optimal ratio, accounting for the fact that Attention load per slot drifts as KV caches grow and completed requests are refilled with random prompt and decode lengths.

Key Contributions

1. Probabilistic workload model. A discrete-time renewal-reward analysis of a single decode slot under continuous batching, identifying a single workload statistic θ that governs provisioning under arbitrary prompt-length and decode-lifetime distributions and admits a nonparametric estimator from request traces.

2. Provisioning rule with load-balancing correction. A closed-form mean-field rule for the A/F ratio that decomposes into three operating regimes (Attention-bottleneck, communication-bottleneck, FFN-bottleneck), each with an interpretable balance condition, plus a Gaussian order-statistic correction for cross-worker synchronization stragglers.

3. Validation across workloads and hardware regimes. A trace-calibrated AFD simulator used to validate the framework across diverse workload and hardware configurations, with the closed-form rule matching the simulator-optimal ratio within 10%.

4. Simulator release. Source code available at https://anonymous.4open.science/r/AF-release-1C11.

Main Findings

• Age-adjusted load, not arrival-average load, drives provisioning. The stationary per-slot token load has mean θ = E[D·P + D(D−1)/2] / E[D]. When P and D are independent this reduces to θ = μ_P + (μ_D − 1)/2 + σ_D²/(2μ_D); without independence an extra term Cov(P, D)/μ_D enters. The paper stresses that θ is not the natural first guess μ_P + μ_D, because length-biasing samples longer requests more often at a random decode step.

• The synchronization barrier scales as √(2 log r). With W_{B,r} the maximum load across the r Attention workers, (W_{B,r} − Bθ)/(√B·ν) converges to the maximum of r i.i.d. standard normals, and E[W_{B,r}] = Bθ + √B·ν·κ_r + o(√B), where κ_r ~ √(2 log r). The relative synchronization overhead is (ν/θ)(κ_r/√B), growing sublogarithmically in r and decaying as B^(−1/2).

• Monte Carlo validation of the Gaussian correction. Under geometric decode lifetimes with B = 256, μ_P = 100, and μ_D = 500 (μ_out = 499), the CLT prediction matches Monte Carlo estimates (50,000 trials per r) within 0.5% for all r ∈ {2, …, 24}. The overhead reaches ~11% at r = 24, accounting for the majority of the ~15% throughput gap observed at large r. After incorporating the correction, the simulation-optimal r* remains at 8.

• Closed-form candidate ratios. The mean-field optimum lies in a set of four candidates: min{(μ_A − β_C)/(α_C B), (μ_A − β_F)/(α_F B)}, √(β_C/(α_C B)), √(β_F/(α_F B)), and (β_C − β_F)/(B(α_F − α_C)).

• Geometric-decode specialization. If P and D are independent and D ~ Geom(p), with μ_out := (1 − p)/p, then θ = μ_P + μ_out and ν² = σ_P² + μ_out(μ_out + 1), so the mean-field rule reduces to the form derived in prior analyses of LLM decoding under geometric assumptions, while the barrier-aware refinement sharpens it.

• Simulation agreement under the calibrated setup. With latency coefficients calibrated for DeepSeek-V3 on Huawei Ascend 910C NPUs, B = 256, μ_D = 500 (σ_D² = 294500), and μ_P = 100 (σ_P² = 9900), the theoretical optimal ratio r*_mf ≈ 9.3 closely matches the simulation optimum, and the theoretical throughput curve tracks simulation across all tested configurations.

• Idle behavior. At r = 1, FFN idles more than 60% of the time waiting for Attention outputs (η_F > 60%); increasing r improves FFN utilization until saturation. The crossover where η_A ≈ η_F marks the balanced configuration.

• Ablations. For B ∈ {128, 256, 512}, the theoretical optimal ratios are r* = {7.08, 9.34, 10.31}: larger batches achieve higher peak throughput, and optimal r* increases moderately with B. Longer prefills (μ_P) and longer decode sequences (μ_D) both push optimal r* upward, because both increase KV-cache token load and thus Attention-side pressure.

• Stated limitation. Because AFD lacks mature open-source implementations, the paper validates only against its trace-calibrated simulator; real-system validation is left to future work.

Methodology in Plain English

The authors model one decode slot at a time. Each request that occupies a slot has a prompt length P and a decode lifetime D (how many decode steps it stays in the slot), drawn independently across requests. While a request is alive, it contributes P + a tokens of KV load at age a. Using the renewal-reward theorem — treating each request as one renewal cycle of length D — the authors compute the long-run average token load θ seen at a uniformly random decode step, along with its variance ν².

Next they handle the fact that all r Attention workers run in lockstep, so the step is gated by the slowest one. Each worker's total load is a sum of B independent slot loads, so a multivariate central limit theorem plus a uniform-integrability bound shows the maximum across workers behaves like the maximum of r standard normals. That produces the correction term √B·ν·κ_r, with κ_r computable by a one-dimensional integral and asymptotically ≈ √(2 log r).

The cycle time per step is then the maximum of three terms: Attention latency (linear in token load), communication latency (linear in rB), and FFN latency (linear in rB). Maximizing throughput per instance — rB divided by (r+1) times the expected cycle time — gives the four candidate ratios, of which the best-performing one is chosen. A barrier-aware refinement replaces the closed-form maximization with a one-dimensional analytic optimization plus a discrete search over candidate r values, evaluable in milliseconds.

Finally, the authors build a discrete-event, cycle-by-cycle simulator of the r A–1 F topology. Each batch moves through a six-state machine (Attention → A2F transfer → Waiting → FFN → F2A transfer → Waiting → repeat), with two batches in flight so that one batch's communication and FFN work overlaps the other batch's Attention. Latency coefficients are obtained by linear regression on real execution traces from DeepSeek-V3 running on Huawei Ascend 910C NPUs.

Why This Matters

Impact on research. AFD is an emerging architecture without mature open-source implementations. This paper supplies analytical foundations before large-scale system building begins, and its central insight — that stochastic workload structure, not just mean request size, determines the provisioning ratio — reframes the A/F ratio as a stochastic-systems problem rather than a tuning knob. It also connects serving-system design to classical renewal-reward and fork-join-style analysis, while showing why standard fork-join assumptions (stationary, identical service times) do not directly apply.

Real-world applications:

• Capacity planning for datacenter LLM inference clusters, where the number of Attention devices per FFN device must be decided before hardware is provisioned.
• Autoscaling and elastic reconfiguration of disaggregated serving deployments, since the rule can be recomputed from updated request traces in milliseconds.
• Cloud and API providers operating LLM inference as a service, where oversizing either resource pool directly raises cost per token.
• Hardware-aware deployment tuning across different accelerator platforms, since the latency coefficients can be recalibrated for other hardware per the framework in Appendix B.

Industry relevance. The calibration uses DeepSeek-V3 on Huawei Ascend 910C NPUs, connecting the analysis to concrete production-class models and hardware. The paper's framing — that mis-sizing causes step-level blocking and costly device idle time in both directions — speaks directly to the cost structure of serving fleets, and the reported ~10% agreement between predicted and simulation-optimal ratios is the kind of accuracy that supports using the rule as a first-pass provisioning recommendation.

Future Directions

1. Real-system validation. The paper explicitly leaves validation on deployed AFD systems to future work as AFD implementations mature; only the trace-calibrated simulator is used here.
2. Load-balancing routing. The analysis notes that cross-worker load-balancing policies can mitigate the straggler gap, though some irreducible variance from stochastic dynamics remains — the interaction between routing policy and the barrier correction is an open design question.
3. Non-geometric and unbounded distributions. Appendix A.7 is said to discuss length-biasing under unbounded distributions for completeness, and the geometric-decode specialization is described as "practically common" rather than universal, leaving room to test other lifetime distributions.
4. Topologies beyond r A–1 F. The general framework is stated for an x A–y F topology reduced to ratio r = x/y; extension to multi-FFN routing and to configurations where communication is not fully hidden by pipelining remains to be explored.

Target Audience

This paper is most useful to systems researchers and engineers working on LLM inference serving infrastructure — particularly those designing or provisioning disaggregated Attention/FFN deployments on accelerator clusters. It also suits applied probabilists and queueing theorists interested in stochastic provisioning problems, and hardware or capacity planners at organizations making device-count decisions for LLM serving fleets. Readers need familiarity with LLM decoding mechanics (KV cache, continuous batching, prefill vs. decode) and comfort with central limit theorem arguments and order statistics; the simulation setup itself is accessible to practitioners who want to reuse the released simulator.

Authors’ abstract

Attentio-FFN disaggregation (AFD) is an emerging architecture for LLM decoding that separates state-heavy, KV-cache-dominated Attention computation from stateless, compute-intensive FFN computation, connected by per-step communication. While AFD enables independent scaling of memory and compute resources, its performance is highly sensitive to the Attention/FFN provisioning ratio: mis-sizing induces step-level blocking and costly device idle time. We develop an analytical provisioning framework for AFD bundles in an $r$A--$1$F topology under stochastic workloads. Two sources of randomness shape the problem: per-slot Attention workload evolves as KV caches grow and completed requests are replenished with random prompt and decode lengths, and synchronized execution across Attention workers introduces a barrier governed by the slowest worker. We address both via a renewal-reward characterization of the per-slot stationary token load, identifying a single workload statistic $θ$ that governs provisioning under arbitrary prefill-decode distributions and admits a nonparametric estimator from request traces. The analysis yields a closed-form mean-field rule for the optimal A/F ratio decomposing into Attention-, communication-, and FFN-bottleneck regimes, together with a Gaussian barrier-aware refinement that quantifies cross-worker synchronization overhead. A trace-calibrated AFD simulator supports the framework across workloads: the predicted optimal ratio matches the simulation-optimal within 10%. Together, these results provide a compact, calibratable account of how stochastic workload structure determines provisioning in disaggregated LLM serving.