Skip to content
AI.info

The Pulse

AMD reports token-cache reuse in coding-agent benchmark

AMD says its ATOM inference engine reused 94.1% of prompt tokens at 48 concurrent coding-agent sessions in one benchmark run. A separate 56-session configuration reached up to 17.5K total tokens per second per chip, including cached input.

AMD reports token-cache reuse in coding-agent benchmark

AMD's ATOM inference engine reused 94.1% of prompt tokens across GPU and CPU memory at 48 concurrent coding-agent sessions in a benchmark run, the company says. In a separate configuration with 56 sessions, AMD reported up to 17.5K total tokens per second per chip on eight Instinct MI355X GPUs.

Source rocm.blogs.amd.com

The throughput figure includes both input and output tokens, including cached input, rather than newly generated tokens alone. A separate single-session result recorded P90 interactivity of 152 output tokens per second per user. The Nov. 17 post describes results from an Oct. 15 run of SemiAnalysis' InferenceX AgentX benchmark, using the Kimi-K3 model.

Replaying coding sessions

AgentX replays a public corpus of Claude Code sessions containing 393 sessions and 98,827 model requests. The corpus has a median input of 142,016 tokens and a median output of 444 tokens, according to the post. Its token counts use the trace's 64-token blocks rather than the Kimi-K3 tokenizer.

The replay preserves request sizes, shared prefixes, dependencies and timing, but substitutes synthetic tokens for the original code and tool output. AgentX counts concurrent live session trees rather than individual requests, so a single session can have several requests in flight.

Coding agents repeatedly send earlier conversation history along with new input. In one example selected for the post, the 48th call sends 195.5K tokens, of which only 1.3K are new. The post says the session illustrates workload patterns rather than representing a corpus average.

Where cached history lives

ATOM stores reusable history in GPU high-bandwidth memory and can extend that cache into CPU DRAM using LMCache. Restoring history from CPU memory trades additional retention capacity for transfer costs; the configuration described does not use an SSD tier.

Kimi-K3 combines multi-head latent attention, or MLA, layers with Kimi Delta Attention, or KDA, layers. A request can resume at a cached boundary only when both the MLA key-value cache and a KDA checkpoint are available there. Otherwise, ATOM falls back to an earlier usable boundary or processes the input from the start.

At 48 concurrent sessions, the configuration adds 128 GiB of CPU DRAM per GPU rank through LMCache. Each MI355X has 288 GB of HBM3E; AMD estimates that model weights take about 195 GB per GPU, leaving roughly 64 GB for runtime buffers and cached state after a 10% reserve.

For the same 48-session run, AMD says 77.3% of prompt tokens were reused from GPU memory and 16.8% were restored from CPU DRAM. The combined 94.1% is a reuse figure for that run, not a direct measurement of a throughput improvement over a configuration without caching.

Scheduling and decoding

New requests also need input processing, known as prefill, while existing requests are generating output. The post describes PrefillDelayer, which holds and groups short prefills so active agents can keep generating, as well as ReplaySSM for compacting speculative state and decode context parallelism for distributing MLA key-value cache across GPUs.

ATOM also uses persistent Mono decode kernels written in FlyDSL to combine work within a decode step. Cached history avoids repeated prefill, but generated tokens still require attention over that history and other per-step computation.

Source

  1. AMD ROCm Blogs

Companies in this story

Related stories

Explore

More articles