Skip to content
AI.info

The Pulse

Cerebras puts split AI inference into production with AMD GPUs

Cerebras is using AMD GPUs to process prompts and its CS-3 systems to generate responses in its inference cloud. The company says a production test delivered five times the throughput per megawatt of a CS-3-only setup without slowing token generation.

Cerebras puts split AI inference into production with AMD GPUs

Cerebras is using AMD GPUs to process AI prompts and its own CS-3 systems to generate responses in a production deployment on its inference cloud. The company says that in one test the split delivered five times the throughput per megawatt of a CS-3-only configuration without reducing token generation speed.

The comparison came from a November 14 test of a shared GPT-OSS-120B endpoint on Cerebras Inference Cloud, measured before and after the split was enabled. Cerebras published the results on November 17 alongside details of how infrastructure partners can adopt the approach.

Dividing the work

An inference request has two main stages. Prompt processing, or prefill, handles the input and produces intermediate model state; response generation, or decode, uses that state to produce output tokens.

Decode generates tokens one at a time, using each generated token as input for the next. Each step requires another read of model data from memory, making memory bandwidth important to the process.

Cerebras says its wafer-scale chip has tens of gigabytes of SRAM and tens of petabytes per second of memory bandwidth: 350 times the SRAM capacity and 5,400 times the memory bandwidth of a typical GB300 GPU. The company says keeping data on the chip lets its systems accommodate more concurrent requests and model layers there.

In the cloud deployment, shifting prefill to AMD Instinct GPUs leaves Cerebras systems to handle response generation. The company plans to deploy the split approach across hundreds of megawatts in its inference cloud and partner deployments by the end of 2027.

Options for partners

Infrastructure partners can use a Cerebras-provided reference architecture to deploy clusters that place their chosen prefill accelerators and Cerebras systems on a unified network. For software, they can use either a complete disaggregated stack or a decode-only software development kit.

The complete stack includes a framework that routes requests between prefill and decode pools and manages replica capacity and health. It also includes a prefill engine that can use open-source inference engines such as vLLM or SGLang, and a Cerebras decode engine for generating responses.

Its KV Portal software transfers intermediate model state, including key-value caches, between the prefill and decode engines over RDMA. With the decode-only SDK, partners can bring their own inference framework and prefill engine while using the Cerebras decode engine and KV Portal.

The SDK also provides development aids, including a mock Cerebras decode replica and a reference integration with an open-source disaggregated inference framework and prefill engine. The full stack is intended for partners prioritizing time to market, while the decode-only option is intended for those seeking more control.

Access and deployment

Early-access partners with the solution for their deployments include Amazon Web Services, Fireworks AI, Gimlet Labs and General Compute. The company is expanding access to additional partners.

Source

  1. Cerebras

Companies in this story

Related stories

Explore

More articles