The Pulse
Team says AI agents helped integrate SGLang with Spyre in under a week
A two-person team used AI coding agents to build an experimental SGLang backend for IBM’s Spyre accelerator in less than a week, according to a PyTorch blog post. The authors report higher throughput than a reference PyTorch implementation and comparable performance to vLLM on the workloads measured.

A two-person team used AI coding agents to integrate the SGLang serving stack with IBM’s Spyre accelerator in less than a week, producing a plugin of roughly 1,500 lines, according to a PyTorch blog post published November 12. The open-source backend is experimental, and its repository describes it as a working prototype rather than a production-supported backend.
The authors report that SGLang on Spyre achieved approximately 3.6 times the throughput of a reference PyTorch inference implementation using torch-spyre. They also found its performance on par with vLLM on the workloads measured for Granite 3.3 8B and Gemma 4 26B A4B.
Building on existing support
Spyre is IBM’s accelerator for enterprise AI inference. Its existing torch-spyre integration registers it as a PyTorch device and connects it to torch.compile, exposing hardware support through standard PyTorch APIs.
The team used that existing support rather than writing the hardware implementation from scratch. AI coding agents took PyTorch implementations of accelerator operations as reference code and adapted integration code to the serving framework’s backend interfaces.
Four small patches extended the serving framework’s backend support: two made existing execution paths available without CUDA-specific dependencies, while two registered Spyre as a supported device and an attention backend that does not use Triton. The compatibility changes can also support other accelerators; the registration changes are specific to Spyre.
What the plugin reuses
The integration draws on the existing vLLM implementation for Spyre, including code for grouping inputs into buckets, handling tensor shapes and allocating the KV cache. Existing implementations of RMSNorm, SiLU, GeLU, rotary embeddings and linear layers were connected to SGLang’s operator dispatch.
In the initial integration, an attention base class provided the surrounding logic, leaving the backend to override the inner scaled dot-product attention operation. The authors say reuse of that class accounted for roughly two-thirds of the difference between the roughly 1,500-line plugin and the approximately 3,500-line existing vLLM backend.
The framework’s device-agnostic KV pool also allowed a single method override for buffer creation, although allocation still required PyTorch-based handling where the existing path relied on Triton. Prefix sharing through RadixCache worked without Spyre-specific changes because the scheduler resolves it before execution reaches the accelerator.
Prototype scope
The open-source repository describes the backend as a working prototype, not yet supported for production. Its documented scope covers decoder-only text generation, tensor parallel size 1 and FP16, and max_running_requests=1 is the validated configuration.
The repository describes SGLang RadixCache with 128-token pages, while the blog’s example sets a 64-token page size. CUDA graphs are not used; decoder blocks and attention kernels are compiled with torch.compile.
The blog’s example assumes that the Spyre runtime, torch-spyre, the plugin and its compatibility patches have already been installed and configured. The repository also states that torch-spyre must be built on a supported Spyre host with the IBM runtime and compiler available.
Sources
Related stories
- RadixArk team describes serving Meta’s Muse Spark Open on one TPU host
- AWS makes SGLang-Neuron publicly available as a preview
- NEAR AI reports SGLang flaw exposed inference server and user data
- PyTorch 2.15 Makes CUDA 13.2 Its Default GPU Build
- EUROPA reports faster 1T-scale MoE training on public supercomputer