Skip to content
AI.info

The Pulse

AWS makes SGLang-Neuron publicly available as a preview

The plugin lets developers serve models and generate reinforcement-learning rollouts on Trainium through SGLang. Its repository describes the release as a public preview whose features and supported models can change.

AWS makes SGLang-Neuron publicly available as a preview

Developers can now use SGLang to serve AI models on AWS Trainium chips through a publicly available plugin, according to an Amazon Web Services post on November 12. RadixArk released the plugin, SGLang-Neuron, with Annapurna Labs, and it is accessible through an AWS Deep Learning Container.

The project repository calls the release a public preview and says its features, APIs and supported models can change in later releases. The plugin also supports generating reinforcement-learning rollouts on Trainium. Developers launch models through the SGLang server interface and invoke the endpoint using the same generation APIs.

The public release follows an October 22 beta announcement by the SGLang team, which said initial access was limited to a small group of teams. That beta covered inference serving and reinforcement-learning rollout generation on Trainium2 and Trainium3.

How the plugin works

SGLang-Neuron is an out-of-tree plugin connecting SGLang to Trainium. Installing it registers the Neuron platform through plugin entry points; setting SGLANG_PLATFORM=neuron selects the platform when the server starts.

SGLang core continues to provide the API server, request lifecycle, RadixAttention and prefix-cache bookkeeping. The plugin implements Trainium-specific models, attention, mixture of experts and the graph runner.

TorchNeuron registers Trainium as a native PyTorch device. The plugin uses standard PyTorch operations for model execution and incorporates kernels written with the Neuron Kernel Interface. Those operations and kernels are compiled with torch.compile into programs that the graph runner prepares and executes during serving.

Serving configurations

The plugin also supports prefill and decode disaggregation. In that configuration, a prefill server computes the prompt KV cache and transfers it to a decode server using NIXL over Elastic Fabric Adapter; the decode server generates the remaining tokens. Each server retains SGLang’s scheduler and request lifecycle, while the plugin manages Trainium execution and the cache-transfer path.

The repository says large language models in the preview run on Trn3 instances, while diffusion models run on trn2.48xlarge. The repository lists a Neuron driver, Docker Engine and Neuron SDK 2.33.0 among the requirements for the container setup. For disaggregated serving, it lists Elastic Fabric Adapter between nodes.

AWS demonstrates the plugin on a Trainium3 Amazon Elastic Compute Cloud instance. Its example launches openai/gpt-oss-120b inside the container and sends a request to the model server. The example sets both tensor parallelism and expert parallelism to 8, with a context length of 10240 and a maximum of 64 running requests.

Earlier rollout experiment

In its beta announcement, the SGLang team described an early run in which it generated rollouts for Qwen3-MoE-30B-A3B on Trainium3. Miles, its reinforcement-learning post-training framework, trained the GRPO policy with Megatron on a single H200 node. The team said the setup completed a stable training run.

The team described rollout generation and policy updates as distinct phases of reinforcement-learning post-training. In its example, Miles connected to SGLang on Trainium as an external rollout engine, while training ran on the separate node.

Sources

Companies in this story

Related stories

Explore

More articles