# RadixArk team describes serving Meta’s Muse Spark Open on one TPU host

Published 2026-11-10 · AI.info Team · AI.info

> The RadixArk team describes running Meta’s 816-billion-parameter Muse Spark Open on a single TPU v7 host using SGLang-TorchTPU. Its post details a megakernel, attention and expert-processing changes, and results on the AgentX benchmark.

## One host, eight devices

A post by Haolin Fu, Xiaoyu Zhang, Junyan Xu, Wangyuan Zhang, Haoguang Cai and the RadixArk Team describes serving [Meta](https://ai.info/companies/meta)’s Muse Spark Open, an 816-billion-parameter mixture-of-experts model, on one TPU v7 host with four chips. The authors write, “We serve Meta’s Muse Spark Open, an 816B-parameter mixture-of-experts model, on a single TPU v7 host of four chips, with SGLang running on the TorchTPU [PyTorch](https://pytorch.org/blog/optimizing-muse-spark-open-on-tpu-v7-with-sglang-torchtpu/) backend.” Their post describes SGLang-TorchTPU with Pallas kernels handling performance-critical work.

The post says Muse Spark Open has about 42 billion active parameters per token. Its 62 decoder layers are all mixture-of-experts layers, and each layer routes a token to eight of 256 experts.

Each TPU v7 chip has two chiplets, which the framework exposes as separate devices; the host therefore has eight devices. The model is sharded across those devices with tensor parallelism, assigning one shard to each.

## A single program for a serving step

The central change is a megakernel: one Pallas program on each device handles all 62 layers of a decode or verification step. SGLang continues to run the embedding, draft model, KV-cache write, language-model head and sampling around that program, the post says.

For a single request, the megakernel takes 1.95 milliseconds per output token, compared with 3.78 milliseconds on the native path, a 1.94-times speedup. Figure 4 measures the verify-step comparison with DFlash2 speculative decoding on GSM8K using an earlier Muse Spark checkpoint. The RadixArk team reports a 1.85-times speedup for two concurrent requests; the gain narrows as concurrency increases.

The authors say the verification step runs within 1.5 to 1.7 times the time needed simply to stream its weights from high-bandwidth memory. To keep transfers from interrupting computation, the kernel pipelines weight loads, KV-cache reads and transfers among devices.

## Changes to attention and expert processing

For attention, the team removed the running maximum used in the online softmax calculation. The post says Muse Spark’s query and key normalization, together with the model’s fixed query gain, bounds attention scores and makes the maximum unnecessary; without that dependency, KV-cache chunks can be processed in a pipeline.

In the serving system, the pipelined attention loop increased decode speed per request by 18% for two requests at a 500,000-token context, according to the team. A separate kernel test with random routing showed decode steps with many requests running 13% to 19% faster after the team split its expert-processing work into two loops.

The team also used DFlash2 speculative decoding, in which a smaller draft model proposes tokens that the full model checks together. On GSM8K with real acceptance, the authors report that DFlash2 reduced the megakernel’s time per output token for one request from 3.93 milliseconds to 1.95 milliseconds, a 2.0-times speedup.

## Agent workload results

For the 1-million-context version of SemiAnalysis’s AgentX coding benchmark, the post reports a range from 433 output tokens per second per request at 6,101 tokens per second per chip, to 25,906 tokens per second per chip at 95 tokens per second per request. Every TPU result shown finished requests within 47 seconds at the 90th percentile.

The authors describe AgentX as based on 393 opt-in Claude Code sessions, replaying request lengths, shared prefixes, tool pauses and subagent structure with synthetic tokens replacing the original content. The comparison uses the benchmark’s 1-million-context trace set, where prompts reach about 990,000 tokens; throughput counts both input and output tokens.

The TPU configuration uses one host with four chips, FP8 compute and NVFP4 expert weights. The post identifies the cost figures as InferenceX’s external owning costs: $1.21 per TPU v7 chip-hour and $1.86 per GB200 GPU-hour.

Each device has about 20 GiB left for KV cache after the model weights. The post describes a sequence-on-lane layout for FP8 cache data and L2 HiCache on TPU, with session-aware eviction, as parts of its approach to increasing cache capacity.

## Sources

1. Original source · Blog post [PyTorch](https://pytorch.org/blog/optimizing-muse-spark-open-on-tpu-v7-with-sglang-torchtpu/) pytorch.org

## Related stories

- [Google details bitwise-matched RL training across 64 TPU chips](https://ai.info/news/google-bitwise-matched-rl-training-tpu)
- [Team says AI agents helped integrate SGLang with Spyre in under a week](https://ai.info/news/sglang-ibm-spyre-ai-integration-under-a-week)
- [vLLM 0.33 Drops CUDA 12.9 Builds](https://ai.info/news/vllm-033-drops-cuda-129-builds)
- [PyTorch 2.15 Makes CUDA 13.2 Its Default GPU Build](https://ai.info/news/pytorch-2-15-cuda-13-2-default)
- [Epoch AI introduces index for measuring inference capacity](https://ai.info/news/epoch-ai-inference-performance-index)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/pytorch-muse-spark-one-tpu-host
