The Pulse
EUROPA reports faster 1T-scale MoE training on public supercomputer
The EUROPA consortium says it increased training throughput on the JUPITER supercomputer from 90 to about 1,300 tokens per second per GPU. Its experimental mixture-of-experts configuration has 951.9 billion parameters, with 32.8 billion active per token.

A European consortium says it increased training throughput for a roughly trillion-parameter mixture-of-experts model from 90 to about 1,300 tokens per second per GPU on the publicly funded JUPITER supercomputer. The EUROPA consortium described the work in a November 12 post on PyTorch’s site, reporting the result on 384 NVIDIA GH200 GPUs across 96 nodes.
The experimental model configuration has 951.9 billion parameters, of which 32.8 billion are active per token. EUROPA consists of Domyn and Fraunhofer IAIS and IIS, and its work is part of the EuroHPC Frontier AI Grand Challenge.
A shared machine
JUPITER is a EuroHPC exascale supercomputer hosted at Forschungszentrum Jülich. Unlike a cluster reserved for a single model developer, it is shared research infrastructure, with hardware configured to support scientific work as well as AI training.
The model configuration used in the experiments has 74 layers and 384 routed experts, with the top eight selected for each token. Its pretraining sequence length is 65,536 tokens, and the setup uses the Muon and AdamW optimizers.
Domyn cautioned that the setup was still changing and that some details could become outdated. In its companion account, the company said some configuration values could change before the training run starts.
Working around bandwidth limits
Each JUPITER Booster node has four GH200 superchips, each pairing a Hopper GPU with a Grace CPU. The connection between nodes is much slower than the link between a GPU and its own CPU, so the team sought to overlap network traffic with computation and reduce the amount of data sent between nodes.
The consortium used 384-way fully sharded data parallelism, or FSDP, with four-way expert parallelism inside each node and no pipeline parallelism. That kept the mixture-of-experts token exchange on the faster links within a node.
Another constraint was host memory. After the kernel’s share, about 108 GiB per GPU was usable, and the team found that some nodes began jobs with substantially less. That limited how much training state could be moved out of GPU memory.
Changes to the training stack
The team used FlexShard, now part of TorchTitan, to assign whole expert matrices to individual ranks. Each rank could then process its own experts while dense matrices were in transit, overlapping that communication with computation.
The researchers said that moving a computationally intensive part of the Muon optimizer step from the CPU to the GPU reduced the optimizer step from 41.7 seconds to 2.2 seconds. In that comparison, throughput rose from 441 to 597 tokens per second per GPU.
They also copied activation checkpoints to Grace memory after the forward pass and brought them back before recomputation in the backward pass. The team reported that offloading raised throughput from 370 to 508 tokens per second per GPU, while sequence length grew first to 49,152 tokens and then, with Muon, to 65,536.
Memory allocation and remaining work
After a software-stack upgrade, the researchers encountered allocation retries even though GPU memory remained free. Their investigation found memory held in separate stream caches that the compute stream could not reuse.
They adapted their gather operation to use an FSDP2 flag that puts its allocation setup on the compute stream. The consortium said the change reduced ranks experiencing allocator retries from 384 to zero; peak reserved memory fell from 88.1 to 74.6 GiB, and throughput rose from 996 to 1,210 tokens per second per GPU.
The team contributed changes to PyTorch and TorchTitan, including activation-checkpoint offloading hooks. It also tested ways to balance training data and expert workloads; Domyn said an expert-routing constraint improved throughput but might affect training efficacy, and that the team was still evaluating whether to keep or relax it.
Data balancing groups documents with similar attention costs into the same training steps, which can affect training dynamics. The group plans a dedicated long-context phase to compensate for that effect.
Sources
Related stories
- AMD-powered Lux supercomputer opens to researchers at Oak Ridge
- Team says AI agents helped integrate SGLang with Spyre in under a week
- RadixArk team describes serving Meta’s Muse Spark Open on one TPU host
- PyTorch 2.15 Makes CUDA 13.2 Its Default GPU Build
- RIKEN starts AI-for-science initiative with shared platform plan