The Pulse
Ai2 Opens Olmo-core 3 for Trillion-Parameter MoE Training
Ai2 released Olmo-core 3, an open training framework designed for large mixture-of-experts models. Its tests reached 1.2 trillion parameters on 512 NVIDIA B300 GPUs, but used random routing rather than measuring the quality of a trained mod
AI.info Team · Written with AI ·
Ai2 says its new Olmo-core 3 training stack has been benchmarked on a 1.2-trillion-parameter mixture-of-experts model, with 58.36 billion parameters active for each token and the work spread across 512 NVIDIA B300 GPUs. The institute released the framework on October 1, pitching it as open infrastructure for training much larger sparse models. The benchmark measured system performance—not whether a model trained with the stack produces good answers.
A trillion parameters, with a test-run caveat
In the 1.2-trillion-parameter test, Ai2 recorded a peak observed throughput of 858 TFLOP/s per GPU, a measure of model computation. But the company says the test used random routing to assess the system. It does not establish the quality of a trained model, or show that the configuration completed a full training run.
Ai2 also describes a 2.38-trillion-parameter configuration using DeepEP v2, an alternative method for handling communication between experts on different GPUs. The institute calls that a short-capacity test, not sustained training. The distinction matters: these figures indicate that the infrastructure can be configured at those scales, not that Ai2 has trained a capable model with trillions of parameters.
Keeping experts on the GPUs
Mixture-of-experts models divide parts of a network into specialized components called experts. The model routes each token to only a subset, so the active computation can remain far below the total number of parameters. Yet the full model still has to be stored and updated, while routing inputs among experts across a cluster adds communication overhead.
Olmo-core 3 changes how Ai2 handles that work. Its earlier MoE implementation used fully sharded data parallelism, gathering and resharding weights for each small batch. The new system uses distributed data parallelism and keeps experts resident on GPUs, sending relevant data to them instead. Ai2 combines that design with expert and pipeline parallelism and a distributed optimizer, splitting the model and training state across devices.
A preliminary comparison on eight B300 GPUs measured a 47-billion-parameter MoE at 52,000 tokens per second per GPU with Olmo-core 3, against 19,400 with Ai2’s earlier implementation—about 2.7 times the throughput. In a separate test, increasing the expert pool from eight to 128 while selecting four experts per token raised total model capacity from 4.6 billion to 47 billion parameters. Active parameters stayed near 3.2 billion per token, and throughput declined by less than 5%.
Lower precision, higher throughput
Ai2 also tested MXFP8, a lower-precision number format, on four B300 GPUs. In that controlled benchmark, training throughput rose about 21% over its BF16 baseline, while peak active memory fell from 103 GiB to 95 GiB. The post says most of the speed improvement came from feed-forward computation and moving data between experts, rather than attention alone.
The institute flags trade-offs rather than treating each optimization as an automatic win. Its report describes cases where overlapping communication and computation slowed end-to-end execution, and where a metric intended to encourage balanced routing improved even as the actual workload grew less balanced. Ai2 calls that failure “token gerrymandering.”
Code is public; the next Olmo is still ahead
Ai2 says researchers and developers can use the framework to train MoEs and experiment with routing and parallelism. The public Olmo-core repository includes installation instructions, including a PyPI package, and carries an Apache-2.0 license. Making the training system available gives outside teams access to more than model weights, though reproducing tests at this scale still requires substantial compute and compatible hardware.
The release is also the foundation for Ai2’s next-generation Olmo, which the institute says will use an MoE architecture. The announcement offers no performance results for that future model. For now, the concrete milestone is narrower: an open stack that Ai2 has tested at trillion-parameter scale, alongside results that separate hardware capacity from completed training and model quality.
“These tests used random routing to measure system performance, rather than the quality of a trained model.”
Kyle Wiggers, Ai2Comms, in Ai2’s announcement