The Pulse
Edge0 Serves a 35B MoE at 20 Tokens a Second in 3 GiB
Researchers behind Edge0 describe an inference engine that serves a 35-billion-parameter mixture-of-experts model from SSD storage at 20.4 tokens per second. The system uses 2.9 GiB of peak active memory on a 24GB Apple Silicon machine by p
AI.info Team ·
A 35-billion-parameter language model runs at 20.4 tokens per second while using 2.9 GiB of peak active memory in a new system called Edge0, according to a paper published on September 16. The result comes from storing most of the model’s weights on SSD rather than keeping them resident in system memory.
The figure needs a qualification: Edge0 does not shrink the model to 3 GiB. Its 4-bit checkpoint occupies 19.5 GB on disk, while the active-memory measurement covers the portion needed during inference on a short context. The researchers tested the 35B tier on a Mac mini with an M4 Pro chip and 24GB of unified memory.
The work is described in “The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction” by Yu Lin, Yiming Wang, Runyuan Cai, Hanze Liu, and Xiaodong Zeng. The authors also released the framework, model checkpoints, adapters, and routing components under the Edge0 project.
Why SSD offload normally stalls MoE inference
Mixture-of-experts models activate only part of their parameters for each token, but the full collection of expert weights still has to live somewhere. The paper’s 35B model activates roughly 3 billion parameters per token, yet its 4-bit checkpoint contains 19.5 GB of weights. Sparsity reduces computation; it does not remove the storage requirement.
Edge0 keeps expert weights in memory-mapped files and loads them from storage when needed. A conventional offload system must wait for one layer’s output before it knows which experts the next layer will use, putting storage reads directly on the decode path. That dependency can turn every generated token into a sequence of disk waits.
Edge0 addresses the delay with a trained prerouter. Each routing head predicts the next layer’s expert selection one token ahead, allowing the system to begin loading the next expert set while the current token is still being processed. The prediction then becomes the routing decision itself, so the staged experts and the experts used for computation are the same set.
The 35B release uses 256 experts across 40 layers
The released 35B tier is built on Qwen3.6-35B-A3B and uses 40 layers with 256 experts per layer, plus a shared expert. Edge0 routes each token to four experts, uses int4 affine group-64 quantization, and supplies 33 prerouter heads. The project also trains an unmerged recovery LoRA adapter to recover quality lost through quantization and replaced routing.
That adapter stays separate from the quantized base model. The paper says merging the adapter into the int4 weights and quantizing again removes most of its effect, so Edge0 serves the adapter as a parallel branch during inference. The model, adapter, and prerouter weights are packaged together in the released checkpoint.
Measurements in the paper put decode speed at 20.4 tokens per second and peak active memory at 2.9 GiB for the 35B tier. Prefill reaches 113 tokens per second on a cold run and 140 tokens per second after the relevant data is warm in the page cache. A fully resident comparison using the same 4-bit weights reaches 3.9 tokens per second while occupying 18.2 GiB, according to the authors.
Prediction improves speed when the model does not fit
The paper also compares ordinary on-demand loading with one-token-ahead staging on a 16GB MacBook with an M2 processor. The 18.4 GiB release directory does not fit in physical memory, forcing the system to fault expert data back from SSD during generation.
With four experts selected per layer, on-demand streaming produces 3.5 tokens per second in the authors’ same-session test. The prerouter reaches 6.4 tokens per second, an 82 percent increase. At routing widths of two and eight, the measured gains are 80 percent and 84 percent respectively.
Edge0 does not achieve those results by reading dramatically less data. At the four-expert setting, the prerouter reads 58.9 MiB per step compared with 50.9 MiB for on-demand loading. Its advantage comes from issuing fewer, larger reads earlier, reducing the time the main execution thread waits for storage.
Quality stays close, but reasoning shows a larger gap
On OpenCompass evaluations, the 35B Edge0 tier averages 79.2 out of 100 across five benchmarks, compared with 83.2 for the original fp16 Qwen3.6 base model. The mean difference is 3.9 points. Edge0 scores 90.9 on HumanEval against 95.1 for the fp16 model, 79.8 against 81.8 on GPQA-Diamond, and 81.0 against 84.6 on MMLU-Pro.
The largest difference appears on AIME 2026, where Edge0 scores 86.6 against 92.7 for the fp16 model. The paper identifies long-chain reasoning as the area where int4 quantization and routing replacement remain most visible. The recovery LoRA narrows the quality loss on the other tests but does not eliminate it.
Edge0 also includes an 8B tier based on a Ling 3.0 hybrid model. That version reaches 28 tokens per second with 1.5 GiB of peak active memory in the authors’ Mac mini test, but the 35B result is the more significant demonstration because its checkpoint is far larger than the available memory budget.
Apple Silicon is the current boundary
The project’s open-source repository says the current implementation uses Apple’s MLX framework and runs on Apple Silicon Macs from the M1 through M4 generations. CUDA support is planned in the repository architecture but is not implemented in the released backend, so the reported results do not establish performance on Nvidia GPUs or ordinary Windows and Linux systems.
The engine also serves one request at a time in FIFO order. The paper says batching and concurrent serving are outside the current evaluation because they change the active expert working set. Long contexts add memory through the KV cache, meaning the 2.9 GiB figure is not a fixed requirement for every workload.
Edge0’s contribution is therefore narrower than a 35B model that literally fits inside 3 GiB. It demonstrates that a consumer machine can stream a much larger sparse model from local storage while keeping active memory low, provided the system can predict the next expert set accurately enough and overlap storage work with computation. The released 35B checkpoint remains a 19.5 GB file; the measured 2.9 GiB is the memory needed while running it.