The Pulse
Microsoft’s MeshRT Claims Up to 56× More Wafer-Scale Throughput
Microsoft Research’s MeshRT paper reports up to 56× higher throughput than a state-of-the-art wafer-scale system, alongside lower latency than measured single-instance SGLang results on an Nvidia H200. The summary does not name the wafer-sc
AI.info Team · Written with AI ·
“Implemented on a commodity wafer-scale accelerator, MeshRT preserves ultra-low latency while improving throughput by up to 56× over the state-of-the-art system for wafer-scale accelerators.”
Congjie He and co-authors, the authors of the MeshRT paper, write. “MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference”
Microsoft Research’s MeshRT paper reports a maximum 56× throughput gain for large language model inference on a wafer-scale accelerator, compared with the state-of-the-art system for wafer-scale accelerators. The summary does not name that comparison system or give an absolute tokens-per-second figure. The researchers also report 4.5 to 7 times lower latency than measured single-instance SGLang results on Nvidia H200 hardware.
The work is listed among the publications for the ACM Symposium on Operating Systems Principles (SOSP) 2026, scheduled for September 29 to October 2 in Prague. Microsoft Research’s page lists September 4, 2026, as the publication date for the MeshRT paper summary; October 16, 2026, is the modification date. The authors argue that runtime coordination has made it difficult to get both low latency and high aggregate throughput from these very large accelerators.
MeshRT moves scheduling decisions into compilation
Batching groups requests so hardware can process them together; mixture-of-experts models route different tokens to different subsets of a model. Both make workloads less predictable while they run. Microsoft’s summary says existing wafer-scale systems struggle to support that runtime dynamism efficiently, forcing tradeoffs that can restrict supported models and reduce throughput.
MeshRT’s approach is to compile dynamic inference events into schedules assigned to individual cores before execution. The paper describes an architecture it calls “compile-time governance of runtime dynamism,” with mechanisms for asynchronous, decentralized execution. Moving more decisions out of the live execution path is intended to reduce runtime overhead without requiring systems to avoid dynamic techniques altogether.
The claim is about the runtime and system design, not a new wafer-scale chip. Microsoft says the implementation ran on a commodity wafer-scale accelerator, but the public summary does not identify the device model or provide detailed workload-by-workload results. Those omissions limit how much can be inferred from the headline multiplier alone.
The H200 comparison has two different baselines
Latency and throughput figures in the summary refer to different comparisons. The up-to-56× throughput result is measured against a state-of-the-art system for wafer-scale accelerators; the summary does not identify it by name. The 4.5–7× latency reduction is measured against single-instance SGLang results on an Nvidia H200; it should not be read as a 4.5–7× throughput comparison with an H200 cluster.
The authors also report 1.6–12.1× higher decode energy efficiency in a low-latency regime, but that range comes with a qualification: it assumes ideal linear scaling of the measured GPU results to an area-normalized H200 cluster. It is therefore a modeled comparison, not a direct measurement of MeshRT against a physically tested H200 cluster. The distinction matters because the study’s most dramatic number is tied to a wafer-scale baseline, while the GPU-related efficiency estimate depends on an assumed scaling behavior.
A system result, not a universal serving guarantee
MeshRT adds to Microsoft Research’s work on software for wafer-scale inference, where large numbers of processing cores and on-chip connections make placement, communication and scheduling part of the performance problem. The paper’s central proposition is that compiling plans for dynamic work can let a wafer-scale runtime retain low latency while serving more requests. That is a systems result, but the available summary does not establish how the gain transfers to other chips, model sizes or production request patterns.
For infrastructure teams, the useful takeaway is the distinction between the demonstrated target and the broader promise: MeshRT reports up to 56× throughput against a state-of-the-art wafer-scale system, alongside lower latency than a measured single-H200 SGLang setup. The summary does not name the wafer-scale comparison system or provide an absolute throughput number, and the energy-efficiency range relies on idealized GPU scaling. Those boundaries define what the reported benchmark does—and does not—show.