The Pulse
Microsoft’s LLM-42 Makes Deterministic Inference Selective
LLM-42 uses a fast decoding path and verification to make selected language-model requests repeatable without slowing every request. Its evaluation reports higher throughput than deterministic SGLang in several workloads when only some requ
AI.info Team · Written with AI ·
“LLM-42 mostly re-uses existing kernels unchanged and incurs overhead only in proportion to the traffic that requires determinism.”
The authors of LLM-42 write
LLM-42 takes the stage at SOSP
The 32nd ACM Symposium on Operating Systems Principles begins its scheduled activities in Prague on Tuesday, September 29, with registration and workshops; opening remarks are scheduled for Wednesday, September 30. Microsoft Research and University of Washington researchers’ LLM-42 is on the program, with its presentation scheduled for Friday, October 2. The paper proposes a way to make selected language-model requests repeatable without forcing every request through slower deterministic GPU kernels.
The conference appearance is not the paper’s first public release: the current arXiv preprint, version 2, is dated January 30, 2026. The first version, submitted January 25, listed five authors, including Arkaprava Basu; version 2 and the Microsoft Research publication page list Raja Gond, Aditya K Kamath, Ramachandran Ramjee and Ashish Panwar. The paper lists Gond, Ramjee and Panwar with Microsoft Research and Kamath with the University of Washington. The work targets a systems problem: identical prompts and decoding settings can produce different outputs when dynamic batching changes how GPU operations are grouped.
A fast path, then a check
LLM-42 keeps normal, dynamically batched decoding as its fast path. It generates a window of candidate tokens, then replays them in a verifier using a fixed-shape reduction schedule. The verifier releases matching tokens and rolls back candidates that fail the check, recomputing from the last consistent state.
The design draws on speculative decoding, where candidate tokens are checked before acceptance, but uses verification to enforce repeatability rather than to accelerate ordinary decoding. The paper says requests opt in through an is_deterministic flag; requests that do not need deterministic output skip verification. The guarantee concerns fixed model and sampling settings, not different temperatures or sampling parameters, which can intentionally produce different outputs.
The motivation is the arithmetic underneath inference. Floating-point addition is not associative, and GPU kernels may change the order of reductions as batch shapes change. Disabling dynamic batching or using batch-invariant kernels can address that source of variation, the researchers write, but those approaches can constrain kernel performance and require specialized implementations.
Tests show gains when only some requests need repeatability
The researchers evaluated LLM-42 with Llama-3.1-8B-Instruct on four NVIDIA H100 PCIe GPUs, comparing it with SGLang 0.5.3rc0 in deterministic and non-deterministic modes. Their workloads included ShareGPT and an ArXiv dataset, alongside synthetic input and output lengths. In one reported example, when one request in a batch of 11 required deterministic output, LLM-42 reached 911 tokens per second—2.2 times SGLang’s deterministic-mode throughput and within 3% of its non-deterministic mode.
Across the offline tests, the paper reports that at a 10% deterministic-request share, LLM-42 was 33% faster than deterministic SGLang on ShareGPT and up to 48% faster on a synthetic workload with 2,048 input tokens and 256 output tokens. The advantage narrowed when every request required deterministic output: on ShareGPT, LLM-42 was 6% slower than SGLang’s deterministic mode. The authors also report that recomputed tokens reached at most 10.97% in their tested configurations, in the ArXiv workload when all traffic required deterministic output.
The prototype still pauses other requests
The measurements come from a research prototype, not a general production deployment. The paper identifies two current limits in online serving: verification creates a global pause that temporarily stalls in-flight requests, and prefill is not batched, reducing efficiency for short prompts. The authors say those implementation issues remain to be addressed.
LLM-42 also does not currently integrate with inference systems that already use conventional speculative decoding. Its contribution is narrower: it offers a selective route to reproducible outputs while preserving ordinary fast-path execution for other requests. The code is available in Microsoft’s LLM-42 repository, built on SGLang.