The Pulse

SGLang Reports 18.7 ms Scoring for 16-Option Decisions

SGLang’s multi-item scoring benchmark measured 18.7 ms p95 latency for 16-option decisions with Qwen3-0.6B. LMSYS says the method reuses shared query computation while keeping candidate evaluations separate, but the tests measure serving pe

AI.info Team · Written with AI · fact-checked · 2026-09-26

SGLang measured 18.7 milliseconds of p95 latency to score 16 candidate actions with Qwen3-0.6B, compared with 39.6 milliseconds using one-token generation. The benchmark, published September 25 by LMSYS, examines how the runtime serves decision tasks that need a score or category rather than a generated explanation. Its multi-item scoring mode reuses shared query computation while keeping each candidate evaluation separate.

One shared query, 16 candidate checks

The tests use choice tasks from the Open-Jev dataset, where a question and its context stay the same while a model evaluates multiple possible answers. SGLang’s multi-item scoring, or MIS, lets the runtime reuse that repeated context within a request. Each candidate still attends only to the shared query and its own text, so the candidates do not become one joint choice prompt.

At 16 options, the post reports p95 decision latencies of 18.7 milliseconds for Qwen3-0.6B with MIS, 39.6 milliseconds with generation, and 24.3 milliseconds with single-item scoring. Qwen3-8B measured 20.6 milliseconds with MIS, against 54.1 milliseconds for generation and 53.1 milliseconds for single-item scoring. The smaller improvement on Qwen3.5-4B was 55.7 milliseconds with MIS versus 84.8 milliseconds with generation and 74.4 milliseconds with single-item scoring.

“MIS latency is nearly flat as the number of options grows from 2 to 16, consistent with amortizing shared-query computation across candidates,” write Sundara Raman Ramachandran, Chuanrui Zhu, Qing Lan, Shri Rajamanikandan Vasudevan, Jian Sheng, I-Ting Chen and Fedor Borisyuk.

The Score API asks for specific labels

The post also describes SGLang’s /v1/score API, which lets a caller specify the label-token scores it needs, such as Yes and No, rather than relying on a generation response’s top-k log probabilities to include those labels. The runtime can skip token sampling, avoid calculating unnecessary log probabilities for input tokens and gather requested label scores in batches. The authors caution that this does not remove the vocabulary projection or full-distribution normalization.

That API and MIS address different parts of the workload: explicit labels define what scores the application receives, while MIS reuses computation across candidates. Enabling the API alone does not turn on MIS. The example configuration requires a supported model and backend; for the Qwen3-0.6B example, LMSYS specifies FlashInfer and disables radix caching and chunked prefill.

Lower latency does not establish better decisions

LMSYS tested Qwen3-0.6B, Qwen3-8B and Qwen3.5-4B on one NVIDIA H200 GPU with SGLang on CUDA 13.0. The benchmarks compare p95 end-to-end time to finish scoring a question’s candidates, not the latency of one candidate alone. At higher offered loads, MIS’s advantage varied by model; the post says it was not consistently faster at low load.

The results cover serving performance, not whether the different scoring approaches choose equally well. The authors also warn that offered request rates do not by themselves show achieved throughput, and recommend measuring tail latency under the intended deployment load. For teams using models to route actions or rank answers, the reported speedup concerns how quickly scores arrive—not whether those scores lead to the right action.