TokenRouter: Efficient Serving System for Token-Level LLM Routing

AI summary

Overview

Research area: LLM inference serving systems and model routing (Natural Language Processing / systems for machine learning).

Technical level: Intermediate. The paper assumes familiarity with LLM decoding, continuous batching, KV-cache management, and tensor parallelism, but the core design ideas are explained conceptually.

Scope: The paper presents TokenRouter, a serving system for token-level LLM routing, describing its programming interface, model-centric runtime, scheduling method, and an evaluation across five routing algorithms, three workloads, and multiple model pairs.

What This Paper Is About

LLM serving systems commonly route whole queries or sessions to one model, but recent algorithms route at the finer granularity of individual tokens, sending only hard tokens to a large model and decoding the rest with a small one. Existing serving systems assume one model per request and one model per server, so when the target model can change every decoding step they suffer from slow-model stalls, delayed batch admission, and large implementation burden. The paper's goal is a serving system that lets developers express token-level routing algorithms simply while the runtime keeps each model's batches efficient.

Key Contributions

1. Request-centric programming interface. A route–send–receive interface that lets developers describe diverse token-level routing algorithms by tracing how a single request flows across cooperating LLMs, hiding batching and asynchronous execution.
2. Asynchronous execution for serving optimization. Decoupled tri-loop execution (client-server, decoding, and inter-model loops) that addresses step desynchronization across cooperating LLMs, plus a handoff-resume mechanism that reduces inter-model switches to little more than a token append.
3. Delayed batching for scheduling optimization. A scheduler that buffers incoming peer requests until a threshold B is reached, where the throughput-optimal threshold B* is derived from a mathematical model of the system.
4. A concrete system and open code. An implementation on top of SGLang 0.5.1 that exposes a single external interface, making it a drop-in replacement for standard single-LLM servers, released at https://github.com/thu-nics/TokenRouter.

Main Findings

• End-to-end throughput and latency gains: Across all 15 algorithm–workload combinations, TokenRouter outperforms the baselines, improving throughput by 2.01–64.15× compared with the stronger baseline, and reducing end-to-end latency of the Std. Serving baseline by 2.03–63.64×.
• Gain breakdown at concurrency 8: Engineering optimizations with extended CUDA graphs raise throughput from the base implementation's 132.78 to 230.79 token/s (1.71× the official R2R implementation's 134.9 token/s). Asynchronous execution and delayed batching further raise it to 296.86 and 372.48 token/s, reaching 2.76× overall.
• Throughput–speed trade-off: As concurrency increases from 1 to 16, TokenRouter's throughput increases by 8.61× while retaining 51.7% of its single-user speed; official R2R reaches only 5.14× and retains 31.3%. At concurrency 16, TokenRouter achieves 18.58× higher throughput than official R2R at concurrency 1 while still providing 1.13× higher per-user speed.
• Robustness to output length: Comparing low-effort and high-effort reasoning workloads (max output 2,048 vs. 8,192 tokens), TokenRouter's throughput holds steady while Std. Serving loses 58.1–85.2% of its throughput.
• Original algorithmic settings (concurrency 4, Table 2): TokenRouter reaches 244.56 token/s for R2R (vs. 89.62 official), 149.31 for CITER (vs. 17.16), 76.02 for Co-LLM (vs. 3.46), and 140.58 for R-Stitch (official code not released), reducing latency by 2.78–21.61× and improving throughput by 2.73–21.97× over official implementations.
• Across model pairs (Table 3): R2R vs. TokenRouter gives 0.6B/8B 105.1 → 336.98 (3.21×), 0.6B/32B 77.08 → 197.40 (2.56×), 1.7B/8B 103.37 → 285.41 (2.76×), and 4B/8B 106.67 → 211.81 (1.99×), i.e. 1.99–3.21× improvement across model pairs.
• Lower-concurrency results (concurrency 1): TokenRouter achieves 2.12–19.74× higher throughput and a 2.12–18.83× latency speedup over standard serving, and 1.60–19.74× higher throughput than the stronger available baseline.
• Ablation of optimizations: CUDA graphs improve throughput by 1.88–2.12×; asynchronous execution and delayed batching further improve throughput by 1.12× and 1.31× respectively at concurrency 4.
• Parallel strategy: Increasing LLM tensor-parallel size from 1 to 2 improves throughput by 40.6% at N=1 and 22.8% at N=4, while increasing SLM tensor-parallel size from 1 to 2 brings only 0.8% and 0.4%; the default is LLM TP=2, SLM TP=1.
• Deployment configurations: Non-overlapping GPU deployment improves TokenRouter throughput by 6.43% on average; cross-node deployment over RoCE at 3.7 GB/s yields throughput comparable to single-node with the same resources, translating to 1.93–32.81× improvements over the respective official implementations.
• Source of the baseline overhead: Profiling the SLM-side per-step runtime in Std. Serving over the final 128 steps of a 2,048-token generation shows prefix matching at 7.78 ms (20.94%), locking cache nodes at 6.49 ms (17.47%), SLM inference at 1.56 ms (4.20%), releasing locks at 5.13 ms (13.81%), updating the radix cache at 12.12 ms (32.62%), and others at 4.08 ms (10.98%).
• Pareto frontier: On AMC23 with the Qwen3-0.6B/Qwen3-32B pair, greedy decoding, and a maximum output length of 8,192 tokens, TokenRouter shifts token-level routing to a new throughput–accuracy Pareto frontier, making fine-grained routing more competitive than coarse-grained routing on frameworks such as SGLang.

Methodology in Plain English

The authors start from the observation that token-level routing algorithms share one lifecycle: a request decodes, decides whether to switch models, transfers its state, and resumes decoding at the peer. They turn this into a programming model with three developer-supplied functions: route (pick a destination after each decoding step; 0 means stay local), send (build the peer message with the unseen token suffix and a position anchor), and receive (convert the returned peer request back to local form; the default accepts all transferred tokens).

On the runtime side, the server is split into one autonomous subserver per LLM, each with its own scheduler, runner, private KV-cache pool, and three decoupled loops. A third "inter-model" loop carries peer requests asynchronously so a routed request can leave a batch and rejoin later without stalling the other model. A "pending" state sits between "running" and "finished" so a returning request keeps its serving state and skips prefix matching and KV reallocation.

For scheduling, each subserver waits until a small threshold of same-model requests accumulates before launching a batch, trading a controlled wait for larger batches. The optimal threshold is derived analytically by modeling routing as a Discrete-Time Markov Chain parameterized by concurrency N, the routing probability matrix, and each model's per-step decoding latency, then searching for the threshold that maximizes throughput.

Experiments run on an 8× A100-80G GPU server against two baselines: the algorithms' official released code and a "Std. Serving" SGLang-based baseline with one server per LLM plus an external dispatcher. Two-model algorithms use Qwen3-0.6B and Qwen3-32B; ME uses Qwen3-0.6B, Qwen3-8B, and Qwen3-32B. Workloads are low-effort reasoning (AIME2024 prompts of about 100 tokens, max output 2,048), high-effort reasoning (AIME2024, max output 8,192, keeping only problems whose Qwen3-32B solutions exceed 8,192 tokens), and agentic multi-turn tasks (SWE-Smith trajectories of around 8,192 input tokens, max output 1,024).

Why This Matters

Impact on research: Token-level routing had shown algorithmic benefits that serving systems could not realize efficiently. TokenRouter supplies the systems substrate that makes those algorithms practical, and separates routing policy from serving optimization so new routing algorithms can be tested without rewriting a serving stack.

Real-world applications:

• Cost-aware production LLM serving, where routine tokens run on a small model and only difficult tokens are escalated to a large one.
• Multi-model collaboration within a single response, where models with complementary expertise produce one answer.
• Long-form reasoning and agentic coding assistants, where token-level routing sustains throughput as output lengths grow.
• Serving systems that need a throughput–latency trade-off knob rather than a few discrete routing granularities.

Industry relevance: TokenRouter exposes a single external interface identical to standard single-LLM servers such as vLLM and SGLang, so it can act as a drop-in replacement; internally it preserves continuous batching and prefix caching per model. It scales by launching additional subservers, and it reduces manual tuning through automatic joint KV-cache allocation and automatically derived batching thresholds. The work was supported by the National Natural Science Foundation of China (No. 62325405, 62104128, U19B2019, U21B2031, 61832007, 62204164, 92364201), the Tsinghua EE Xilinx AI Research Fund, BNRist, and Infinigence-AI.

Future Directions

• Relaxing the modeling assumption that the number of tokens between two consecutive sends follows a geometric distribution, which the authors explicitly leave for future work.
• Extending the analysis and scheduling to routing patterns where that assumption fails or where token spans, rather than single tokens, are routed.
• Broadening evaluation beyond the reported model pairs and deployments, including the cross-node configurations already measured at lower throughput than single-node.
• Making the delayed-batching threshold adaptive to workload shifts, beyond the current DTMC-derived B* computed from routing probability, concurrency, and per-step latency.

Target Audience

Serving-system engineers and ML infrastructure researchers who build or deploy multi-model LLM inference; algorithmic researchers developing token-level routing who need an execution platform; and practitioners seeking cost-quality improvements in production LLM serving without changing their client-side protocol.

Authors’ abstract

Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.