SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

AI summary

Overview

• Research area: Large language model (LLM) routing — learning which candidate model should answer a given query — with a focus on interpretable, probabilistic query representations.
• Technical level: Intermediate. The paper assumes familiarity with supervised learning, embeddings, and benchmark metrics, but the core idea (typed questions about a query, answered as probability distributions) is explained through examples.
• Scope: The paper proposes SeLMRoute, a routing framework that separates interpretable semantic evidence extraction, candidate performance learning, and deployment objectives, and evaluates it on LLMRouterBench in both performance-oriented and performance-cost settings.

What This Paper Is About

Most LLM routers decide which model to use from dense query embeddings, model representations, preference data, or clusters of similar examples. These representations capture similarity but rarely state what a query actually requires, and they often discard the router's uncertainty about a query's properties. SeLMRoute's goal is to insert an explicit, inspectable, probabilistic "semantic state" between the raw query and the learned model-selection rule, so routing decisions can be explained and reused across different model pools and deployment objectives.

Key Contributions

1. An explicit probabilistic semantic state for LLM routing: interpretable task requirements are estimated independently of the candidate model pool.
2. A three-way separation of concerns: semantic evidence extraction, model performance learning, and routing objectives are decoupled, allowing the same query representation to support different candidate pools and deployment objectives.
3. Empirical evidence on representation choice: raw probability mass gives the strongest mean performance among the evaluated semantic representations and remains competitive with dense, lexical, and domain-level baselines.
4. An evaluation under grouped splits: SeLMRoute is tested with in-distribution grouped splits, dataset-level and domain-level distribution shift, different routing learners, reduced semantic probe sets, an open-weight decision model, direct decision-model routing, and a separate performance-cost benchmark.

Main Findings

• Main accuracy: On LLMRouterBench (15 datasets, 20 candidate models, 11,481 queries), SeLMRoute with the ProbabilityMass representation achieves an average accuracy of 72.08% ± 0.45 over five 70/30 grouped splits.
• Out-of-fold result: Grouped five-fold out-of-fold evaluation reaches 72.64%, compared with 69.23% for the strongest fixed candidate (Qwen3-8B).
• Probability mass beats other semantic representations: ProbabilityMass (72.08% ± 0.45) outperforms the Full 88-feature semantic representation (71.30% ± 0.18) and the Hard 16-feature representation (71.20% ± 0.59).
• It also beats dense, lexical, and domain baselines: GTE-Qwen2 + CatBoost reaches 71.16% ± 0.74, DomainOnly 71.08% ± 1.03, and TF-IDF 70.87% ± 0.78 under the same protocol.
• Comparison with published routers (external): LLMRouterBench reports EmbedLLM at 71.24, GraphRouter at 70.29, Model-SAT at 71.88, and Avengers at 71.94; the authors treat these as external reference points because the published results use their own split protocol.
• Decision model matters: Replacing JEV with the open-weight Laya model yields 70.53% AvgAcc under the same grouped OOF protocol — still above the best fixed model, but significantly below the JEV-based state.
• Direct routing is worse: An ablation asking JEV to choose a model directly from anonymized candidate profiles performs worse, supporting the architectural separation between semantic evidence extraction and downstream performance learning.
• Cost-aware setting: In a separate setting with 13 flagship models over 10 datasets and 12,446 benchmark instances, SeLMRoute improves performance in all five grouped splits, with a mean PerfGain of 2.66%, but the strict protocol establishes no positive monetary savings.
• Representation size: The default semantic state is 40-dimensional (8 binary Noul probes plus 8 four-level Score probes, 8 + 8 × 4 = 40); the Full representation has 88 features and the Hard representation has 16.

Methodology in Plain English

The pipeline has three stages. First, a typed decision model (JEV is the primary one) answers a fixed set of sixteen human-readable questions about the incoming query — for example, whether correctness requires mathematical calculation, whether external knowledge is needed, how deep the reasoning is, how ambiguous the request is, and how sensitive correctness is to exact details. Each answer is kept as a probability distribution rather than collapsed to a single label, producing a 40-number semantic state. A seventeenth Choice question estimates a coarse task family, but that output is deliberately excluded from the routing representation so the router never sees an explicit "mathematics," "code," "logic," or "knowledge" label.

Second, a lightweight supervised performance model — a multi-output CatBoost regressor — maps that semantic state to predicted scores for each candidate model. Because model identities never appear in the semantic state, the same query encoding can be reused if the candidate pool changes; only the empirical performance mapping needs updating. In the performance-cost setting, where some candidate-query cells are missing, candidate-specific regressors are used instead so each model trains only on its observed outcomes.

Third, a configurable objective turns predictions into a choice: quality-only routing picks the highest predicted score, while a cost-aware variant maximizes (1 − λ) times normalized predicted quality minus λ times normalized predicted cost, with λ selected on an inner validation partition. No test score is used to choose the reported operating point.

Evaluation uses a grouped protocol: instances are grouped by source dataset and normalized query text so repeated queries never appear on both sides of a split — the benchmark contains 11,481 instances corresponding to 11,423 unique groups, with 47 groups containing repeated queries. Dense (GTE-Qwen2), lexical (TF-IDF), and domain-only baselines are run through the same downstream regression procedure.

Why This Matters

For research, this paper reframes routing as a representation question rather than purely an architecture question, and it shows that keeping probability mass — retaining uncertainty — can outperform both a larger feature set and a hard-label version of the same judgments. It also supports the concern raised by LLMRouterBench that some router gains may come from coarse domain structure, since a DomainOnly baseline reaches 71.08% in this evaluation.

Real-world applications:

• Cost-controlled LLM serving: a deployment can route to the cheapest model expected to satisfy a quality threshold, using the same semantic state when prices change.
• Heterogeneous model pools: teams that swap, add, or retire candidate models can reuse existing query encodings instead of re-embedding history.
• Regulated or auditable deployments: the semantic state exposes inspectable evidence (for example, high probability mass on code reasoning, high exactness, low dependence on current information) behind a routing decision.
• Domain-specific assistants: the same representation supports model choice across medicine, finance, code, mathematics, and affective tasks without hand-built per-domain rules.

Industry relevance: routing is a direct lever on inference spend and latency in production LLM systems, and the paper's cost-aware experiment — deliberately honest that it produced no positive monetary savings — signals how hard it is to convert accuracy gains into actual cost reductions.

Future Directions

• Closing the cost gap: no positive monetary savings were established in the strict performance-cost protocol, despite a mean PerfGain of 2.66%, leaving cost-aware routing an open problem.
• Improving semantic extractor quality: JEV-based states significantly outperformed Laya-based states (70.53% AvgAcc grouped OOF), so how much routing accuracy depends on the decision backend remains unresolved.
• Reducing probe sets: the paper includes reduced semantic probe sets as an evaluation axis, raising the question of how few probes are needed before performance degrades.
• Transfer under shift: the paper covers dataset-level and domain-level distribution shift, and the gap to the instance-level oracle (Gap@O of 22.00% ± 0.67 for ProbabilityMass) indicates substantial headroom remains.

Target Audience

This paper is most useful to machine learning systems researchers and engineers working on LLM serving, inference cost optimization, or model selection infrastructure; to practitioners who need routing decisions they can inspect and justify; and to researchers studying interpretable representations, probabilistic belief states, or benchmark methodology for routing. Readers without a background in supervised learning and evaluation protocols will find the experimental sections dense, though the semantic-probe framing is accessible through the worked examples.

Authors’ abstract

Large language model (LLM) routing aims to select the most suitable model for each incoming query. Most existing routers learn this decision directly from query embeddings, model representations, preference data, or clusters of similar examples. Such approaches can be effective, yet the representation used for routing rarely states what a query actually requires. We introduce SeLMRoute, a routing framework that separates the extraction of candidate-independent semantic evidence from the learning of candidate performance and the application of deployment objectives. A decision model first evaluates a set of interpretable questions about the query, such as its reasoning requirements and use of external knowledge, with each judgment retained as a probability distribution. The resulting probabilistic semantic state is used by a lightweight supervised router to estimate candidate model performance. Routing objectives are applied after performance estimation, which allows the same semantic state to support performance-oriented and cost-aware decisions. On the LLMRouterBench (15 datasets, 20 candidate models, 11,481 queries), SeLMRoute achieves an average accuracy of $72.08\% \pm 0.45$, while grouped five-fold out-of-fold evaluation reaches $72.64\%$, compared with $69.23\%$ for the strongest fixed candidate. The representation achieves the highest mean performance among the evaluated semantic, dense, lexical, and domain-level representations. In a separate 13-model performance-cost setting, SeLMRoute improves performance in all five grouped splits, with a mean PerfGain of $2.66\%$. Our code is available at https://github.com/Indigma-Innovations/SeLMRoute.