Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing

AI summary

Overview

Research area: Large language model (LLM) routing — the problem of deciding, per query, which model from a pool of candidates should answer it. The paper is positioned at the intersection of routing systems, transfer learning, and foundation-model-style pretraining.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, cross-attention, episodic/meta-learning style pretraining, and ranking losses, although the central argument is stated in accessible terms.

Scope in one sentence: The paper proposes RouteFM, a single pretrained router that is frozen at deployment and adapts to new domains, modalities, candidate pools, and context budgets through behavioral context alone, rather than being refit per environment.

What This Paper Is About

Existing LLM routers are trained by what the authors call "local fitting": a router is optimized for one specific query workload and one specific candidate model pool, and must be re-optimized when domains, modalities, or available models change. The authors ask whether routing can instead be treated as a reusable capability, learned once across many heterogeneous routing environments and then deployed unchanged. RouteFM is their answer: a router that infers what each anonymous candidate model can do purely from observed behavior on past queries, and compares candidates relative to the current target query.

Key Contributions

1. A foundation-model framing for LLM routing. The paper reframes routing away from environment-specific local fitting toward a single shared routing function whose parameters are fixed while the query distribution and candidate pool vary.

2. RouteFM, an architecture combining global routing pretraining with in-context capability inference. It builds a compact capability profile per candidate from behavioral observations alone, retrieves target-relevant evidence from both the profile and the raw observation sequence, and compares candidates jointly within the pool — without model identities or persistent model embeddings.

3. Episodic routing pretraining with routing-aware supervision. Training episodes vary task, candidate composition, candidate ordering, and observation budget; supervision combines point-wise regression, pairwise ranking, and a routing-regret objective defined over a soft routing distribution.

4. Evidence that a single frozen router transfers across domains, modalities, and deployment changes. The reported gains are largest in low-context regimes, when behavioral evidence is scarce.

Main Findings

• In-domain routing on RouterEval: RouteFM achieves the highest routing quality across all observation budgets tested (K = 8, 16, 32, 64). Its largest margin is at K = 8, where it outperforms the strongest baseline by 0.84 quality points, with margins shrinking as more behavioral evidence becomes available. RouteFM's values are 0.6192 (K=8), 0.6433 (K=16), 0.6627 (K=32), and 0.6611 (K=64).

• Cross-modal transfer to MMR-Bench: MMR-Bench is excluded from pretraining. RouteFM again achieves the highest routing quality at every budget, outperforming the strongest non-RouteFM baseline by 2.23 quality points at K = 8. The margin narrows to 0.07 points in the large-observation regime. RouteFM's values are 0.7323 (K=8), 0.7457 (K=16), 0.7487 (K=32), 0.7504 (K=64), and 0.7614 (Large).

• Candidate-pool comparison is the most important architectural component. Removing the candidate-pool Transformer causes the largest degradation in the architecture ablation, a 1.33-point drop at K = 8 (0.7190 versus 0.7323), indicating that comparing candidates jointly matters more than estimating them independently.

• Target-to-context attention contributes complementary evidence. Removing it reduces performance consistently, most visibly under limited observations (0.7266 at K=8), showing that direct access to individual behavioral observations adds information beyond the compressed capability profile.

• Query representation matters. Replacing the Qwen3-VL query representation with a BGE text-only encoder degrades performance across budgets (0.7308 at K = 8 up to 0.7491 in the large regime), indicating the representation contributes to cross-modal routing.

• Pretraining does most of the work. Under a support-controlled protocol where the from-scratch and fine-tuned variants receive 20% labeled target-domain support data for parameter optimization, training the same architecture from scratch consistently underperforms frozen RouteFM, with a 1.08-point gap at K = 8. Fine-tuning the pretrained RouteFM on the target domain yields only small changes — marginal gains at some budgets and slightly lower performance at others.

• New models can be incorporated from very little evidence. After adding a newly introduced candidate, RouteFM surpasses all refitted baselines after observing only eight examples from the new candidate. With 16 observations, routing over the expanded pool outperforms routing over the original incumbent pool.

• Adaptation cost shifts from retraining to context construction. RouteFM is applied with frozen parameters using available behavioral context, while learned environment-specific baselines require an additional fitting stage before inference. Simple non-parametric methods remain faster in absolute latency.

• Context-budget efficiency. Under a dense context-budget protocol with each competing router evaluated at its full available context as reference, RouteFM matches the full-context performance of Weighted kNN with only 1% of the available observations, MLP with 3%, and EmbedLLM with 26%. Progressively larger context fractions are required to match the stronger full-context references.

Methodology in Plain English

The core idea. Routing is treated as a target-conditioned ranking problem: given a query, rank the available models by suitability. The capability to extract target-relevant evidence from a model's past behavior and compare models is assumed to be shared across environments, so it is learned once.

Keeping candidates anonymous. RouteFM never sees model names, providers, parameter counts, or persistent model-ID embeddings. Each candidate is described only by behavioral context: triples of (historical query embedding, observed quality, normalized relative cost). Cost is normalized within each episode and used only as a relative indicator. Queries are represented by frozen 4,096-dimensional Qwen3-VL embeddings.

Encoding observations. Query semantics, quality, and cost are encoded jointly rather than treated as query-independent attributes of a model. Quality and cost scalars are embedded separately, then multiplied element-wise with the projected query representation to model their interaction. The result is one behavioral token per historical observation.

Compressing behavior into a capability profile. Because the number of observations varies across candidates, RouteFM compresses the variable-length observation sequence into a fixed set of 24 learnable latent capability tokens (hidden dimension 256). Four profiling blocks alternate cross-attention (tokens retrieve from observations) and self-attention (tokens exchange information). The latent tokens have no predefined meanings such as math or coding; their roles emerge end-to-end from routing supervision. RouteFM keeps both the profile and the original observation sequence.

Comparing against the target query. The target query is projected into the router hidden space and attends to two sources in parallel: the capability profile (how the target aligns with overall capabilities) and the raw observation sequence (fine-grained evidence that compression may lose). The two are combined by residual fusion with a learned scalar, then all candidate representations are processed jointly by a Transformer encoder over the candidate dimension with no positional encoding, making the comparison permutation-equivariant. Masking excludes padded or invalid candidates.

Supervision during pretraining. Each episode is built from a single task with an anonymous candidate set, behavioral contexts, and a disjoint set of target queries; episodes vary task, candidate composition, candidate ordering, and context size. For quality and cost, the loss combines point-wise regression with pairwise ranking. A routing-regret term is added on top of a soft routing distribution over predicted qualities, penalizing the gap between the best available model's quality and the expected quality under that distribution. The overall objective weights quality, cost, and regret terms, with the regret contribution progressively increased over the pretraining curriculum.

Scale and training. Pretraining uses LLMRouterBench, RouterBench, RouterEval, and MixInstruct, containing 212,470 query–task instances and 1.73M valid query–model observations. Episodes contain 6–24 anonymous candidates, 16–32 target queries, and a per-model observation budget K drawn from {8, 16, 32, 64, 128, 256, 512, 1024}. Training runs for 10,000 AdamW steps with effective batch size 16, from random initialization.

Deployment. At inference the router predicts target-specific quality and relative cost for each candidate, and the decision maximizes quality minus a weighted cost term, where the weight controls the quality–cost trade-off. Varying that weight supports different deployment preferences without retraining. RouteFM is frozen throughout evaluation and never uses evaluation labels for parameter updates.

Evaluation. In-domain testing uses held-out RouterEval queries from tasks seen in pretraining, with four anonymous models per pool and K ∈ {8, 16, 32, 64}. Transfer is tested on MMR-Bench using dataset-wise five-fold splits that separate behavioral context from evaluation queries, plus a large-observation regime using 40% of each dataset as context and 60% for evaluation. Baselines include Weighted kNN, MLP, Embedding-Ridge, EmbedLLM, RM-Softmax, TRouter, InferenceDynamics, UniRoute, ICL-Router, and Context-Mean, all reproduced under a unified framework with matched candidate pools, observations, evaluation queries, and query representations. The primary metric is the average realized quality of the selected model.

Why This Matters

Impact on research. The paper argues against the default assumption that each deployment needs its own trained router. It shows that a single frozen model can carry routing capability across domains, modalities, candidate pools, and context sizes, with the largest advantages precisely where supervision is expensive to collect. It also introduces a distinct formulation — routing as episodic pretraining over heterogeneous environments with anonymous candidates — and reports ablations isolating which components carry the transfer.

Real-world applications.

• Serving gateways with changing model rosters. Providers that add, replace, or deprecate models can expand or alter the pool and supply only a handful of behavioral observations for new entrants, without retraining the router.
• Cost- and latency-sensitive deployments. Because the final decision maximizes predicted quality minus a weighted cost term, one frozen router can serve different quality–cost preferences by changing that weight rather than refitting.
• Low-supervision and cold-start settings. The reported advantage being largest at eight observations per candidate matters for organizations that cannot run large labeled evaluation sweeps for every model.
• Cross-modal pipelines. The transfer to a multimodal benchmark excluded from pretraining suggests routers can be reused when the input modality of deployment differs from pretraining.

Industry relevance. The results reframe adaptation as a context-construction problem rather than a retraining problem. The paper does note that simple non-parametric methods remain faster in absolute latency, so the practical trade-off is between inference-time cost and avoiding target-domain parameter optimization.

Future Directions

• How far does cross-modal transfer extend? The paper reports transfer to MMR-Bench, an unseen multimodal benchmark, but does not establish whether the same capability carries to modalities or tasks substantially more distant from the pretraining mixture.

• What is the ceiling on observation efficiency? RouteFM matches Weighted kNN's full-context quality with 1% of observations, MLP with 3%, and EmbedLLM with 26%, but requires progressively larger context fractions to match stronger full-context references. Closing that remaining gap is an open problem the paper identifies only implicitly.

• Can routing be made even cheaper at inference? RouteFM avoids target-domain parameter optimization but is slower in absolute latency than simple non-parametric methods, leaving room for future work on reducing inference overhead while preserving predictive capacity.

• How should behavioral context be selected and maintained? Adaptation is described as primarily a context-construction problem, yet the paper does not characterize which observations are most informative or how context should be curated as pools and workloads evolve.

Target Audience

Researchers and engineers working on LLM serving infrastructure, model selection, and routing systems; practitioners who maintain multi-model deployments and face changing model rosters or limited supervision; and machine learning researchers interested in foundation-model-style pretraining applied to decision problems rather than generation. Readers should be comfortable with transformer attention mechanisms, episodic training, and ranking losses. The introduction and problem formulation are accessible to a broader audience, while the architecture appendix and loss definitions are aimed at readers who intend to reimplement or extend the method.

Authors’ abstract

Large language model (LLM) routing aims to assign each query to the most suitable model from a heterogeneous candidate pool, improving the quality--efficiency trade-off of LLM inference. Existing routers are typically learned through local fitting: a router is optimized for a particular query workload and candidate pool, and often requires additional supervision or retraining as the routing environment changes. We ask whether LLM routing can instead be approached from a foundation-model perspective, learning a reusable routing capability that generalizes across tasks, candidate models, and deployment conditions. To this end, we introduce RouteFM, which learns to characterize anonymous candidate models from behavioral context and infer their target-specific capabilities, rather than binding routing decisions to fixed model identities or a single environment. Through episodic pretraining across heterogeneous routing environments, this capability can be reused by a frozen router and adapted to new environments through context alone. Experiments demonstrate transfer across changes in domains, modalities, candidate pools, and context budgets, with the largest gains when behavioral evidence is limited. On MMR-Bench, which is excluded from pretraining, RouteFM outperforms the strongest baseline by 2.23 quality points with only eight observations per candidate. These results support moving LLM routing from repeated local fitting toward a pretrain once, route anywhere paradigm. Our code is publicly available at https://github.com/LAMDA-Model-Reuse/RouteFM.