Task-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarios

AI summary

Overview

Research area: Natural language processing — specifically large language model (LLM) routing, dataset synthesis, and cost–performance optimization for model selection.

Technical level: Advanced. The paper combines practical data-synthesis engineering with a probabilistic latent-variable formulation (variational inference / ELBO), though the high-level idea — synthesize training data when none exists — is accessible to intermediate readers.

Scope: The paper proposes a task-profile-guided data synthesis framework and a task-type-aware router (TRouter) to make LLM routing work when no in-domain labeled training data is available.

Publication details: arXiv:2604.09377v1 [cs.CL], 10 Apr 2026. Authors: Hui Liu (City University of Hong Kong), Bin Zou (University of Hong Kong), Kecheng Chen (City University of Hong Kong), Jie Liu (City University of Hong Kong), Wenya Wang (Nanyang Technological University), Haoliang Li (City University of Hong Kong). Code: https://github.com/less-and-less-bugs/ColdStartLLMRouter. License: CC BY 4.0.

What This Paper Is About

LLMs differ widely in both quality and price, so routing systems try to pick the right model for each query according to a user's cost–performance preference. Existing routers, however, assume they have labeled in-domain training data — queries answered and scored by every candidate model — and they degrade badly when that data is missing, which is the normal situation for individual users and early-stage products. This paper introduces a way to synthesize a realistic training set from nothing but domain descriptions, plus a router that uses an inferred task type as a latent variable to predict cost and performance more reliably.

Key Contributions

1. Identifying the cold-start problem in LLM routing. The authors state they are the first to name and target this problem, showing that both classification-based (e.g., RouterDC) and regression-based (e.g., MetricRouter) routers lose robustness under cross-domain deployment compared with a simple rule-based alternative (Adaptive LLM).

2. A multi-level task-profile-guided data synthesis framework. Starting only from manually authored domain descriptions, the framework builds a three-level taxonomy (domain, subcategory, difficulty) with a Task Type Generator, enforces taxonomy quality with a self-critiquing Task Type Quality Evaluator, and generates non-redundant question–answer pairs per difficulty-level task profile with a Question-Answer Pair Generator.

3. TRouter, a task-type-aware router. TRouter extends the regression-based routing paradigm by introducing an unobserved latent task-type variable, decomposing the conditional distribution of a metric (performance or cost) into a task-type posterior given the query and a task-conditioned metric distribution given the model, then training with an ELBO objective whose KL term uses the synthesized task taxonomy as a prior.

4. Validation across multiple LLM pools and evaluation protocols. The framework is tested in both cold-start (trained only on synthetic data) and in-domain settings, under traditional metrics and LLM-as-a-judge judging, with task types drawn from different taxonomy levels.

Main Findings

• Synthetic data alone closes much of the cold-start gap. In the LLM-as-a-judge protocol (Table 4), TRouter trained only on synthetic data (𝒟_syn) reaches a utility sum of 0.6551 in the cold-start setting, above the best non-in-domain baseline (Adaptive LLM at 0.6191) and above several in-domain baselines. In the traditional-metrics protocol (Table 3), the cold-start synthetic variants reach utility sums of 0.5274 (synthesized with gpt-4.1) and 0.5382 (synthesized with gemini-2.5-flash), versus 0.4913 for Adaptive LLM.

• Cross-domain-trained routers degrade severely. RouterDC and MetricRouter, adapted to cold-start via cross-dataset training, score utility sums of 0.4676 and 0.2319 respectively in Table 3, and 0.4777 and 0.4406 in Table 4, with the regression-based MetricRouter being "particularly vulnerable."

• TRouter also wins in-domain. In Table 3, TRouter reaches a utility sum of 0.5925 versus MetricRouter 0.5741, RouterDC 0.5667, C2MAB-V 0.5234, GraphRouter 0.4989, and FrugalGPT 0.4548; the Oracle upper bound is 0.7037. In Table 4, TRouter reaches 0.6733 versus MetricRouter 0.6512, C2MAB-V 0.6471, RouterDC 0.6054, FrugalGPT 0.5589, GraphRouter 0.5028, and Oracle 0.9615.

• In-domain data still helps, but the size of the gain depends on the evaluation protocol. The authors report that in Table 3 the in-domain TRouter improves the utility sum by more than 0.05 relative to its cold-start counterpart, while the gap narrows considerably under LLM-as-a-judge in Table 4.

• Difficulty-level conditioning improves sampling efficiency. With a fixed number of sampling iterations, difficulty-conditioned task profiles produce an almost linear increase in validated QA pairs on the Mathematics and Creative Writing domains, whereas domain- and subcategory-conditioned profiles show slowing gains from accumulating redundancy (Figure 3).

• TRouter is sample-efficient and robust to shot count. Although main experiments use 30 shots, performance with five or fewer shots is already near-optimal across user preferences (Figure 4). MetricRouter is described as unstable, initially decreasing then increasing with more shots, while TRouter remains robust.

• Task regularization matters. Removing the task regularization in Eq. (6) degrades TRouter to MetricRouter, and TRouter consistently outperforms MetricRouter across all shot numbers on the same 𝒟_syn.

• The best taxonomy level for the task variable 𝒯 depends on the setting. In-domain, difficulty-level 𝒯 gives the best results (utility sum 0.5925, versus 0.5809 for subcategory and 0.5641 for domain). In cold-start, domain-level regulation is best (0.5402, versus 0.5294 for subcategory and 0.5274 for the difficulty-level default). The authors attribute this to finer-grained difficulty types introducing interference when priors are not yet calibrated.

• Synthesis scale. With gpt-4.1 the pipeline produced 10 domains, 103 subcategories, 447 difficulty nodes, and 17,880 QA pairs; with gemini-2.5-flash it produced 10 domains, 98 subcategories, 380 difficulty nodes, and 15,200 QA pairs.

Methodology in Plain English

Step 1 — Build a task tree. Starting from six seed domains that expand to ten, an LLM-based Task Type Generator recursively splits tasks into children, writing a short name, a definition, and an example for each. The tree has three levels: domain, subcategory, and difficulty. Prompt constraints cap how many nodes each level can have so the taxonomy does not become too fine-grained.

Step 2 — Clean the tree. Before a set of child task types is accepted, a Task Type Quality Evaluator shuffles the candidates and critiques them against criteria such as minimal redundancy, specificity, and completeness. It iterates until no changes are made across three consecutive assessment rounds.

Step 3 — Generate questions and answers. Each difficulty-level node, plus its parent descriptions, becomes a "task profile." The QA Pair Generator writes 40 QA pairs per profile in batches of 8, using a sentence-transformer to measure semantic similarity; any new pair whose maximum similarity to an existing pair exceeds 0.9 is thrown away as a near-duplicate. This repeats until the target count is reached.

Step 4 — Score the synthetic data. Every candidate model answers the synthetic questions. Cost is the sum of input and output tokens weighted by per-token prices; performance is estimated either by traditional metrics or by an LLM-as-a-judge, instantiated with gpt-4.1-nano for cost efficiency. This produces the router training set 𝒟_syn without any human annotation.

Step 5 — Route with a task variable. Instead of predicting cost and performance directly from query features alone, TRouter assumes each query implies a distribution over task types. A task recognition module encodes the query and all task-type definitions with all-MiniLM-L6-v2 into 256-dimensional vectors, concatenates them, and passes them through an MLP and a temperature-scaled softmax to output the task-type distribution. Separate metric prediction modules then produce task-specific predictions, which are combined by weighting with that distribution. Training minimizes a cross-entropy loss over task labels plus mean-squared-error losses over predicted metrics, which corresponds to maximizing an evidence lower bound (ELBO) whose KL term pulls the predicted distribution toward the synthesized taxonomy prior. At inference, the model maximizing the user's utility function U(m,q) is selected, where U = μ_r · r(m,q) − μ_c · c(m,q) and μ_r + μ_c = 1.

Step 6 — Evaluate under preference scenarios. Three preference settings are used — Cost First (μ_r, μ_c) = (0.2, 0.8), Balanced (0.5, 0.5), and Performance First (0.8, 0.2), deliberately avoiding (1, 0) to prevent degenerately always picking the largest model. Performance and cost are normalized to remove scale discrepancies. Four tasks are used (Alpaca, GSM8K, SQuAD, Multi-News), each with 2000 cases, split 7:1:2 into train, validation, and test. Candidate models include Qwen3 variants from 0.6B to 235B and commercial models including Gemini-2.5-Flash, Gemini-2.5-Flash-Lite, Gemini-2.0-Flash, Gemini-2.0-Flash-Lite-Preview, and Doubao-Seed-1.6-Flash.

Why This Matters

The paper shifts LLM routing from "collect a labeled dataset first" to "describe your domain and let an LLM build the dataset." If that works, routing becomes practical for the many deployments that never had the budget to benchmark every model on their own traffic.

Real-world applications:

• Startup and small-team deployments. A startup running a customer chatbot that prioritizes cost can route cheap queries to small models without first annotating thousands of examples.
• Research and analysis tools. A lab analyzing complex literature, which the paper uses as its example of a performance-first user, can favor high-capability models while keeping the trade-off explicit.
• Enterprise LLM platforms and router providers. Because TRouter is regression-based, the authors note it supports flexible specifications such as batch routing, which matters when a provider must satisfy model quotas or rate limits.
• Early-stage products and individual users. The paper explicitly frames cold-start as the situation of individual users and early-stage products that lack resources to collect labeled training data.

Industry relevance: Cost is expressed in real per-million-token prices (for example, Qwen3-235B-A22B at ¥2.00 input and ¥8.00 output per million tokens, versus Doubao-Seed-1.6-Flash at ¥0.15 and ¥1.50), so the utility framing maps directly onto billing decisions. The paper's practical recommendation is that end-users and router providers adopt aligned evaluation protocols — consistently using LLM-as-a-judge when deploying routers — because misaligned protocols leave utility on the table.

Future Directions

• Reducing the human seed requirement. The framework needs minimal human input to specify candidate domains; the authors propose that providers can seed domains manually or generate them automatically with an LLM, but how far that can be pushed toward zero human intervention remains open.

• Stronger or automated QA validation. The authors acknowledge that synthesized QA pairs are not exhaustively validated and argue task-type-level modeling makes TRouter robust to noise in individual pairs. Whether explicit filtering or cross-consistency checks would further improve routing — or whether they are unnecessary — is untested here.

• Mitigating LLM-as-a-judge bias. The paper uses gpt-4.1-nano-2025-04-14 as the judge while evaluating Qwen and Gemma family models to reduce self-preference bias, and recommends additional cross-checking mechanisms. What those mechanisms should be is not specified.

• Aligning evaluation protocols between training and deployment. The paper shows the cold-start versus in-domain utility gap depends heavily on which protocol scores candidate models (traditional metrics versus LLM-as-a-judge), which raises the question of how to choose or reconcile protocols when the deployment-time judge is unknown.

• Extending the taxonomy design. The interaction between taxonomy depth and routing performance differs between cold-start and in-domain settings in Table 5, suggesting that adaptive or hybrid task-type granularity is an unresolved design question.

Target Audience

Researchers and engineers working on LLM serving, model selection, and inference cost optimization; practitioners building router systems for products without labeled in-domain data; and NLP researchers interested in LLM-based data synthesis or in applying latent-variable models to system-level decisions. Readers without a background in variational inference can follow the framework and experimental results but will need to consult Appendix B (referenced but not included in the provided content) for the full ELBO derivation.

Authors’ abstract

Large language models (LLMs) exhibit substantial variability in performance and computational cost across tasks and queries, motivating routing systems that select models to meet user-specific cost-performance trade-offs. However, existing routers generalize poorly in cold-start scenarios where in-domain training data is unavailable. We address this limitation with a multi-level task-profile-guided data synthesis framework that constructs a hierarchical task taxonomy and produces diverse question-answer pairs to approximate the test-time query distribution. Building on this, we introduce TRouter, a task-type-aware router approach that models query-conditioned cost and performance via latent task-type variables, with prior regularization derived from the synthesized task taxonomy. This design enhances TRouter's routing utility under both cold-start and in-domain settings. Across multiple benchmarks, we show that our synthesis framework alleviates cold-start issues and that TRouter delivers effective LLM routing.