Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection

AI summary

Overview

• Research area: Natural Language Processing; specifically LLM routing (dynamic model selection) combined with synthetic data generation and unsupervised skill estimation.
• Technical level: Advanced. The paper assumes familiarity with LLM routing paradigms, embedding-space clustering, consensus/majority voting, and correlation metrics such as Kendall's tau.
• Scope (one sentence): The paper defines "Routing with Generated Data" (RGD), a setting where routers are trained only on LLM-generated queries and answers rather than ground-truth labels, and proposes CASCAL, a query-only consensus-and-clustering router evaluated on four benchmarks and two pools of six models each (12 models total).

What This Paper Is About

Most LLM routing methods assume access to labeled, in-domain data — either to train a classifier or to validate inferred model skills. In practice, user request distributions are heterogeneous and unknown, so such labels may not exist. This paper removes that assumption: it defines a setting where a generator LLM is prompted with a short natural-language task description to produce training queries (and optionally answers), and routers must learn which model in a pool is best for a given query from that generated data alone. The goal is to find routing methods that stay effective even when the generator is weak and its generated answer labels are unreliable.

Key Contributions

1. Introduces Routing with Generated Data (RGD), a routing setting in which routers are trained exclusively on queries and answers generated from high-level task descriptions by generator LLMs, with no ground-truth in-domain data.
2. Systematically compares routing families under RGD, evaluating query-answer routers (LLMRank, Avengers, and a ground-truth variant of the proposed method) against query-only routers (Smoothie-Train and CASCAL) across four benchmarks and two model pools, one with models over 20B parameters and one with models under 10B.
3. Identifies two properties of effective generators: a generator must correctly answer its own questions, and its questions must induce sufficient performance differentiation across the model pool (measured via Kendall's tau between rankings from generated and validation data).
4. Proposes CASCAL (Consensus-Aware Skill Clustering and Aggregation for LLMs), a query-only router that estimates correctness through confidence-weighted consensus voting and finds model-specific skill niches via hierarchical clustering, plus two variants: CASCAL (Top-1) and CASCAL-GT, which uses ground-truth labels instead of consensus.

Main Findings

• Query-answer routers degrade faster than query-only routers as generator quality drops. On Pool-Large, moving from validation data to Exaone-3.5-7.8B-Instruct-generated data, LLMRank drops 8.5% (65.6% to 57.1%), CASCAL-GT Top-1 drops 9.2% (65.7% to 56.5%), and Avengers drops 4.5% (63.4% to 58.9%), while CASCAL drops only 2.5% (63.6% to 61.1%) and Smoothie-Train Top-3 stays essentially flat (59.3% to 59.4%).

• CASCAL outperforms the best query-answer router by 4.6% absolute accuracy when trained on weak generator data (as stated in the abstract).

• CASCAL beats Smoothie-Train in every RGD scenario and both pools. On Pool-Large, CASCAL reaches 63.1%, 61.1%, and 61.1% under the Gemini, Qwen, and Exaone generators, versus Smoothie-Train Top-3 at 59.9%, 59.2%, and 59.4%. On Pool-Small, CASCAL beats Smoothie-Train Top-3 with 46.5% vs 46.1% (Gemini), 45.6% vs 45.0% (Qwen), and 45.3% vs 43.4% (Exaone).

• Weak generators can push query-answer routers below random baselines. With Exaone-generated data on Pool-Small, Avengers reaches 40.7% and LLMRank 41.0%, at or below the Random-3 Vote baseline of 41.6%, while CASCAL reaches 45.3%. On Pool-Large, CASCAL is the only method to consistently match or exceed Random-3 Vote across all RGD scenarios.

• Stronger generators usually produce better routers, but generated data rarely beats real data. Weak generators never outperform stronger ones. A rare exception appears for query-only methods trained on Gemini-generated data: in Pool-Small, CASCAL Top-1 improves from 46.6% to 47.7% (+1.1%) and Smoothie-Train Top-1 from 43.4% to 44.4% (+1.0%).

• Weak generators produce unreliable answer labels. Measured by agreement with Gemini-3-Flash answers, Exaone-3.5-7.8B-Instruct agrees only 65.6% of the time on MMLU-Pro and 75.4% on MedMCQA, while Qwen3-32B agrees 75.1% and 79.0%. Gemini-3-Flash itself has validation accuracy of 86% on MMLU-Pro and 97% on MedMCQA. SuperGPQA and BBEH were excluded from this analysis because Gemini-3-Flash could not reliably answer those queries.

• Ranking quality collapses for large model pools under weak generators. Kendall's tau between rankings from generated and validation data averages 0.30 (Gemini), 0.03 (Qwen, near random), and −0.14 (Exaone) on Pool-Large, with all generators producing negative correlations on BBEH (ranging from −0.20 to −0.89). On Pool-Small, all generators retain strong correlation (average tau above 0.7), which the authors use to explain why CASCAL degrades less on the smaller pool. The top two rank positions remain relatively stable even in low-quality data, allowing routers to still beat random baselines.

• Filtering generated data recovers much of the lost performance. Filtering 20k Exaone-generated queries down to about 3k MMLU-Pro, 1.8k SuperGPQA, 2.5k MedMCQA, and 4.2k BBEH queries lifts CASCAL on Pool-Large from 57.7% (Top-1) and 61.1% (Top-3) with 5k unfiltered queries to 59.6% and 62.3%, respectively. Filtered data beats unfiltered data by 3.5% on SuperGPQA (Top-3) and 2.0% on MedMCQA (Top-3). Validation-data CASCAL reaches 62.2% (Top-1) and 63.6% (Top-3).

• More generated data is not automatically better. Doubling Exaone-generated queries from 5k to 20k moves Top-1 from 57.7% to 58.9% and Top-3 from 61.1% to 61.2% on Pool-Large, a much smaller gain than filtering.

• Asymmetric per-pool degradation is more severe on the small pool. From validation to Exaone data on Pool-Small, Avengers drops 9.5% (50.2% to 40.7%), CASCAL-GT Top-1 drops 10.5% (49.7% to 39.2%), and LLMRank drops 7.8% (48.8% to 41.0%), while CASCAL drops only 1.1% (46.4% to 45.3%).

Methodology in Plain English

The setup. The authors take four benchmarks — MMLU-Pro, MedMCQA, SuperGPQA, and BigBench-Extra-Hard (BBEH) — and split each into 6:4 train-test partitions stratified by task, where each MMLU-Pro, MedMCQA, and SuperGPQA task corresponds to a subject. BBEH tasks that cannot be fit into a discrete output format are removed, because consensus and aggregation require a discrete set of answer classes.

The model pools. Two disjoint pools of six models each. Pool-Large: GPT-OSS 120B, LLaMA-3.3 70B, Qwen-3 32B, GLM-4 32B, Exaone-4 32B, and Gemma-3 27B. Pool-Small: Gemma-2 9B, GLM-4 9B, Yi-1.5 9B, Qwen-3 8B, Exaone-3.5 7.8B, and DeepSeek-Math 7B. Query embeddings come from Qwen3-Embedding-8B.

Generating the data. Instead of using real training questions, a descriptor model (Gemini-2.5-Flash, given 5 in-context queries) writes a short natural-language description of each task. Each generator model is then prompted from that description to produce 5,000 query-answer pairs per domain. Four scenarios are compared: real data, Gemini-2.5-Flash, Qwen3-32B, and Exaone-3.5-7.8B-Instruct — one frontier model, one stronger open-weight model, and one small 8B model. The authors manually check that the descriptions do not leak validation examples.

The baselines. Learning-free baselines are Top-1, Top-3 Vote, Random-1, and Random-3 Vote; these use real validation data and thus set a ceiling. Query-answer routers are LLMRank (a neural ranker over interpretable prompt features) and Avengers (embeds queries, clusters them with k-means at k=64, scores each model's accuracy per cluster). The query-only baseline is Smoothie-Train, which models embedding differences between LLM outputs as a multivariate Gaussian to derive per-model quality scores.

How CASCAL works. CASCAL never uses generated answer labels for training.

1. Consensus scoring: each model's log-probabilities are z-normalized using per-model means and standard deviations from the training split. A model's score on a query is the sum of the normalized confidences of all models that produced the same answer, so answers backed by many confident models score higher. A softmax converts scores into probabilities.
2. Skill clustering: for each model and task, the queries where that model agrees with the majority answer are collected, embedded, and clustered with k-means. The number of clusters K is chosen from {2, 3, 4, 5} by silhouette score, falling back to a single centroid if no K exceeds a threshold of 0.05. Centroids within cosine distance 0.15 of each other are merged (weighted by seed count), and centroids whose top-3 model rankings have Jaccard similarity of at least 0.95 are pruned.
3. Ranking: every training query is assigned to its nearest centroid, and models are ranked within each cluster by average consensus score.
4. Inference: a test query is assigned to a task by nearest embedding, then to a cluster by cosine similarity, then the top-3 models for that cluster are selected and their outputs aggregated by consensus score (which behaves almost identically to majority voting; they diverge only when all three models give different answers, in which case the most confident answer wins).

Variants and filtering. CASCAL (Top-1) skips aggregation and routes to a single model (K=1). CASCAL-GT replaces consensus scores with ground-truth labels, making it a query-answer router. For the filtering experiment, the authors generate 20k Exaone queries, designate the two highest-scoring models by average consensus score as "top-2," and retain only queries where (1) both top-2 models align with the pool's majority answer and (2) at most two other models share that majority, so the query is hard enough that weaker models diverge.

Why This Matters

Impact on research. The paper removes a widely held assumption in the routing literature — that labeled in-domain data is available — and shows that the two dominant paradigms (query-answer and query-only) fail in very different ways when that assumption is dropped. It also supplies a diagnostic framework for evaluating generators: agreement with a strong reference model measures answer quality, and Kendall's tau against validation rankings measures whether generated queries differentiate models. The finding that weak LLMs are better at asking routing-relevant questions than at answering them reframes what a "good generator" needs to be.

Real-world applications.

• Multi-model API gateways and model-serving platforms that must decide, per request, which of several hosted models to call without a labeled dataset for the customer's traffic.
• Cost-constrained deployments where only a small or weak model is available to produce synthetic calibration data, and where query-answer routing would be driven below random by noisy labels.
• Enterprise or regulated domains (for example clinical or financial question answering, given the MedMCQA setting) where the real request distribution is unknown or private and cannot be labeled.
• Cold-start deployments of a new model pool, where a task description plus a generator is faster to obtain than a labeled validation set.

Industry relevance. The paper is co-authored with Capital One and evaluates on heterogeneous pools of open-weight models from multiple vendors, which mirrors the practical situation of an organization choosing among many available checkpoints. The concrete finding that CASCAL is the only method on Pool-Large to consistently match or exceed Random-3 Vote, and that it holds a substantial margin above Random-3 Vote on Pool-Small with weak generator data (45.3% vs 41.6%), speaks directly to budget-constrained routing where weak generators are the only practical option.

Future Directions

• Generator training for routing quality. The authors point toward optimizing data generation for routing quality, for example via reinforcement learning approaches that reward query differentiation rather than answer accuracy.
• Better differentiation for large model pools. On Pool-Large, Kendall's tau between generated and validation rankings falls as low as −0.14, and all generators produce negative correlations on BBEH. Finding queries hard enough to separate strong models from each other remains open.
• Extending consensus-based routing beyond discrete answers. CASCAL requires tasks with a discrete set of output classes for consensus and aggregation, which forced removal of some BBEH tasks; open-ended generation is not covered.
• Improving the filter itself. Filtering recovers performance on Pool-Large (62.3% Top-3 versus 61.1% unfiltered) but does not exceed validation data overall, so the design of filtering criteria and the trade-off between filter strictness and retained data volume are unresolved.

Target Audience

Researchers and engineers working on LLM routing, model ensembling, and multi-agent selection will get the most from this paper, particularly those dealing with heterogeneous or unknown request distributions and no labeled in-domain data. It is also relevant to practitioners building model-serving infrastructure that must pick among many open-weight checkpoints, and to researchers interested in synthetic data generation, weak supervision, and confidence-weighted consensus methods. Readers without background in routing, embedding-space clustering, or agreement metrics will need to consult the related-work section first.

Authors’ abstract

Large Language Model (LLM) routers dynamically select optimal models for given inputs. Existing approaches typically assume access to ground-truth labeled data, which is often unavailable in practice, especially when user request distributions are heterogeneous and unknown. We introduce Routing with Generated Data (RGD), a challenging setting in which routers are trained exclusively on generated queries and answers produced from high-level task descriptions by generator LLMs. We evaluate query-answer routers (using both queries and labels) and query-only routers across four diverse benchmarks and 12 models, finding that query-answer routers degrade faster than query-only routers as generator quality decreases. Our analysis reveals two crucial characteristics of effective generators: they must accurately respond to their own questions, and their questions must produce sufficient performance differentiation among the model pool. We then show how filtering for these characteristics can improve the quality of generated data. We further propose CASCAL, a novel query-only router that estimates model correctness through consensus voting and identifies model-specific skill niches via hierarchical clustering. CASCAL is substantially more robust to generator quality, outperforming the best query-answer router by 4.6% absolute accuracy when trained on weak generator data.