The Router Within: Eliciting Native Skill Routing from a Frozen LLM

AI summary

Overview

Research area: LLM agents, skill/tool retrieval, and mechanistic read-out from frozen language model internals.

Technical level: Advanced. The paper assumes familiarity with transformer hidden states, contrastive (InfoNCE) training, product-of-experts fusion, and retrieval evaluation metrics, though its central idea is simple to state.

Scope: The paper introduces Gavel, a two-stage skill router that selects which skill an LLM agent should load by reading routing signals directly from the frozen agent model's own intermediate hidden states and predictions, using only 7.9M trained parameters.

What This Paper Is About

LLM agents extend their abilities by loading "skills" — packaged instructions and scripts stored as documents — but picking the right one from a library of thousands is hard. Today's systems either stuff every skill's name and description into the prompt (which crowds the context and limits library size) or hand selection to a separate external retrieval model (which never sees the agent's live rollout and does not improve as the agent does). This paper shows the frozen agent LLM already contains the signal needed to route skills inside its own forward passes, and that two small linear maps are enough to extract it — without placing any skill text in the context.

Key Contributions

1. Gavel, a native skill router. A method that elicits routing decisions from the frozen agent LLM's own forward passes rather than from a prompt-stuffed menu or an external model, so routing accuracy improves automatically as the backbone model improves.

2. A two-stage read-out architecture. A token-level "glance" that scores the entire skill library from one mid-layer hidden state through two trained linear projections, plus a "verdict" that re-examines shortlisted candidates using the model's own generative likelihood and yes/no relevance judgment, fused as a product of experts (three signals, each interpretable as the same log posterior of skill given task).

3. Compressed skill banks with provable error bounds. An installation-time ε-cover procedure (farthest-first traversal) that shrinks each skill's stored key bank roughly 8.5× while guaranteeing every matching score drops by at most ε and none inflates — meaning new skills cost one forward pass and no retraining.

4. SkillTraj, a new benchmark. 372 simulated multi-turn agent trajectories that test routing at the moment a skill becomes needed, under four scenarios: user request (116), tool evidence (106), agent plan (105), and wrong-skill recovery (45), each with noisy multi-turn context.

Main Findings

• Gavel beats far larger external stacks. On Qwen3-32B it outperforms progressive disclosure and retrieve-and-rerank pipelines that add 1.2B to 16B external parameters, by 3.8 points on SkillRet, 13.4 on SRA-Bench, and 1.3–2.7 on Eval-Core, using only 7.9M trained parameters.

• Mid-rollout routing shows the widest gap. On SkillTraj's four scenarios, Gavel leads the strongest alternative by 8.6 to 21.9 points. External retrievers struggle either way they are fed: embedding the whole noisy trajectory mixes every topic, while embedding only the last message drops evidence that matters.

• Trained embedders fail out of domain. SkillRouter's 0.6B embedder and SkillRet-Emb-0.6B match the glance on SkillRet (whose data resembles their training set), but fall below even BM25 when query style shifts to task.md project files (Eval-Core) or skill style shifts to procedural web pages (SRA-Bench).

• The glance alone is a strong retriever. The retrieval stage by itself is the best retrieval stage on SRA-Bench and Eval-Core, and comes within a point of the best trained embedder on SkillRet.

• Native attention keys do not work. Appendix A reports that using the model's own untrained attention keys scores Hit@1 of .001 on a 10,123-skill library versus .918 for the trained glance head — the signal is present but one to two orders of magnitude too faint and miscalibrated. Initializing from native heads also loses to random initialization.

• All three signals are necessary. Removing any one hurts: dropping the glance and committing to the model's judgment alone scrambles ambiguous queries (poorly calibrated self-assessment); reading skill-body likelihood instead of task likelihood drives the likelihood weight α to zero; handing the glance's 20 candidates to a metadata menu falls far behind on SRA-Bench, where the decisive detail lives in the skill body.

• Deployment works end to end. Integrated into mini-swe-agent with a trained trigger gate, Qwen3-32B loads the correct skill on 90.9% of Skill-Use tasks — against 1.1% for the same model under progressive disclosure, and ahead of open frontier models in Codex (MiniMax-M3 .864, GLM-5.1 .706, Qwen3.6-Max .684, DeepSeek-V4-Pro .650) and of retrieve-and-rerank pipelines (.897 and .800).

• Routing scales with the backbone. On SRA-Bench, both the glance and the full pipeline improve as the Qwen3 backbone grows and generations advance. Gavel on a 0.6B model already beats a 32B model under progressive disclosure.

• The read-out layer is chosen without supervision. Matrix-based entropy of hidden states bottoms out roughly 70% of the way through the network (after block 45 of 64 in Qwen3-32B); training at each depth confirms the entropy floor is where routing accuracy peaks.

Methodology in Plain English

The researchers started from the observation that a transformer's middle layers already compress the meaning of the input, but entangle that meaning with next-token prediction noise, so the raw states cannot rank skills on their own.

Their fix has two phases:

Installation (per skill, no training). Each skill document is rendered in a fixed prompt and passed once through the frozen model. Hidden states at a chosen middle layer are projected by a trained key map into a bank of unit vectors. To keep the bank small, a farthest-first traversal keeps only representative keys such that every discarded key is within ε of a retained one — provably limiting scoring error to ε.

Routing (per task). A trained query map projects the task's hidden states at the same layer into queries. Each task token takes its best match against every skill's bank (a max-similarity late-interaction score), and each token votes only for its top-k skills, so a few decisive tokens are not drowned out by the many generic ones. This "glance" ranks the entire library in one sweep.

Verdict. Only the candidates scoring within a small margin of the leader (about nine on average) get a second look. For each, the model resumes the installation pass with the task appended, and reads two things: the mean log-likelihood it assigns to the task's tokens given the skill (a generative signal), and the log-odds of "yes" over "no" when asked whether the skill serves the task (a discriminative signal). These, plus the glance score, are summed as a product of experts — so any one signal can veto a candidate the others tolerate. The top-scoring skill loads.

Crucially, gradients stop at the hidden states, so the backbone stays frozen and only the two linear maps (7.9M parameters, ~0.025% of the 32B model) are ever trained, once, on synthetic query–skill pairs from SkillRet.

Why This Matters

Research impact. The paper reframes skill selection as a read-out problem rather than a retrieval or prompting problem, and gives evidence that agent LLMs carry usable routing information in their intermediate layers that off-the-shelf retrievers cannot access. It supports a broader thesis already visible in probing and embedding research — hidden states contain far more than the next token requires — and extends it to an action-selection task. The ε-cover construction also gives a principled way to keep per-skill indexes bounded as libraries grow.

Real-world applications.

• Enterprise and developer agent platforms that maintain large libraries of skills, plugins, or MCP servers and cannot fit thousands of metadata entries into a context window. This is exactly the regime Codex caps at 2% of context.
• Coding and bash agents that must decide mid-execution whether to invoke a specialized tool or workflow, where the need surfaces from a tool error or a freshly drafted plan rather than a clean user request.
• Long-running autonomous rollouts involving recovery from a wrong skill choice, where a misleading document remains in context and the router must reroute anyway.
• Cost-sensitive deployments, since the approach replaces a 1.2B–16B parameter external stack with 7.9M parameters and reuses forward passes the agent already computes.

Industry relevance. The result that a 32B model with a native router outperforms far larger frontier models running in production harnesses at skill invocation is directly relevant to anyone building agent frameworks. It suggests current harness designs (preloading metadata) may be leaving substantial capability unused, and that routing quality can be improved by upgrading the agent backbone rather than by bolting on a larger retriever.

Future Directions

• Beyond skills. The authors explicitly leave open whether the same read-out can route tools, memories, or MCP servers from an agent's forward passes. Any decision over a large external library is a candidate.
• Learned aggregation. The glance's vote decay is the one hand-crafted component. Trained retrieval tokens were tried but learned recency instead of content-based attention, likely because the training data consists of clean single-turn requests. Training on data that rewards selectivity — long, noisy, mid-rollout contexts — is the obvious next step.
• Closing the remaining calibration gap. The verdict's discriminative signal (V) needed a small weight (γ = 0.025) because self-reported judgments are poorly calibrated. Improving that calibration could raise the ceiling further.
• Portability and scaling. The paper demonstrates transfer to a Gemma backbone and shows accuracy improving with backbone capability, but the exact transfer behavior across model families, longer contexts, and much larger libraries (tens of thousands of skills, which the paper cites as the emerging reality) is not fully characterized.

Target Audience

Researchers and engineers working on LLM agents, tool and skill selection, retrieval-augmented generation, and model interpretability. Practitioners building agent harnesses (coding agents, plugin systems, MCP-based tool platforms) will find the deployment results and cost analysis directly actionable. Readers interested in what frozen transformer hidden states encode — probing, embedding extraction, and representation reuse — will find Appendix A's negative results on native attention keys especially informative. The paper is advanced in technique, but its two-stage "glance then verdict" idea is legible to anyone with a working understanding of how agent context windows and retrieval pipelines behave.

Authors’ abstract

Skills extend an LLM agent beyond its parametric knowledge, and the gain they promise rests on picking the right one. Deployed harnesses route by preloading every skill's metadata into the context, which disperses the agent's attention and caps the library size. Retrieval pipelines move the selection out of the context, but also out of the agent's capability. We show that the frozen agent LLM already carries the routing signal in its own forward passes, and that two linear maps suffice to read it out with no skill text in the context. Gavel (Glance And Verdict from a frozen LLM) reads it in two steps. A glance projects the task's and each skill's mid-layer states through the two maps, the only parameters trained, and scores the full library against compact per-skill banks that one forward pass builds at installation. A verdict then resumes the shortlisted skills' forward passes and reads the model's own likelihood and yes/no judgment, fused with the glance as a product of experts. Trained once, Gavel transfers zero-shot to three public benchmarks and SkillTraj, our new benchmark of 372 simulated agent trajectories. On Qwen3-32B it outperforms progressive disclosure and retrieve-and-rerank pipelines that add 1.2B to 16B external parameters, by up to 13.4 points on written tasks and up to 21.9 when the need for a skill arises mid-rollout. Routing accuracy improves as the backbone does, and in a bash-agent harness the same 32B triggers the correct skill on Skill-Use more often than far larger frontier models running in Codex.