RouteWeaver: Weaving Mode Selection and Execution into Unified LLM Routing

AI summary

Overview

Research area: Natural Language Processing — specifically LLM routing, the subfield that decides which language model(s) to call for a given query, combined with reinforcement learning for LLM-based decision policies.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for LLMs (GRPO, PPO-style clipped objectives, KL regularization, advantage estimation) and with the three routing paradigms it unifies.

Scope: The paper proposes RouteWeaver, a single routing policy that jointly decides how model calls are organized (single-round, multi-round, or agentic) and which Worker models to call, trained with a new COMET-GRPO objective plus a progressive exploration schedule, and evaluated on nine benchmarks across five domains.

What This Paper Is About

Existing LLM routers adapt which models they call, but the execution paradigm itself — one-shot selection, sequential multi-round calls, or multi-agent workflows with roles and dependencies — is usually fixed in advance, so the router can never decide that a different style of computation suits a different query. RouteWeaver makes the execution mode itself a routing decision, learning in one policy both when to use single-round, multi-round, or agentic execution and how to select and coordinate models inside whichever mode it picked. The goal is a router that adapts the organization of model calls, not just the model identities.

Key Contributions

1. A unified formulation of routing across execution paradigms. The authors cast single-round, multi-round, and agentic routing as mode-conditioned trajectories of one policy, expressed through a shared action grammar (a <mode> element, <route> elements, environment-inserted <observation> elements, and a final <answer>), so a single policy can switch between paradigms per query instead of committing to one schema.

2. COMET-GRPO (COordinated Mode–Execution Training). A GRPO variant that splits the trajectory-level advantage into two signals — an across-mode advantage comparing the selected mode's average reward against the overall rollout-group average, and a within-mode advantage comparing the individual trajectory against other trajectories under the same mode — and applies each only to the tokens it should supervise (mode tokens vs. Worker choices, roles, references, sub-queries, and the final answer). The two losses are normalized separately so the short mode-decision segment is not down-weighted merely by length.

3. Progressive exploration. A three-phase training schedule that forces externally assigned modes early, then linearly hands mode selection to the router, so the policy first learns how to execute each mode before deciding when to use it; a temporary one-sided support penalty keeps probability mass on every mode during the transition.

4. An efficiency knob inside the same policy. By mixing a task reward with a cost-based efficiency reward via a weight α ∈ {0, 0.1, 0.3, 0.5}, one trained framework yields multiple accuracy–cost operating points.

Main Findings

• Best average accuracy across five domains: RouteWeaver reaches 61.4% average accuracy, an 11.03% relative improvement over the strongest baseline (Router-R1 at 55.3%), and leads every baseline in each of the five domains.

• Domain-by-domain leadership: Math 74.6 (+12.52%), Code 58.2 (+10.23%), Knowledge 55.0 (+12.47%), Reason 76.7 (+10.84%), Recall 42.5 (+5.20%) relative to the strongest baseline in each column. Relative improvements over the strongest single-round, multi-round, and agentic baselines are 26.08%, 11.03%, and 31.20% respectively.

• No paradigm wins everywhere: Among baselines, multi-round routers are strongest on Math, Code, and Knowledge, while single-round routers are strongest on Reason and Recall — the paper's motivation for query-adaptive mode selection.

• Cost–accuracy trade-offs from one policy: Relative to α = 0, α = 0.1 lowers Worker inference cost by 23% for a 1.0-point accuracy drop; α = 0.3 lowers cost by 61% for 3.1 points; α = 0.5 lowers cost by 86% for an 11.2-point drop.

• Generalization to held-out benchmarks: On AIME, GPQA-Diamond, and BBH, RouteWeaver averages 53.7%, an 18.28% relative improvement over Router-R1 (45.4%), with gains of 11.4, 7.1, and 6.4 points respectively. Agentic execution accounts for 75% of AIME queries but only 12% of BBH queries.

• COMET-GRPO ablation: Replacing it with ordinary GRPO drops average accuracy from 61.4% to 58.8% and shifts mode usage from .35/.41/.24 to .13/.82/.05 (single/multi/agentic), concentrating the policy on multi-round; the largest drop is on Knowledge (5.8 points).

• Progressive exploration ablation: Removing the forced-to-free schedule drops average accuracy to 55.6%, with large decreases on Code (9.6 points) and Reason (8.4 points), and the policy collapses entirely to single-round execution (1.00/.00/.00). Both ablated variants still exceed the strongest fixed-paradigm baseline (55.3%).

• Task-dependent routing behavior (α = 0): Math uses 19% single-round, 50% multi-round, 31% agentic; Code is 84% multi-round; Knowledge is 72% single-round; Reason routes 99% of queries to agentic execution; Recall routes 96% to single-round.

Methodology in Plain English

The router is a single language model (Qwen3-4B-Instruct-2507, fully fine-tuned) that talks to a pool of six frozen Worker models it only knows by anonymous IDs with short capability descriptions — it never sees model names or prices. For each query it first emits a mode tag, then issues routing actions written in a fixed tag grammar: in single-round mode exactly one Worker call; in multi-round mode two to four sequential calls where each later call also sees the previous output; in agentic mode calls that additionally name a role (planner, executor, verifier, summarizer) and reference earlier call IDs, with up to eight Worker calls and ten router action turns. Returned outputs are pasted back into the router's context until it writes a final answer.

Training uses reinforcement learning with two rewards: a task reward (correct / incorrect / invalid trajectory scored as 1, 0, or −1) and an efficiency reward based on normalized Worker inference cost. The key training trick is credit assignment. In standard GRPO every token of a rollout gets the same advantage, so a bad outcome is blamed equally on the mode choice and on the execution. COMET-GRPO instead computes one advantage from how the chosen mode's average compares to the query's overall rollout average, and another from how this trajectory compares to others in the same mode, then applies the first only to the mode tokens and the second only to the execution tokens.

The second trick is a curriculum. If the router picked modes freely from the start, it could lock onto one mode before learning the others. So early training forces an assigned mode on every query (teaching execution within each mode), then linearly reduces the forcing probability while adding a penalty that keeps every mode's average probability above a threshold, and finally releases mode selection entirely to the router.

Evaluation covers nine benchmarks over math (MATH, Omni-MATH), code (APPS, LiveCodeBench, TACO), knowledge (SuperGPQA), reasoning (SATBench), and factual recall (TriviaQA, SimpleQA-Verified), against ten baselines spanning all three paradigms, plus held-out tests on AIME, GPQA-D, and BBH. The router is trained on 3,200 queries from seven datasets with fixed per-batch domain quotas (32 queries per batch).

Why This Matters

Impact on research. The paper reframes routing as a two-level decision problem — paradigm and model — rather than a model-selection problem under a fixed pipeline. It also contributes a general credit-assignment recipe (split advantages by decision level, normalize each loss separately) that applies to any hierarchical policy where a short, high-leverage decision precedes a long execution. The ablation showing that removing progressive exploration collapses the policy to a single mode is a concrete warning about mode collapse in multi-paradigm RL training.

Real-world applications:

• Cost-aware production assistants: a single deployed router that spends one cheap call on easy factual questions but escalates to multi-round or role-structured agentic workflows for hard math or reasoning queries, with α tuning accuracy against spend.
• Model-marketplace gateways: serving layers that front heterogeneous vendor models and must pick both a model and an interaction pattern per request.
• Coding assistants: the paper's routing analysis shows code queries overwhelmingly route to multi-round execution (84%), suggesting where to invest orchestration effort in developer tools.
• High-stakes verification workflows: queries where agentic planner/verifier/summarizer structures are chosen automatically, as with the 99% agentic share observed on the reasoning benchmark.

Industry relevance. The efficiency results are directly deployable: at α = 0.1 the system cuts Worker inference cost 23% for a 1.0-point accuracy loss, and at α = 0.3 it cuts 61% for 3.1 points — meaningful at fleet scale. Keeping the Worker pool frozen and training only a 4B router means new models can be added to the pool without retraining the Workers, and the anonymous-ID interface means the router is not tied to specific vendor pricing.

Future Directions

• Richer execution structures: the authors explicitly name extending the unified perspective to richer execution structures and tool-augmented environments as future work — e.g., modes involving retrieval or code execution rather than only Worker calls.
• Finer-grained credit assignment: COMET-GRPO splits credit at two levels (mode vs. within-mode); whether step-level or role-level decomposition inside agentic workflows would help further is left open, and the paper positions its approach relative to episode- and step-level advantage methods.
• Mode-set design and scaling: the policy chooses among exactly three modes with fixed grammar constraints (at most eight Worker calls, ten action turns); how the framework behaves with additional modes or larger call budgets is not reported.
• Generalization beyond matched domains: held-out gains are demonstrated on AIME, GPQA-D, and BBH, which the paper describes as unseen benchmarks within the same task domains; transfer to genuinely new domains and to safety-critical deployment (where errors may propagate across multiple model calls, as the ethics statement notes) remains open.

Target Audience

Researchers and graduate students working on LLM routing, model cascades, and multi-agent orchestration; reinforcement-learning practitioners interested in hierarchical credit assignment and exploration curricula for LLM policies; and applied engineers building model-gateway or serving infrastructure who need to trade accuracy against inference cost across a heterogeneous pool of models. Readers without RL background will find Sections 1, 5, and the routing-behavior analysis accessible, while Sections 3–4 require comfort with policy-gradient objectives.

Authors’ abstract

LLM routing dynamically selects and coordinates heterogeneous language models across diverse tasks. However, existing approaches are largely built around a predefined execution paradigm, such as single-round, multi-round, or agentic routing, which fixes how model calls are organized and limits adaptation to different query needs. We propose RouteWeaver, a unified routing framework that integrates these routing modes within a single LLM policy, jointly learning which mode to use and how to select models within that mode. To jointly optimize mode selection and within-mode execution, we develop COMET-GRPO (COordinated Mode-Execution Training), which decouples the trajectory-level advantage into separate signals for mode selection and within-mode execution, enabling more targeted credit assignment to each decision level. We further introduce progressive exploration, which gradually shifts training from externally assigned modes to autonomous selection, so that the router first learns how to execute each mode before deciding when to use it. We evaluate RouteWeaver across mathematics, coding, knowledge, logical reasoning, and factual recall. RouteWeaver achieves an average accuracy of 61.4%, an 11.03% relative improvement over the strongest baseline, and outperforms all compared baselines in each of the five domains. Ablations show that both COMET-GRPO and progressive exploration are important for adaptive routing; without progressive exploration, the routing policy collapses entirely to single-round execution.