CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents

AI summary

Overview

• Research area: Multi-agent LLM systems — specifically Mixture-of-Agents (MoA) orchestration, query routing, and reinforcement-learning post-training of LLM agents.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning objectives (clipped surrogate losses, KL penalties, advantage normalization), Mixture-of-Experts routing, LoRA adapters, and hidden-state representation analysis.
• Scope (one sentence): The paper proposes and empirically evaluates CERA-MoA, an iterative reinforcement-learning framework in which a query router and a population of continually trained LLM agents update each other in a closed loop, with the router scoring agent competence from frozen mid-layer hidden states and activating a variable, minimal subset of agents per query.
• Paper details: arXiv:2609.18779v1 [cs.AI], 16 Sep 2026. Authors: Jiaxuan Jiang, Liyuan He, Zhixuan Fang. Affiliations listed: IIIS, Tsinghua University (Beijing, China); School of Artificial Intelligence, Shanghai Jiao Tong University (Shanghai, China); Shanghai Qi Zhi Institute (Shanghai, China). License: CC BY 4.0.

What This Paper Is About

Mixture-of-Agents systems typically treat two things separately: deciding which agent should answer a query (routing), and training the agents themselves. That separation means routing strategies cannot track an agent's changing abilities during post-training, and agents receive manually partitioned or uniformly distributed training data rather than data matched to their growing strengths, so they stay generalists instead of becoming specialists.

CERA-MoA is the authors' attempt to close that loop: routing decisions determine which training queries each agent receives, and each agent's measured performance in turn updates the router's estimate of that agent's competence. The goal is a system that both improves task performance and induces differentiated specialization across agents.

Key Contributions

1. A sample-level co-evolutionary MoA framework. CERA-MoA runs continual reinforcement learning for LLM agents through adaptive query routing, where the routing mechanism adapts to shifting agent capabilities while simultaneously allocating training queries to encourage distinct problem-solving specialization.
2. A predictive familiarity estimator. Instead of using external evaluators or full rollout generations, the estimator reads intermediate hidden states from the router's frozen backbone and converts them into a competence score, requiring no backbone updates — only a lightweight predictor head is trained per agent.
3. Cumulative-threshold adaptive routing. Rather than a fixed top-k allocation, the router activates the smallest score-ranked subset of agents whose cumulative familiarity score meets a threshold τ, trading task performance against computational cost per query.
4. Empirical comparison against static routing and fixed-workflow fine-tuning. Experiments span four domains across three homogeneous base models plus a heterogeneous 3-model pool, on both in-distribution and out-of-distribution benchmarks.

Main Findings

• Highest in-distribution averages across base models (Table 1). On Qwen3-4B, CERA-MoA reaches an average of 63.2 versus 49.6 for the untuned Qwen3-4B baseline and 61.0 for the strongest listed fixed-workflow baseline AT-GRPO. On Llama-3.2-3B-Instruct the average is 47.9 (base model 33.6); on Phi-4-mini-Instruct it is 55.6 (base model 43.9); on the heterogeneous pool it is 63.0.
• Large per-domain gains on reasoning-heavy benchmarks. For Qwen3-4B, reported scores include MATH 79.6 (base 71.5), DAPO-M 41.8 (base 25.6), BBH 83.4 (base 66.8), and RLVR-IF 72.4 (base 59.6). GSM8k is 93.5 and MAGPIE is 92.8.
• Best out-of-distribution transfer (Table 2). CERA-MoA records the highest OOD average in each configuration: 72.8 on Qwen3-4B (AT-GRPO 71.5), 53.9 on Llama-3.2-3B-Instruct, 62.5 on Phi-4-mini-Instruct, and 71.8 on the heterogeneous pool (AT-GRPO 69.1).
• Heterogeneous pools work without forcing consensus. The paper argues fixed workflows force architecturally diverse models into an unnatural consensus that dilutes pre-training strengths, whereas familiarity-based routing matches each query to the most suitable backbone. Gains are reported as largest on the hardest reasoning sets (DAPO-MATH, LogicBench, OlympiadBench).
• The familiarity estimator outperforms alternative routing metrics (RQ3, Table 3 on Qwen3-4B). Direct reward regression scores 60.8 ID / 70.1 OOD and reward-driven multi-class classification scores 61.3 ID / 70.4 OOD, both below CERA-MoA's 63.2 ID / 72.8 OOD. Figure 2 attributes this to estimated rewards oscillating during training while familiarity scores evolve smoothly, because familiarity is expressed as a geometric push-and-pull distance between two network heads over a fixed feature space.
• Adaptive routing beats fixed top-k on efficiency (RQ4, Table 3). Fixed Top-2 gives 63.3 ID / 72.5 OOD at 666.37 average tokens; fixed Top-1 gives 61.4 ID / 70.1 OOD at 325.70 tokens; CERA-MoA's adaptive threshold gives 63.2 ID / 72.8 OOD at 367.77 tokens — roughly a 45% reduction in average generation tokens relative to fixed Top-2 while retaining comparable performance.
• Specialization emerges without human role assignment (RQ5, Figure 3). Among four Qwen3-4B agents, Agent 2 was allocated the majority of queries as a primary solver due to long chain-of-thought reasoning; Agent 1 specialized in concise math and logic, Agent 3 in coding, and Agent 4 in general reasoning. A t-SNE visualization of intermediate hidden states shows queries routed to the same agent forming tightly clustered semantic neighborhoods.
• Hyperparameter values are not reported in the available content. The paper names a temperature hyperparameter λ, a cumulative threshold τ, exploration coefficient c_ucb, entropy weight w_ent, separation margin m, KL scaling β, clipping range ε, the number of sampled completions G per query, and the agent population sizes N = 4 (homogeneous) and N = 3 (heterogeneous), but does not give numeric values for λ, τ, c_ucb, w_ent, m, β, ε, or G, nor training-set sizes or compute budgets.

Methodology in Plain English

The system is a loop with three parts: a router, a pool of agents, and a voting-based aggregator.

Scoring competence without generating text. For each incoming query, the router runs the prompt through its own frozen backbone and grabs the hidden states from two middle layers — the floor of L/2 and the floor of 3L/4 — concatenating them into a semantic representation of size 2H for a backbone of hidden dimension H and depth L. The reasoning given is that intermediate hidden states capture semantic information complementary to final-layer states, which are tied more directly to next-token prediction. Each agent has two small MLP heads over this representation (a layer-norm, a Linear(2H, 512), a SiLU, then a Linear(512,256)): a trainable predictor head g_i and a permanently frozen target head ḡ_i, with the target heads initialized orthogonally across agents so they provide a stable yet diverse reference space. The normalized Euclidean distance between the two projections gives d_i(q), and the familiarity score is f_i(q) = exp(−λ d_i(q)), so a smaller distance means higher familiarity.

How the router learns. The frozen target head is not a semantic prototype or competence label; it is just a fixed anchor. After agents attempt queries, each agent's mean reward is compared against the average reward of the other participating agents to give a relative advantage A_i,q, with the special case that a solo-activated agent is assigned A_i,q = r_i,q. Positive advantage pulls the predictor head toward the anchor (shrinking distance and raising familiarity); negative advantage pushes it away by a margin m. Supervision is therefore relative rather than absolute, which the authors argue is more stable under continually shifting agent behavior.

How queries are allocated. During training, agents are ranked by a routing score that adds a UCB-style exploration bonus based on how many samples the agent has already received and a term weighted by the agent's historical mean generation entropy. The paper is explicit that this score governs prioritization only — the cutoff for activation is evaluated strictly against familiarity scores. The router activates the smallest top-ranked prefix of agents whose cumulative familiarity reaches τ, falling back to the entire population if the total falls short. At inference, the exploration terms are switched off (c_ucb = w_ent = 0).

How agents learn. Selected agents store allocated queries in a last-in-first-out buffer, then sample G completions per query and update using Dynamic Sampling Policy Optimization (DAPO) combined with sequence-level importance sampling from Group Sequence Policy Optimization (GSPO). Advantages are normalized by the empirical mean and standard deviation of the G rewards with a 10⁻⁴ term in the denominator; the importance ratio is a length-normalized geometric mean over tokens; a clipped surrogate and a token-level KL penalty against a reference policy complete the loss, normalized by the number of active completion tokens.

Deployment options. The framework is agnostic to how agents are parameterized. In the homogeneous setting, four agents are independent LoRA adapters over one shared backbone, giving a 4-agent MoA on a 4B parameter base model with an aggregate footprint of roughly 4.4B parameters. In the heterogeneous setting, three distinct open-source models are used directly: Qwen3-4B, Llama-3.2-3B-Instruct, and Phi-4-mini-Instruct.

Aggregation at inference. For tasks with deterministic solutions, the aggregator performs familiarity-weighted majority voting, selecting the answer from the agent with the highest familiarity score within the winning consensus group. For open-ended tasks it outputs the completion of the most familiar agent.

Evaluation. Domains and their datasets: mathematical reasoning (GSM8k, Hendrycks MATH, DAPO-MATH-17k), code generation (MBPP, Eurus-2-Code, TACO), instruction following (MAGPIE-IF, RLVR-IFEval), and general reasoning (BIG-bench Hard). OOD generalization is tested on IFEval, HumanEval, AGIEval, ARC-c, LogicBench, and OlympiadBench with no additional fine-tuning. Baselines split into orchestration optimization (ICL-Router, LinUCB, RouteMoA) and fixed-workflow agent fine-tuning (GSPO as a single-agent RL baseline, MAPoRL, AT-GRPO). For orchestration baselines, four experts are simulated by applying role-specific prompts to the base model. Generation hyperparameters and task-specific reward functions are held identical across all baselines and runs.

Why This Matters

The paper targets a concrete structural gap: capability-aware routing and agent post-training are usually designed in isolation, so neither can exploit the other. Demonstrating that the two can be optimized together — and that doing so produces measurable specialization rather than just higher average scores — gives the multi-agent literature a template for treating routing as part of the training loop rather than as a fixed wrapper around trained models. The efficiency argument is also notable: because competence is estimated from hidden states rather than from full rollouts or an external judge, routing decisions avoid the per-query cost of generating candidate answers just to compare them, and the cumulative threshold allows the number of activated agents to vary with query difficulty instead of being fixed in advance.

Plausible applications, extrapolating from the domains and settings the paper evaluates:

• Retrieval-free or retrieval-augmented assistants that mix specialist models. A service could keep several small backbone models or LoRA adapters and route each request to the one or few best matched, rather than running all of them or defaulting to one large model.
• Code generation and code-assist tooling. The paper trains and tests on MBPP, Eurus-2-Code, TACO, and HumanEval, and reports that one of four Qwen3-4B agents specialized in coding.
• Instruction-following and formatting-heavy pipelines. MAGPIE-IF, RLVR-IFEval, and out-of-distribution IFEval are all evaluated, with the top reported in-distribution MAGPIE score of 92.8 on Qwen3-4B.
• Mathematical and general reasoning workloads where cost per query matters. The reported token reduction (367.77 average tokens versus 666.37 for fixed Top-2) speaks to serving cost, and the model pool ranges down to 3B–4B parameter backbones rather than frontier-scale models.
• Deployments with heterogeneous, already-owned model fleets. The 3-model heterogeneous pool result suggests a way to use architecturally diverse checkpoints as a collaborative advantage rather than standardizing on one.

Industry relevance. The paper's practical pitch is a routing layer that sits on top of small models and adapters, updates as those models are fine-tuned, and controls inference cost by activating fewer agents on easy queries. That combination — continual fine-tuning plus cost-aware dispatch — maps onto how production LLM systems are actually operated. Caveats for practitioners: the paper reports no latency, memory, or dollar-cost measurements beyond average token counts, no serving throughput analysis, and no numeric values for its main hyperparameters, so reproducing the efficiency claims outside a research setup would require tuning those values, which the available content does not specify.

Future Directions

1. Multi-turn and debate settings. The authors explicitly propose extending single-turn semantic routing to multi-turn interactions or multi-agent debate to better support iterative problem-solving, which is the natural next step since the current design routes one query to one subset of agents.
2. Better answer aggregation. The authors propose upgrading familiarity-weighted voting to advanced answer aggregation such as an adaptive meta-thinker. Related work discussed in the paper covers LLMs as self-orchestrators, topology graph generators, and meta-thinkers, none of which CERA-MoA currently uses.
3. Publishing the missing hyperparameters and training budgets. The available content leaves λ, τ, c_ucb, w_ent, m, β, ε, G, dataset sizes, and compute costs unspecified, which limits independent reproduction and independent verification of the performance/efficiency trade-off.
4. Open questions the paper raises but does not resolve. How the approach behaves when the agent pool is large rather than N = 4 or N = 3; how well heterogeneous pools scale beyond three models; whether the emergent specialization shown in the t-SNE analysis stays stable as agents receive more data; and whether the router's fixed hidden-state features remain sufficient if the shared backbone itself is later updated, since the current design deliberately freezes the backbone and trains only the predictor head.

Target Audience

• Multi-agent system and MoA researchers looking for a routing formulation coupled to agent training rather than layered on top of fixed agents.
• Reinforcement-learning practitioners working on LLM post-training, particularly those already using DAPO, GSPO, or GRPO-style objectives and interested in sequence-level importance sampling and KL-penalty design.
• Applied engineers building inference-serving systems with multiple small models or LoRA adapters, who care about routing accuracy and token cost per query.
• Researchers studying emergent specialization and representation analysis, given the paper's use of mid-layer hidden states, orthogonal target heads, and t-SNE clustering of routed queries.
• Readers seeking a concise comparison of routing and fine-tuning baselines (ICL-Router, LinUCB, RouteMoA, GSPO, MAPoRL, AT-GRPO) across math, code, instruction-following, and general reasoning benchmarks in one set of tables.

Authors’ abstract

Current Mixture-of-Agents (MoA) paradigms generally treat query routing and agent fine-tuning as separate processes, limiting their ability to respond to evolving agent capabilities. This disconnect prevents routing strategies from adapting to evolving agent capabilities during post-training and prevents agents from achieving synergistic data-driven specialization. To resolve this, we introduce CERA-MoA (Co-Evolving Router with continually learning Agents for Mixture-of-Agents), an iterative reinforcement learning framework where the dynamic router and independent agent policies co-evolve. We design a predictive familiarity estimator that leverages mid-layer hidden states to evaluate semantic competence among agents, avoiding the overhead of full rollouts. Based on these familiarity scores, a cumulative-threshold adaptive routing mechanism dynamically activates a tailored minimal agent subset, achieving a trade-off between task performance and efficiency. By proactively allocating targeted training samples to agents based on their evolving competence, CERA-MoA promotes capability differentiation. Extensive experiments across various domains demonstrate that CERA-MoA outperforms state-of-the-art static-agent routing and fix-workflow fine-tuning baselines.