Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration

AI summary

Overview

Research area: Interpretability of multi-expert / multi-LLM orchestration systems (machine learning, LLM systems).

Technical level: Intermediate. The paper uses concepts such as routing entropy, KL divergence, Gini coefficients, and gradient-norm attribution, but each is explained in functional terms.

Scope: The paper introduces INFORM, an interpretability analysis that decomposes a learned multi-expert orchestrator into interaction structure, execution order, and functional attribution, and applies it to orchestrators coordinating ten LLM experts on GSM8K, HumanEval, and MMLU.

What This Paper Is About

Multi-expert systems route prompts through a sequence of collaborating LLMs, but the policy deciding which expert is invoked, in what order, and why is usually a black box. The authors build INFORM to open up that policy and, in particular, to test whether the experts that get selected most often are actually the experts the system depends on. Their central claim is that routing dominance is a poor proxy for functional necessity: frequently routed experts can be interaction hubs with limited influence, while sparsely routed experts can be structurally critical.

Key Contributions

1. INFORM, an orchestration interpretability framework. It treats orchestration as an explicit, analyzable computation and separates three things that are normally conflated: expert interaction structure (captured by a conditional transition matrix), execution order (captured by the marginal selection distribution), and functional attribution (captured by gradient sensitivity of selection decisions to expert representations).

2. A formal distinction between relational and intrinsic importance. Relational importance is defined as total incoming routing mass in the interaction graph; intrinsic importance is defined as the gradient norm of the selected expert's log-probability with respect to that expert's representation. The paper shows these two measures diverge in practice, and that the alignment between them is task-dependent.

3. Empirical characterization of how orchestration emerges during training. Using a learned orchestrator over ten instruction-tuned experts, the paper documents gradual onset of routing confidence and specialization, non-deterministic ordering preferences that remain above zero entropy, and an asynchrony in which expert centralization precedes stable routing confidence.

4. Targeted ablation and perturbation evidence. Masking the single most intrinsically important expert on MMLU induces 5.5x higher routing KL divergence than sequencing divergence. A comparison against MetaGPT on HumanEval shows adaptive orchestration reduces average model calls from 5.00 to 1.44 while raising Pass@1 from 85.9 to 87.1.

Main Findings

• Routing dominance and functional necessity diverge. Across tasks, the experts that receive the most routing mass are not always the ones with the highest gradient attribution. The paper identifies "interaction hubs": experts that are frequently selected as successors but have limited standalone representational influence on the routing decision.

• Alignment is task-dependent. GSM8K shows the strongest alignment between intrinsic attribution and routing mass, indicating grounding in intrinsic competence. HumanEval shows partial alignment, where some dominance is driven by interaction effects. MMLU shows the weakest alignment, which the authors attribute to domain heterogeneity and strong context-dependent complementarity.

• Orchestration behaviors emerge asynchronously. Routing entropy (confidence) and the Gini coefficient over incoming routing mass (centralization) do not move together. Centralization increases before routing entropy fully stabilizes, which the authors summarize as the orchestrator learning who to trust before resolving how confidently to route.

• Expert ordering stays soft and non-deterministic. Ordering entropy decreases across training but remains significantly above zero on all tasks. GSM8K shows the strongest reduction, suggesting emergence of a preferred reasoning initializer, while HumanEval and MMLU retain higher entropy, consistent with multiple viable entry points.

• Collaboration structure sharpens over training. On HumanEval, the collaboration matrix starts diffuse, reflecting exploratory transitions, and by Epoch 5 develops prominent vertical bands, meaning a small subset of experts consistently receives most routing probability regardless of source expert. Several experts receive negligible probability mass.

• Initial-expert preferences concentrate with training. In the reported sequence distribution for HumanEval, expert 3 rises from 0.158 at Epoch 1 to 0.354 at Epoch 5, and expert 8 rises from 0.068 to 0.303 over the same span, while other experts lose mass.

• Routing sensitivity is task-specific and training-dependent. Perturbation effects via KL divergence are largest in early epochs, then become structured. Removing numerical tokens causes the largest shift on GSM8K, whereas sentence shuffling and removal of reasoning cues have stronger effects on MMLU and HumanEval.

• Brittleness under destructive perturbations. Clean data yields stable, monotonic accuracy gains across epochs on MMLU, while structural perturbations such as removing numbers or reasoning cues induce high volatility and sharp late-epoch degradation. Token cost rises under sentence shuffling, which the authors describe as the model "rambling" under input ambiguity.

• Masking intrinsically important experts collapses interaction structure. On GSM8K and MMLU, masking the top intrinsic expert causes substantially larger divergence in the routing distribution than in the sequencing distribution, indicating these experts act primarily as interaction hubs. On MMLU this divergence remains consistently high; HumanEval shows the opposite pattern in the truncated content.

• Heterogeneous consortia are less stable. With experts spanning 1B to 7B parameters, routing confidence converges more slowly, centralization is weaker, the MMLU Gini coefficient keeps fluctuating, sequencing entropy stays higher, and intrinsic importance is spread across a wider set of models. Routing mass nevertheless clusters sharply on a few models, producing a persistent alignment gap.

• Learned orchestration beats a rigid SOP baseline. Against MetaGPT on HumanEval with matched expert sizes, INFORM cuts per-role call counts to 0.69 (Engineer), 0.26 (QA), 0.32 (Architect), 0.10 (PM), and 0.07 (ProjM), for 1.44 average calls versus 5.00, translating to roughly a 3.5x speedup with a roughly 1.4 percent performance gain.

• Scope of the attribution claim. The authors state that the gradient-based approach captures local sensitivity of routing decisions rather than full causal structures, and provides functional attribution without formal guarantees.

Methodology in Plain English

The authors build a differentiable orchestrator that manages ten frozen, instruction-tuned LLM experts. The orchestrator encodes expert outputs with a frozen BERT-based encoder into a shared space, then uses two modules. An interaction module produces a conditional transition matrix over experts using query-key attention plus a static semantic prior; a selection module turns that connectivity into a marginal distribution over which expert to invoke, sampled with the Gumbel-Softmax trick during training. Training combines task performance, alignment of the final output to a larger oracle LLM that is absent at inference, symmetry enforcement, and sparsity penalties, with an adaptive-k schedule.

INFORM then interrogates that trained policy in three ways. First, it reads the transition matrix as a directed weighted graph and computes incoming routing mass as relational importance, plus entropy and Gini coefficient to measure rigidity and specialization. Second, it inspects the marginal selection distribution to see which experts emerge as first movers and how ordering entropy changes. Third, it backpropagates the log-probability of the selected expert to that expert's representation and takes the gradient norm as intrinsic importance, which measures whether selection depends on the semantic content of the expert rather than a heuristic.

To test whether these measurements reflect real structure rather than artifacts, the authors run four prompt perturbation classes (removing numbers, masking numbers, shuffling sentences, removing reasoning cues), measure KL divergence between baseline and perturbed sequence distributions, and perform ablations including replacing the learned interaction graph with a uniform static graph, fixing a static execution sequence, masking the top-attributed expert, and comparing relational-only and intrinsic-only variants. The primary setup uses a homogeneous consortium of ten experts drawn from LLaMA-3.1 8B, Qwen3 8B, and DeepSeek-R1 8B with controlled decoding-temperature variation; a secondary heterogeneous consortium uses LLaMA-3.2 1B, Qwen2.5 3B, and Mistral 7B. The orchestrator sees the first 30 tokens generated by each expert, and evaluation uses held-out test subsets.

Why This Matters

Impact on research. The paper reframes orchestration policy as an object of study rather than only an optimization target. Its relational-versus-intrinsic distinction gives a way to identify redundant or decorative routing, and it argues that treating routing as order-invariant misses a genuinely sequential dependency. Because the framework only requires a differentiable routing policy, it is positioned as applicable beyond the specific attention-based orchestrator analyzed, and the authors discuss generalization to confidence-based cascades and note an appendix discussion of black-box and API-based environments.

Real-world applications:

• Cost control in production LLM pipelines. The finding that orchestrators can route to experts they do not fundamentally depend on points to pruning opportunities, consistent with the call-count reduction observed against MetaGPT.
• Debugging and monitoring deployed multi-agent workflows. Routing entropy, centralization, and perturbation sensitivity can serve as runtime signals for brittle routing, degeneration into static ensembles, or silent cost inflation, all of which the paper names as failure modes.
• Safety and reliability in high-stakes or tool-augmented settings. Detecting experts that are structurally critical but sparsely routed helps flag single points of failure before deployment.
• Model selection and consortium design. The homogeneous-versus-heterogeneous comparison offers evidence on how mixing capability tiers affects stability, which is directly relevant to building mixed-size expert pools.

Industry relevance. Teams running cascades, routers, or multi-agent frameworks need to know whether their routing decisions reflect task-relevant dependency or incidental heuristics. The paper's alignment-gap framing, and its demonstration that high gradient attribution appears where routing mass does not, offers a concrete diagnostic.

Future Directions

• Moving beyond local sensitivity toward causal attribution. The authors explicitly limit their claim to local sensitivity of routing decisions and disclaim formal causal guarantees, leaving room for stronger causal identification of expert necessity.
• Extending INFORM to black-box and API-based systems. Appendix K discusses applicability, but gradient attribution as defined requires internal access; adapting the framework to non-differentiable or externally hosted experts is an open problem.
• Stabilizing heterogeneous orchestration. The heterogeneous consortium shows slower convergence, weaker centralization, and a persistent alignment gap, and the authors hypothesize this reflects a struggle to arbitrate between raw capability and specialization, an unresolved trade-off.
• Generalizing the interpretability principles across orchestration regimes. The paper reports that applying INFORM to confidence-based cascade orchestration reveals a similar misalignment between stopping dominance and functional necessity, suggesting further cross-regime testing.

Target Audience

Researchers and practitioners working on LLM routing, mixture-of-experts, multi-agent collaboration, and model interpretability, particularly those who need to audit whether an orchestration policy's selections reflect genuine functional dependency. It is also relevant to engineers building cost-sensitive or safety-critical multi-model systems who want metrics beyond task accuracy for evaluating routing behavior. Readers should be comfortable with basic probability concepts such as entropy and KL divergence, though the paper explains how each is applied.

Authors’ abstract

Multi-expert systems, where multiple Large Language Models (LLMs) collaborate to solve complex tasks, are increasingly adopted for high-performance reasoning and generation. However, the orchestration policies governing expert interaction and sequencing remain largely opaque. We introduce INFORM, an interpretability analysis that treats orchestration as an explicit, analyzable computation, enabling the decoupling of expert interaction structure, execution order, and functional attribution. We use INFORM to evaluate an orchestrator on GSM8K, HumanEval, and MMLU using a homogeneous consortium of ten instruction-tuned experts drawn from LLaMA-3.1 8B, Qwen3 8B, and DeepSeek-R1 8B, with controlled decoding-temperature variation, and a secondary heterogeneous consortium spanning 1B-7B parameter models. Across tasks, routing dominance is a poor proxy for functional necessity. We reveal a divergence between relational importance, captured by routing mass and interaction topology, and intrinsic importance, measured via gradient sensitivity: frequently selected experts often act as interaction hubs with limited influence, while sparsely routed experts can be structurally critical. Orchestration behaviors emerge asynchronously, with expert centralization preceding stable routing confidence and expert ordering remaining non-deterministic. Targeted ablations show that masking intrinsically important experts induces disproportionate collapse in interaction structure compared to masking frequent peers, confirming that INFORM exposes functional and structural dependencies beyond accuracy metrics alone. Our code is available at https://github.com/parmanu-lcs2/inform.