OFA-MAS: One-for-All Multi-Agent System Topology Design based on Mixture-of-Experts Graph Generative Models

AI summary

Overview

• Research area: Multi-agent systems (MAS) built on large language models, combined with graph generative models and graph neural networks. The paper is listed under the Computing methodologies categories of Multi-agent systems, Graph algorithms, and Neural networks, and appears in the Proceedings of the ACM Web Conference 2026 (WWW '26).
• Technical level: Advanced. The paper assumes familiarity with graph neural networks, autoregressive graph generation, mixture-of-experts models, and LLM-based agent orchestration.
• Scope (1 sentence): The authors propose OFA-MAS, a single universal graph generative model that designs collaboration topologies for LLM-based multi-agent systems across many task domains, replacing the prevailing "one-for-one" paradigm in which a separate design model is trained per domain.

What This Paper Is About

Multi-agent systems perform well or badly depending largely on their collaboration topology – which agents communicate, when they interact, and how knowledge propagates. Existing graph-learning approaches to automatic topology design follow a "one-for-one" paradigm: a separate model is trained for each task domain (for example MMLU), which limits generalization to unseen domains, scales poorly in maintenance cost, and ignores shared structural patterns across domains (the authors give the example of an "Analyst → Inspector → Solver" workflow shared by mathematical theorem proving and software debugging). The goal of this paper is a one-for-all designer: a single model that takes a task query written in natural language and generates an appropriate collaboration graph for it.

Key Contributions

1. Paradigm shift. The authors pioneer the transition from "one-for-one" to "one-for-all" in graph-based MAS topology design, which they describe as the first universal framework for cross-domain collaboration graph generation.
2. A new architecture. They propose OFA-MAS, an autoregressive graph generation framework combining a Task-Aware Graph State Encoder (TAGSE) with a Mixture-of-Experts (MoE) generation module.
3. A training curriculum and data pipeline. They design a three-stage easy-to-hard training strategy (unconditional pre-training, LLM-guided conditional training, supervised fine-tuning) together with an LLM-driven data synthesis pipeline that builds multi-domain "query–topology" pairs at reduced annotation cost.
4. Empirical validation. Experiments on six multi-domain benchmarks plus an out-of-distribution test on GAIA show OFA-MAS outperforming existing specialized methods.

Main Findings

• State-of-the-art average performance: OFA-MAS (fine-tuned) reaches an average score of 93.02% across MMLU, GSM8K, AQuA, MultiArith, SVAMP, and HumanEval, outperforming all baselines. Per-dataset scores are MMLU 89.54, GSM8K 95.30, AQuA 85.05, MultiArith 99.11, SVAMP 94.90, and HumanEval 94.21.
• It beats the strongest one-for-one baseline: OFA-MAS surpasses EIB-LEARNER (91.38% average) by 1.64 percentage points.
• Pre-training alone is already competitive: OFA-MAS (pre-trained), trained only on synthetic data from Stages 1 and 2 with no real or benchmark-specific supervision, achieves a 92.15% average, above EIB-LEARNER's 91.38%. Its per-dataset scores are MMLU 88.90, GSM8K 94.90, AQuA 83.18, MultiArith 99.11, SVAMP 94.27, and HumanEval 92.56.
• Fixed topologies are inconsistent: No single fixed structure wins everywhere. Chain averages 84.53% (and does well on sequential tasks like HumanEval), Tree averages 83.43%, Complete averages 82.16%, and Random averages 85.33%. LLM-Debate averages 87.53%.
• Other design baselines: AgentPrune averages 88.09%, AgentDropout 88.48%, and G-Designer 90.04%. Generic single-agent prompting is weaker still (CoT 82.84%, Self-Consistency (CoT) 83.91%, Vanilla direct answer 80.80%).
• Strong zero-shot transfer: On the unseen GAIA benchmark, OFA-MAS achieves 8.79% average accuracy, the best of all methods, with 13.21% on Level-1, 9.30% on Level-2, and 3.85% on Level-3. By comparison, G-Designer reaches only 5.50% average, Chain 7.38%, AgentPrune 5.86%, EIB-LEARNER 5.47%, and Complete 4.60%.
• Evaluation caveat on GAIA: The authors note that GPT-4o-mini is used as the base model without agentic techniques such as tool calling, so GAIA performance is not directly comparable with commercial frameworks.
• Not reported in the provided content: The specific values of the expert count K, the task vector dimension d_task, the loss weights λ_balance and λ_gate, dataset sizes, and the number of communication rounds K in the MAS execution protocol.

Methodology in Plain English

Reframing the problem as generation. Instead of adapting a fixed set of agents, OFA-MAS generates a collaboration graph from scratch for each query. It learns a conditional distribution P(G | Q, R_OFA), where Q is the task query and R_OFA is a universal role pool spanning all domains, and it searches for the topology that maximizes this distribution.

Step-by-step graph building. The graph is built autoregressively. At each step the model first picks the role of the next agent to add, then predicts which of the already-added agents should send information to it (as directed incoming edges, forming a directed acyclic graph). The full probability of a graph is the product of these per-step node and edge decisions. This design lets the model produce graphs of varying size and shape, which the authors argue fits tasks of different difficulty.

Encoding the task into every step. A pre-trained sentence transformer embeds the task query, which an MLP-based task encoder projects into a task vector that acts as a global conditioning signal. Agent role descriptions (e.g., Analyst, Programmer) are embedded with the same sentence transformer so that task requirements and agent capabilities live in one semantic space. The Task-Aware Graph State Encoder (TAGSE) then performs context-gated message passing: a gate controlled by the task vector element-wise modulates each node's message, with L1 regularization on the gate values to keep only task-relevant channels active. Messages are aggregated with a role-aware attention mechanism and combined with the previous node state through a residual update.

Specializing without losing universality. A Mixture-of-Experts module contains a lightweight gating network that takes the task vector and outputs a softmax distribution over K expert networks, plus separate expert sets for predicting the next node role and for predicting each candidate edge. The final prediction is a weighted combination of the experts' outputs. The authors motivate this with examples: mathematical reasoning tasks may activate experts favoring sequential chains, while code debugging tasks may favor hierarchical review structures.

Three-stage training. Stage 1 is unconditional pre-training on a set of classic communication topologies (Chain, Star, FullConnected) of varying size and role allocation, with the task vector set to zero, teaching the model the basic "grammar" of valid graphs. Stage 2 is conditional pre-training on synthetic data, where an LLM is prompted as a "proxy system designer" to generate large numbers of (task query, MAS configuration) pairs across domains; this teaches the alignment between task semantics and graph structure without running real MAS executions. Stage 3 is supervised fine-tuning on a smaller set of empirically validated pairs, built by executing real MAS systems with established structures (sequential chains, layered, star) on training queries across all domains and keeping the high-performing topologies.

Training objective. The total loss is the graph loss plus a balance loss (which encourages the gating network to use all experts) plus a sparse-gating regularization term, weighted by scalar hyperparameters. The graph loss is computed with teacher forcing and sums a cross-entropy loss for node role prediction and a binary cross-entropy with logits loss for edge prediction at each generation step.

Why This Matters

Impact on research. The paper challenges the default assumption that MAS topology designers must be domain-specific. It shows that a single model can exploit shared structural patterns across domains and still beat specialized models, and it offers a template for combining task-conditioned graph encoders, MoE routing, and staged curricula. It also introduces an LLM-driven synthesis route around the expensive problem of obtaining ground-truth optimal topologies, and the benchmark evidence extends to an unseen dataset (GAIA), which is where one-for-one methods are weakest.

Real-world applications (several of which the paper itself names as contexts where MAS are being integrated):

• Web search engines and online knowledge bases that must serve cross-domain user queries without asking users to pre-categorize their requests.
• Cloud-based task execution platforms that orchestrate multiple specialized agents.
• General-purpose agentic services where maintaining a separate topology model per supported domain is unsustainable.
• Multi-step reasoning workflows, such as mathematical problem solving and code generation/debugging, which are the paper's evaluation domains.

Industry relevance. The "intractable scalability and maintenance" argument is operational: training multiple specialist models requires a dedicated data collection, training, and tuning pipeline per domain. A single universal designer reduces this to one model to maintain, while still allowing domain-specific behavior to emerge through expert routing and task conditioning.

Future Directions

• Reducing reliance on real-data fine-tuning. OFA-MAS (pre-trained) already reaches 92.15% average without benchmark-specific supervision, which raises the question of how much of Stage 3 is necessary and whether better synthetic data can close the remaining gap.
• Improving synthetic data quality. The pipeline uses an LLM as a "proxy system designer"; the paper does not report verification of those synthetic topologies, leaving open how to validate or filter them.
• Closing the OOD gap. GAIA averages remain low (8.79% for OFA-MAS, with 3.85% on Level-3), so generalization to genuinely hard unseen tasks is unresolved. The authors also note that GAIA results were obtained with GPT-4o-mini without agentic techniques such as tool calling, so the effect of tool use is untested.
• Scaling and hyperparameter characterization. The provided content does not report the number of experts, task-vector dimension, or loss weights, so the sensitivity of the design to these choices remains an open question.

Target Audience

Researchers and practitioners working on LLM-based multi-agent systems, automatic agent orchestration, and graph generative models will benefit most. It is also relevant to engineers deploying agentic services on the web who need cross-domain topology design without per-domain model maintenance, and to readers interested in applying mixture-of-experts and curriculum learning to structured prediction problems.

Authors’ abstract

Multi-Agent Systems (MAS) offer a powerful paradigm for solving complex problems, yet their performance is critically dependent on the design of their underlying collaboration topology. As MAS become increasingly deployed in web services (e.g., search engines), designing adaptive topologies for diverse cross-domain user queries becomes essential. Current graph learning-based design methodologies often adhere to a "one-for-one" paradigm, where a specialized model is trained for each specific task domain. This approach suffers from poor generalization to unseen domains and fails to leverage shared structural knowledge across different tasks. To address this, we propose OFA-TAD, a one-for-all framework that generates adaptive collaboration graphs for any task described in natural language through a single universal model. Our approach integrates a Task-Aware Graph State Encoder (TAGSE) that filters task-relevant node information via sparse gating, and a Mixture-of-Experts (MoE) architecture that dynamically selects specialized sub-networks to drive node and edge prediction. We employ a three-stage training strategy: unconditional pre-training on canonical topologies for structural priors, large-scale conditional pre-training on LLM-generated datasets for task-topology mappings, and supervised fine-tuning on empirically validated graphs. Experiments across six diverse benchmarks show that OFA-TAD significantly outperforms specialized one-for-one models, generating highly adaptive MAS topologies. Code: https://github.com/Shiy-Li/OFA-MAS.