Learning Self-Orchestration in Decentralized Agent Swarms

AI summary

Overview

Research area: Multi-agent systems, multi-agent reinforcement learning, and LLM agent post-training (specifically: decentralized coordination and credit assignment).

Technical level: Advanced. The paper assumes familiarity with policy-gradient reinforcement learning, preference optimization, and LLM fine-tuning pipelines.

Scope: The paper introduces market-based post-training (MPT), a framework in which a population of LLM agents jointly learns task actions and its own turn-allocation (orchestration) behavior through bidding and auction, and evaluates it on embodied crafting, robot navigation, and deep research.

What This Paper Is About

Most LLM multi-agent systems place agents inside a fixed orchestration structure — a manager delegating to workers, a verifier checking a generator, or a prescribed workflow — so the policy deciding which agent acts next stays frozen even when the agents themselves are trained. The authors argue this is a mistake, because participation determines which agents collect experience and learning signal, and as agents improve, the best pattern of participation should change too. The goal is a decentralized swarm that learns both how to act and when to participate, using only collective task outcomes, with no central coordinating agent.

Key Contributions

1. A joint learning framework (MPT). Agents bid for the right to act at each turn; the highest bidder acts, and its bid also sets how much accumulated value is passed backward to the previous acting agent, coupling turn allocation with credit assignment. Action policies are trained by REINFORCE on this local credit, and bidding is trained by counterfactual preference comparisons.
2. An empirical evaluation across three domains. The authors compare MPT against monolithic post-training under matched training FLOPs on embodied crafting (Minecraft Iron Pickaxe), robot navigation (VLN-CE R2R in Habitat), and deep research (OpenResearcher), reporting higher held-out performance on all three.
3. Evidence that learned self-orchestration is the source of the gains. With post-trained action policies held fixed, the learned bidding mechanism is compared against Round-Robin, AgentNet, and Graph-of-Agents, and ablations (round-robin turn order, random credit values) isolate bidding and credit assignment as separate contributors.
4. Demonstrations of reuse, inference-time scaling, and robustness. Trained swarms adapt faster than the monolithic baseline to new crafting objectives, benefit from larger agent populations at inference time without additional training, and lose less performance under agent dropout as population size grows.

Main Findings

• Held-out gains over monolithic RL on all three tasks. On 32 unseen world seeds for embodied crafting, MPT scores 41.5 ± 5.9 mean episode reward versus 20.3 ± 4.1 for RL. On 88 R2R-CE val-unseen episodes, 1.23 ± 0.18 versus 0.52 ± 0.14. On 50 held-out OpenResearcher questions, 21.0 ± 4.1% versus 9.0 ± 2.1% mean accuracy.
• Training-time curves also favor MPT at matched FLOPs. In Figure 5, MPT reaches a final embodied-crafting reward of approximately 41 versus 20 for the monolithic baseline; on robot navigation it ends at approximately 0.9 versus 0.3; on deep research it reaches approximately 27% versus approximately 6% mean accuracy.
• Participation patterns change during training. Tracking Iron Pickaxe participation across four checkpoints, at decision step 120 the Logger's share drops from roughly 80% before RL to 45% after RL, the Miner's rises from roughly 10% to 40%, and the Smith's from roughly 10% to 15% — early turns stay Logger-dominated while later turns shift toward the Miner and Smith.
• Learned self-orchestration beats alternative coordination strategies with identical action policies. On Iron Pickaxe over 32 seeds: MPT 41.5 ± 5.9, Graph-of-Agents 25.4 ± 4.6, Round-robin 23.8 ± 4.2, AgentNet 20.9 ± 4.2. Against the strongest alternative (GoA) this is a 63.4% relative improvement.
• Both action and bidding must be learned. Post-trained bidding with initial actions gives 21.5 ± 4.7; initial bidding with post-trained actions gives 27.2 ± 4.8; both post-trained gives 41.5 ± 5.9; both initial gives 15.4 ± 4.1 (a 2.69× improvement for the fully trained configuration).
• Learned bids also improve credit assignment, not just turn selection. At approximately 5×10^18 training FLOPs on Iron Pickaxe, MPT reaches about 42 mean reward, versus 31 when bids for credit assignment are replaced by values sampled from U(0,1), and 15 for the round-robin ablation, which improves little over training.
• Trained swarms adapt faster to new objectives. Continuing from Iron Pickaxe checkpoints to Torch and Stone Bricks, at roughly 10^18 adaptation FLOPs the society reaches about 35 and 38 mean reward versus 17 and 19 for monolithic RL.
• Inference-time populations scale. Increasing population from 3 to 12 agents (one to four copies of each trained agent) raises pass@3 from approximately 78% to 97% on Wooden Pickaxe, 69% to 94% on Stone Pickaxe, and 66% to 91% on Furnace.
• Larger swarms tolerate dropout better. At dropout probability p = 0.9, the 12-agent population retains approximately 33 mean episode reward versus 19 for the 3-agent population.

Methodology in Plain English

Each episode is a sequence of turns with one agent acting per turn, and a verifier assigns the final reward. Every agent is a separately fine-tuned LLM with its own parameters; it produces two things at each decision — an action and a bid representing how willing it is to act now.

Who acts. At each turn agents submit bids independently; the highest bidder acts, ties are broken at random, and if every agent bids zero the turn is skipped. There is no dispatcher deciding this.

How credit flows. After an episode, reward is propagated backward through the sequence of acting agents. At turn t the acting agent keeps a fraction (1 − b_t) of the accumulated value as its own training credit and passes the remaining fraction b_t backward to the previous acting agent, with the first agent's passed share discarded. This is a bucket-brigade-style chain, so downstream outcomes still teach earlier actions. Each agent then updates only on the turns it actually played, using REINFORCE with its own mean baseline.

How bids are learned. The paper calls a turn "contested" when the runner-up bid is nonzero. For such a turn, the authors restore the environment and agent states in a replica and resample the selected agent's bid up to three times until it falls below the runner-up's bid, so the runner-up acts instead; if no such bid is found, that turn is dropped from bidding training. The alternative branch is rolled out to episode end, and both branches are scored by summed environment reward from the contested turn onward. The higher-scoring bid becomes the preferred response and the other the rejected one; equal-return pairs are discarded. Training uses a Bradley–Terry preference loss on this pair. Action and bidding updates alternate on the same model parameters — four action steps followed by one bidding step.

Experimental setup. All three tasks use three agents with complementary roles. Embodied crafting uses three Qwen3-VL-4B-Instruct agents (Logger, Miner, Smith) in MineStudio, each episode 120 decisions, with a frozen STEVE-1 controller executing instructions for 40 game frames; post-training draws from 1614 seeds. Robot navigation uses three NaVILA-8B agents with partitioned action spaces (Turner, Mover, Stopper) on VLN-CE R2R in Habitat, at most 64 decisions. Deep research uses three gemma-4-E2B-it agents (Searcher, Opener, Finder) over OpenResearcher, at most 64 decisions, with a binary terminal reward judged by qwen3.5-flash. All agents get a supervised fine-tuning cold start; the monolithic baseline is trained on the combined role demonstrations with total SFT compute matched, and uses REINFORCE with discount factor γ = 0.95.

Why This Matters

Impact on research. The paper reframes orchestration as a learnable component of collective intelligence rather than a hand-designed scaffold, and connects turn allocation directly to credit assignment. It also distinguishes itself from prior economic multi-agent work (such as auction-based prompt evolution) by updating model weights rather than only prompts, and from standard multi-agent RL by tying the choice of next actor to how downstream reward is distributed.

Real-world applications.

• Robot fleets and embodied task pipelines: heterogeneous robots or controllers with partitioned capabilities (movement, rotation, stopping, manipulation) that must decide on the fly which unit should act next, and that must keep working when a unit goes offline.
• Deep-research and tool-use assistants: specialist agents for search, page opening, and in-page finding that self-select at each step instead of following a fixed script.
• Long-horizon operational workflows such as automated software or infrastructure tasks, where the sequence of specialists needed changes as the task state evolves.
• Elastic deployment: serving a variable number of model replicas and letting the learned bidding mechanism absorb the extra capacity, or degrade gracefully when replicas fail.

Industry relevance. The inference-time scaling and dropout results speak directly to deployment concerns: extra agent copies improve pass@3 without retraining, and redundancy cushions failures. The finding that learned bids give better credit-assignment signals than random ones suggests that participation signals are a cheap, useful training resource for teams already running multi-agent pipelines.

Future Directions

• Scaling the trained population. Training used three agents throughout, while inference scaling was tested by replicating trained agents up to 12. Does training with larger, more diverse populations produce different or better orchestration?
• Cost and efficiency of counterfactual rollouts. Bidding training requires restoring states and re-running alternative branches (with up to three bid resamples per contested turn). Reducing this cost, or approximating counterfactual returns, would broaden applicability.
• Heterogeneous and partially decentralized settings. All agents here share the same observation and history structure. Whether bidding works when agents have private or conflicting information, or when bids could be strategically misreported, is unexplored.
• Continual and lifelong reuse. The reuse experiments cover two follow-on crafting tasks. How orchestration behaves across many sequential objectives, including forgetting of earlier skills, remains an open question.

Target Audience

Researchers and practitioners in multi-agent reinforcement learning and LLM agent systems who are designing coordination architectures and post-training pipelines. It is most useful to readers already comfortable with policy-gradient methods and preference optimization; readers new to RL will find the mechanism intuitive but the training details demanding. Engineers building production multi-agent systems with heterogeneous specialist models and elastic serving capacity are a secondary audience.

Authors’ abstract

The capabilities of an agent swarm depend on both its individual members and how they coordinate. Yet existing multi-agent systems often train agents within fixed orchestration structures, separating the learning of individual capabilities from the organization of collective behavior. We introduce market-based post-training (MPT), a framework for jointly learning action policies and self-orchestration in decentralized agent swarms. Agents express their willingness to participate through bids, which determine who acts and how collective outcomes provide local training credit. Learning both action and bidding behavior allows individual capabilities and patterns of participation to adapt together without a central coordinating agent. We evaluate MPT on embodied crafting, robot navigation, and deep research. Results show gains over monolithic post-training and confirm that agent participation indeed evolves with learning. With the same post-trained action policies, learned self-orchestration outperforms alternative coordination strategies. Ablations show that both action and bidding benefit from learning, and that learned bids provide more effective action-training signals. Beyond the original training objectives, trained swarms adapt faster to related tasks and achieve better performance with larger populations at inference time. Larger populations also suffer smaller performance losses under agent dropout, demonstrating that learned self-orchestration supports reuse, scaling, and robustness in decentralized agent swarms. Code and trained models are available at https://github.com/Embodied-Minds-Lab/MPT.