Agensh: Scaling Organizational Intelligence to 1,024 Agents

AI summary

Overview

Research area: Multi-agent large language model systems and agentic software-engineering harnesses, with an emphasis on decentralized organization rather than orchestrator-driven coordination.

Technical level: Advanced. The paper assumes familiarity with agent harnesses, tool-calling loops, context management, and benchmark evaluation for code reproduction.

One-sentence scope: The paper introduces Agensh, a self-organized multi-agent harness with no central orchestrator, and reports scaling experiments from 1 to 1,024 concurrent agents on the hardest tasks in ProgramBench.

What This Paper Is About

Most multi-agent coding systems use an orchestrator-worker structure in which a main agent plans, decomposes tasks, assigns them to workers, and integrates results. That design is bottlenecked by how much the single orchestrator can allocate and coordinate. Agensh removes the orchestrator entirely: many equal workers run concurrently and asynchronously, discover and claim their own sub-tasks, communicate with peers, and merge verified contributions into shared infrastructure. The paper's goal is to test whether increasing the number of cooperating agents acts as a scaling dimension for both the quality and the speed of complex, long-horizon work.

Key Contributions

1. An orchestrator-free multi-agent harness. Agensh couples a five-step asynchronous cooperation loop — gather context, claim sub-task, take action, verify results, merge progress — executed by every worker concurrently, with each worker proposing and claiming its own sub-tasks instead of receiving assignments.

2. Agentic organization infrastructure with three components. A shared workspace holds proposed, ongoing, and completed work; a message interface carries organization-wide announcements and urgent direct messages; shared context retains reusable findings and work intentions. The shared context idea is adopted from DeLM, and the paper adds a context grep tool for searching the full recorded history beyond recent memory.

3. A plug-and-play implementation layered above a single-agent harness. The underlying harness owns the local agentic loop (conversational state, model invocation, tool execution), while Agensh supplies organization-level behavior. The cooperation loop is realized through workflow instructions in each worker's prompt rather than hard-coded into the runtime, and the prompt is identical for every worker except the worker ID.

4. Scalability evidence up to 1,024 agents. The paper reports test-pass-rate gains on the five hardest ProgramBench tasks from 1 to 128 agents, and on pandoc from 1 to 1,024 agents, along with trajectory evidence of emergent self-organized cooperation.

Main Findings

• Agent count raises final quality on the five hardest tasks. With the same model, GPT-5.6-sol (high), the same underlying single-agent harness, Copilot, and a 6h budget, the mean final test-pass rate across the five tasks rises from 19.31% with 1 agent to 20.68% with 8 agents, 26.52% with 32 agents, and 28.78% with 128 agents. The 1-to-128 gain is 9.47 percentage points, or an approximately 49% relative improvement.

• Scaling reaches 1,024 agents on pandoc. Under the same 6h budget, the final test-pass rate rises from 33.89% with 1 agent to 50.94% with 128 agents and 55.06% with 1,024 agents. The largest organization improves on the 128-agent result by 4.12 percentage points and on the single-agent result by 21.17 percentage points.

• Larger organizations reach a given score sooner. During the first two hours, on pandoc, 128 agents exceed a 30% test-pass rate at the 30-minute checkpoint, while 32 and 8 agents first exceed that threshold at the 60- and 90-minute checkpoints respectively. The single-agent run remains below that threshold throughout the first two hours.

• Cooperation forms emerge and standardize as the organization grows. All workers follow the same loop and the same prompt except for worker IDs, yet recorded trajectories show progressively broader cooperation: at 8 agents, agreeing on technical interfaces and resolving overlapping claims (in gromacs and FFmpeg); at 32 agents, managing integration across multiple workers, including withdrawing an approval after a counterexample in PHP-src; at 128 agents, specialization and standardized workflows, including an integration protocol in pandoc that other workers reused and later revised; at 1,024 agents, role specialization at organization scale, such as multiple workers serving as integrators, contacting candidate integrators, selecting the first valid responder, cancelling other requests, and handing over code only after that selection.

• The evaluated tasks are extremely large. Reference repositories for the five tasks range from pandoc at 2,767 files, 104,336 lines of code, and 4.86 MB, to PHP-src at 26,266 files, 2,812,757 lines of code, and 119.76 MB. The other tasks are FFmpeg (10,090 files, 1,549,005 lines, 75.51 MB), gromacs (8,982 files, 815,539 lines, 47.30 MB), and ctags (7,313 files, 246,228 lines, 9.15 MB).

• Comparison to prior harnesses. The paper contrasts Agensh with orchestrator-worker frameworks including Codex sub-agent, Claude Code sub-agent and agent teams, Copilot fleet, and Kimi Agent Swarm. It also relates its shared-context and decentralized-coordination approach to DeLM, ChatCollab, SlackAgents, a compiler-building experiment coordinating 16 Claude agents through a Git repository and task-lock files, and STORM.

Methodology in Plain English

The researchers selected the five most difficult tasks in ProgramBench, choosing them based on the official ProgramBench extended results, where difficulty is determined by the average pass rate of state-of-the-art models. ProgramBench requires an agent organization to reproduce a reference program's behavior from scratch with Internet access disabled and a 6h budget, using only the pre-compiled binary's observable behavior as an information source. The 200-instance benchmark's five hardest tasks span multimedia processing (FFmpeg), molecular simulation (gromacs), document conversion (pandoc), language interpretation (PHP-src), and code indexing (ctags).

Agensh was run above Copilot as the single-agent harness, with a maximal input token limit of 272,000 and a maximal output token limit of 128,000, using gpt-5.6-sol with high reasoning effort across all reported configurations. The shared workspace is implemented with Gitea, the message interface with Mattermost, and the shared context follows the core idea of DeLM with adapted tool formats and worker instructions.

For the runtime, each worker is event-driven: a paired router listens for Gitea repository activity and Mattermost messages, writes events to a durable queue, and wakes a dispatcher that orders and combines pending events into the next prompt. Shared context reaches workers either at the start of a turn or, during an active turn, appended to infrastructure tool call results. An idle detector sends a prompt after 10 minutes of inactivity, and a follow-up prompt reminds a worker to publish findings if a completed turn produces no shared context entry.

To reduce scope contention, agent activation was staggered: one agent every 30 seconds during the first hour and one every 3 seconds thereafter. All agents are terminated at t + 6h, with a reminder at t − 45 min to stop dispatching new features and land open pull requests, and a second reminder at t − 5 min to confirm that bash compile.sh && ./executable works on the default branch. In the 1,024-worker experiments, workers were distributed across 16 nodes, with 64 agents per node. Because a single-agent run can rarely sustain the full 6-hour runtime, the single-agent baseline was given a stop hook instructing it to keep deriving fresh tasks from interaction with the reference binary.

Why This Matters

Impact on research. The paper argues that the number of agents is a new scaling dimension for multi-agent organizations, distinct from scaling model size or context length. It shows that removing the central orchestrator shifts the scalability constraint away from one coordinator's allocation capacity, and it documents self-organized cooperation forms that appear without any central assignment.

Real-world applications:

• Reconstructing or reimplementing large legacy software systems whose reference behavior must be matched, especially when source retrieval is impossible.
• Engineering work under hard latency constraints or fixed time budgets, where reaching a target quality level earlier is the objective rather than only the final score.
• Offline or air-gapped environments where remote package and source retrieval are unavailable.
• Operating coordination infrastructure for large fleets of concurrent autonomous workers, including workspace versioning, messaging priority, and shared memory management.

Industry relevance. The harness is built on commonly deployed tools (Gitea, Mattermost) and is designed to connect to different underlying harnesses, such as Claude Code and Copilot, through lightweight adapters without changing the cooperation loop, the shared services, or the underlying agentic loop. That makes the organizational layer separable from any single vendor's agent runtime, which matters for teams deploying many concurrent agents on the same codebase.

Future Directions

• Whether scaling continues beyond 1,024 agents. The paper reports results up to 1,024 agents on pandoc; behavior at larger organization sizes is not reported.
• Whether the scaling trend generalizes beyond the tested tasks. Results are reported for the five hardest ProgramBench tasks (1 to 128 agents) and for pandoc (up to 1,024 agents); the other ProgramBench instances and other benchmarks are not reported.
• Cost and resource efficiency of larger organizations. The paper reports test-pass rates and timing checkpoints but does not report token consumption or compute cost per configuration.
• Robustness and failure dependence at scale. The trajectories show workers covering for failed attempts and multiple workers holding the same specialized roles, but the paper does not quantify how organization size affects recovery rates or the consequences of individual worker failures.

Target Audience

Researchers and engineers working on multi-agent LLM systems, agent harness design, and agentic software engineering will benefit most, particularly those interested in decentralized coordination, shared-context memory, and scaling laws for agent organizations. Practitioners building concurrent coding agents on top of existing harnesses such as Copilot or Claude Code will find the infrastructure design and prompt-level cooperation protocol directly applicable. Readers seeking beginner-level material on agent orchestration will find the paper assumes substantial background.

Authors’ abstract

A multi-agent system can reduce latency on complex tasks by executing work concurrently. Several pioneering harness frameworks support multi-agent systems. However, the scalability of current multi-agent harnesses is often constrained by a central orchestrator's capacity to allocate tasks and coordinate workers. To address this limitation, we introduce Agensh, a scalable self-organized multi-agent harness without a central orchestrator: concurrent workers execute a multi-agent cooperation loop, continuously gathering context, claiming and self-assigning sub-tasks, taking action and sharing findings, verifying results, and merging progress in an asynchronous manner. The loop is supported by the agentic organization infrastructure comprising three components: a shared workspace holds proposed, ongoing, and completed work; a message interface lets workers communicate; and shared context retains reusable findings and work intentions. To test the scalability of Agensh, we evaluate it on the five hardest ProgramBench tasks with GPT-5.6-sol (high). Scaling from 1 to 128 agents raises the mean final test-pass rate from 19.31% to 28.78%, an approximately 49% relative improvement. Larger organizations reach comparable test-pass rates earlier. On pandoc, scaling from 1 to 1,024 agents raises the final test-pass rate from 33.89% to 55.06%. Worker trajectories further show that different forms of self-organized cooperation gradually emerges and standardizes as the organization grows. These results reveal the number of agents as a new scaling dimension for multi-agent organizations to expand the frontier of general intelligence, offering a practical solution for complex tasks under hard latency constraints or time budgets.