The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation

AI summary

Overview

• Research area: Multi-agent AI / language-model agent collaboration, benchmark design, and evaluation of agentic systems in partially observable environments.
• Technical level: Intermediate. The task itself (maze solving) is simple to grasp, but the experimental design involves homogeneous vs. heterogeneous model pairings, mediation analysis of "grounding," and logistic-regression modeling of disagreement events.
• Scope: The paper introduces a collaborative maze-solving benchmark for unconstrained, natural-language AI–AI collaboration and uses it to evaluate 32 leading open- and closed-source models in solo, homogeneous, and heterogeneous pairings.

What This Paper Is About

As AI systems increasingly become compositions of independently developed agents with different information, privileges, and tools, their usefulness depends on whether they can collaborate — including under partial observability. The problem is that existing multi-agent work relies on fixed interoperability protocols (e.g., MCP, A2A, ACP) or central orchestration, which the authors argue cannot cover every open-world situation or be enforced across multiple parties. The paper builds a scalable maze-solving benchmark that imposes no output-format constraints, in order to measure unguided, natural communication and find out whether strong solo performance carries over to teamwork.

Key Contributions

1. Formal definition and empirical evidence for the "collaboration gap": models that perform well solo exhibit a statistically reliable performance drop when collaborating with an identical copy of themselves, with the gap appearing especially severe in distilled models.
2. Analysis of heterogeneous collaboration dynamics: task performance is heavily influenced by which agent initiates the dialogue, and the stronger model generally acts as an upper bound on performance.
3. "Relay inference," a new collaborative strategy: a stronger model "seeds" the initial steps of a task before handing off to a weaker one, substantially boosting collaborative performance of weaker models.
4. A scalable benchmark design: the "distributed map" methodology (splitting information across two partial views) enables automated grading and modulates problem complexity, providing a template for designing new collaboration tasks.

Main Findings

• Solo performance is broadly strong: with a fully visible 6×6 maze, most evaluated models reach weighted outcomes above 0.5, and half the models reach near-perfect solve rates. Performance drops significantly for about a third of models when the same information is distributed over two map copies.
• The collaboration gap is near-universal: virtually all studied models experience a statistically reliable performance drop moving from solo to collaborative settings, a degradation the authors argue is unlikely to be explained by context or turn limits.
• Model size and style matter, with exceptions: larger and "thinking" models tend to perform better than smaller, "regular" models, but grok-3 and kimi-k2 both underperform grok-3-mini, which is reportedly not a distilled model but purpose-built for its size.
• Distilled models are disproportionately affected: the paper points to large drops such as gpt-5 to gpt-5-nano and gemini-2.5-pro to gemini-2.5-flash-lite, suggesting current distillation strategies may lose more than information from the tail-ends of a base model's knowledge distribution.
• Grounding separates good from bad collaborators: in a qualitative comparison of first messages, o3 immediately provides a fully determined schema, requests missing information, and grounds the starting position, whereas gpt-4.1-mini only attempts to ground symbol meanings without proposing a communication schema or defining its starting position. A mediation analysis found a statistically significant relation between grounding proxies and weighted outcomes.
• Ordering effects in heterogeneous pairings: gpt-4.1-mini paired with o3 performs markedly better when o3 starts (0.77 ± 0.04) than when gpt-4.1-mini does (0.62 ± 0.06). Stronger models also show "style imitation," partially adopting conventions introduced by weaker partners.
• Weaker models do not always set a lower bound: gemini-2.5-flash-lite and gpt-4.1-mini each perform better with copies of themselves than when combined. gemini-2.5-flash-lite does not improve when paired with the stronger gpt-4.1 but performs well when paired with gemini-2.5-flash, which the authors read as evidence of in-family affinity.
• Exceptions to the ordering trend: claude-sonnet-4 is the sole exception in the flagship panel — its collaborative performance with gemini-2.5-flash and gpt-4.1 surpasses both models' homogeneous performance (though within the confidence interval). grok-3-mini is described as an especially capable collaborator that keeps performance close to its homogeneous baseline and actively challenges incorrect moves rather than deferring.
• Conflict dynamics differ by pairing type: the odds of encountering any conflict are significantly lower for homogeneous collaborations (OR = 0.60). Strong- and medium-strength models are more likely to encounter disagreements than weaker ones, factual conflicts are most likely in medium-strength models, and non-factual conflicts increase with capability. Once a conflict occurs, homogeneous pairs are just as likely as heterogeneous pairs of comparable strength to be disputing strategy or grounding (OR = 0.97).
• Priming beats recovery: a single priming message from o3 substantially boosts both gpt-4.1-mini and gemini-2.5-flash-lite, while late interventions show diminishing returns — prolonged initial exchanges between weak models make strong recovery harder, likely due to style imitation effects.
• Grading is stable: ablation across multiple rounds of grading with gpt-4.1, plus comparisons using o3 and gemini-2.5-flash, found no evidence for statistically significant noise in the grading process or unfair biases across models.

Methodology in Plain English

The team uses maze solving as the target task, following earlier studies of human collaboration. Each maze is an N×N grid with a start and goal state, path cells, and wall cells; the main experiments use 6×6 mazes with 30% wall density and an average solution path of 7 to 9 steps, sized so that most models can solve them solo.

The key design twist: instead of giving both agents the complete maze, the researchers randomly obfuscate roughly half the cells of each agent's copy with "?" symbols, so that the two partial maps together recover the full map. This forces the agents to exchange information in order to plan a route. To fairly compare solo against collaborative performance, a single agent can also be shown both partial maps at once, measuring its ability to handle distributed information.

Two rules are given: both agents must agree on a move before it is executed, and only one move can be executed at a time. Except for a predetermined completion phrase, no communication protocol or output format is enforced. Agents use text-based visual representations of the maze, and each turn they see only their own map copy, the rules, and the dialogue so far. Rollouts run until the completion phrase is uttered or a maximum turn count is reached.

Because the transcripts are unstructured, a third agent acts as a "grader" to extract the agreed-upon moves, which are then normalized under a large set of possible coordinate schemes (e.g., different origins, row-column vs. column-row) and checked against the ground-truth maze. This auto-grading enables experiments at a scale — described as many tens of thousands of transcripts — that human annotation could not support.

Experiments span three settings: a solo baseline under full and distributed visibility (100 samples per setting, with a "critic" step allowing the agent to review its proposed solution), homogeneous collaboration with an independent copy of the same model (100 rollouts per agent, maximum 50 turns), and heterogeneous collaboration between different model families and/or strengths (at least 50 rollouts per pairing, maximum 50 turns). Relay inference experiments freeze the first K ∈ {2, 4, 6, 8} turns and swap in either the weaker or stronger model, with at least 100 rollouts per model pairing and relay point. Outcomes are reported as weighted outcomes, defined as the ratio between the optimal path length and the distance from the last valid position on a path to the goal, plus binary success rate.

Why This Matters

Impact on research. The paper frames collaboration as a distinct axis of capability that current training recipes — largely built around solo reasoning benchmarks — insufficiently capture. It argues for collaboration-aware evaluation, training strategies that explicitly target collaborative skills, and deliberate interaction design, and it challenges the community to treat collaborative intelligence as a core design objective rather than an emergent property. The authors also conjecture that since a large gap exists even in the stylized case of mazes, the gap may be wider — quantitatively and qualitatively — in more complex real-world use cases.

Real-world applications:

• Collaborative software engineering with split context: just as two maze agents must reconcile different map views, coding agents may each observe only a subset of relevant files — because a codebase does not fit in a context window, or because of role-based access controls — and must agree on folder structure, imports, coding conventions, package versions, shared dependencies, and resolve errors.
• Open-world agent interoperability: because defining a fixed communication protocol in advance for every situation is infeasible and cannot be enforced across agents controlled by multiple parties, successful deployment will depend on flexible, on-the-fly communication.
• Heterogeneous specialist agents: the case for capable small specialist models to lead the agentic age needs an important caveat — the more specialized an agent becomes, the more likely it meets challenges outside its expertise, increasing the need to collaborate to fill capability gaps.
• Human–AI collaboration: the finding that priming improves weaker models while weaker models leading subdues performance holds lessons for settings typically led by humans, and connects to initiatives to increase people's competence in interacting with AI, as well as the "specialist librarian problem," where the AI must first decipher a person's true needs before solving them.

Industry relevance. Current multi-agent integration depends on predetermined interoperability protocols such as MCP, A2A, and ACP, or centrally orchestrated architectures. The finding that even tightly orchestrated systems can be bottlenecked by ineffective communication, poor collaboration, and conflicting behaviors among agents suggests that naively breaking up problems across multiple agents could introduce "collaborative slippage." The paper also notes that replicating the study comes at considerable cost — driven by frontier model pricing and the rising cost of generating subsequent turns, which for average message length K tokens and maximum turns T gives an average input context length of K·(T−1)/2 — meaning such evaluations are likely prohibitive for many labs and practitioners.

Future Directions

• Training explicitly for collaboration. The authors call for training strategies that enhance collaborative capabilities, arguing that capabilities needed for collaboration "cannot be patched on, but must be designed in from the start." How such training should be implemented is left open.
• Designing and scaling new collaboration tasks. The "distributed" information methodology is proposed as a scalable approach to designing new collaboration tasks; the paper leaves exploration of these to future work, including partial observability in collaborative software engineering.
• Exploiting relays and priming deliberately. Relay inference shows that priming with a strong model is more effective than using one as a backup expert, raising open questions about how to structure interaction order and hand-offs across heterogeneous agent teams.
• Understanding why distillation hurts collaboration. The disproportionate gap in distilled models suggests current distillation may lose more than tail-end knowledge, but the mechanism is not resolved.

Target Audience

This paper is most useful for AI researchers studying multi-agent systems and emergent collaboration in language-model agents; benchmark and evaluation designers looking for scalable, auto-graded task constructions that isolate a specific capability; engineers building or orchestrating heterogeneous agentic systems (including those working with MCP, A2A, or ACP); and human–AI interaction researchers interested in how grounding, priming, and interaction order shape joint outcomes.

Authors’ abstract

The trajectory of AI development suggests that we will increasingly rely on agent-based systems powered by language models, composed of independently developed agents with different information, privileges, and tools. The success of these systems will depend critically on effective collaboration among these heterogeneous agents, even under partial observability. Despite intense interest, the literature lacks empirical studies evaluating agentic collaboration without relying on fixed communication protocols, limiting insights for open-world deployments. We propose an illustrative collaborative maze-solving benchmark that (i) isolates collaborative capabilities, (ii) modulates problem complexity, (iii) enables scalable automated grading, and (iv) imposes no output-format constraints, benchmarking unguided, natural communication. Using this benchmark, we evaluate 32 leading open- and closed-source models in solo, homogeneous, and heterogeneous pairings. Our results reveal a surprising collaboration gap: models that perform well solo often degrade substantially when required to collaborate. We identify mitigations that show remarkable influence. For example, a small nudge via a relay inference approach, where a stronger agent leads before handing off to a weaker one, closes much of the gap. Our findings argue for (1) collaboration-aware evaluation, (2) training strategies to enhance collaborative capabilities, and (3) deliberate interaction design to elicit agents' collaboration skills, principles relevant to AI-AI and human-AI settings where agents must establish common ground.