Who Takes the Lead? Model Ability and Task-Conditioned Collaboration with Humans

AI summary

Overview

Research area: Human–Computer Interaction / human–AI collaboration, specifically the evaluation of language-model agents that share control of a task with a human partner.

Technical level: Intermediate. The maze task itself is intuitive, but interpreting the paper requires comfort with agent evaluation metrics, experimental conditions, and bootstrap-based uncertainty estimates.

Scope: The paper asks how an LLM agent's standalone task capability relates to when and how it hands control to a human teammate, using a purpose-built shared-control maze in which the human's positional advantage is manipulated by construction.

What This Paper Is About

When a language-model agent stops asking a human for help, that can look like a collaboration failure — or it can be a correct division of labor. Raw rates of human involvement cannot distinguish the two, because frequent involvement might mean the agent is wisely using a better-positioned partner, or it might mean the agent cannot do the task on its own. The authors build a controlled environment where the human's marginal usefulness is set by geometry alone, measure each model's solo ability first, and then check whether co-play behavior tracks the human's actual advantage.

Key Contributions

1. A shared-control maze environment. An agent-controlled avatar and a human-controlled avatar share one target in a procedurally generated grid maze; either avatar reaching the target ends the episode, and only one avatar moves per team turn. The model chooses among three actions: MOVE_SELF(d), GUIDE_HUMAN(d) (hand the move over with a directional recommendation), and DEFER_TO_HUMAN (hand the move over with no recommendation), which separates using the human as an actor from ceding directional authority.

2. A construction-level manipulation of the human's comparative advantage. Initial avatar placement fixes Δ = d⁰_A − d⁰_H, the difference in the two shortest-path distances, so the agent route is shorter (Δ = −2), tied (Δ = 0), or the human route is shorter (Δ = +2). Because Δ is set through geometry alone, the maze, objective, and protocol are identical across conditions, and the oracle distances are never stated in the prompt. Start cells are sampled by exact BFS distance, so realized route lengths hit the requested distances exactly rather than approximately.

3. A solo capability baseline measured before co-play. Four models are run alone on 50 matched-seed 5×5 mazes (shortest path length 5, ten-turn horizon) using the same adjacency-list state representation and prompt family, giving a reference point against which co-play allocation can be interpreted.

4. Condition-specific allocation measures that go beyond aggregate involvement rates, including initial self-action rate (ISR), human-favored uptake (HFU = 1 − ISR(+2)), and self-action selectivity (SSC = NFSR − ISR(+2)), where NFSR averages self-action across the Δ = −2 and Δ = 0 conditions.

Main Findings

• A clear capability split in solo play. GPT 5.6 Sol finished every maze on the oracle path (S@7 = 1.00, S@10 = 1.00, E = 1.00, MDO = 1.00). Qwen 3.8 27B reached S@10 = 0.98 with MDO 0.98, and Command A+ reached S@10 = 1.00 with MDO 0.97. Command R completed only 22% of solo episodes with MDO 0.53, forming a substantially weaker comparison group.

• Near-ceiling models allocate by route advantage. Qwen 3.8 27B and GPT 5.6 Sol selected self-action on 100% of initial decisions when the routes were tied or the agent route was shorter, but involved the human on 82.5% of initial decisions when the human route was shorter. Command A+ showed the same shape but weaker: 97.5% self-action on tied routes, human involvement on 77.5% of human-favored decisions. Across models the human-favored involvement range was 77.5% to 82.5%.

• The weaker model shows no such selectivity. Command R's initial self-action rate was 0.700 (Δ = −2), 0.450 (Δ = 0), and 0.575 (Δ = +2) — lower when routes were tied than when the human was favored — giving an SSC of 0.000, versus 0.825 for Qwen 3.8 27B and GPT 5.6 Sol and 0.738 for Command A+.

• Whole-game behavior tells the same story. Qwen 3.8 27B, Command A+, and GPT 5.6 Sol shifted substantially toward guidance when the human avatar's current route was shorter. Command R stayed near 35–40% self-action and 60–65% guidance regardless of which avatar was closer, consistent with relatively indiscriminate involvement rather than response to comparative advantage.

• Involvement happens through guidance, not deference. No model selected deference on an initial decision in any condition, and whole-game deference shares were zero or near zero. When models involved the human, they kept a hand on direction selection via a recommendation.

• Team completion tracked capability, with one exception. Qwen 3.8 27B and GPT 5.6 Sol completed every episode by turn ten in all three conditions. Command A+ completed 97.5%, 100%, and 87.5% in the AI-favored, tied, and human-favored conditions. Command R completed 17.5%, 22.5%, and 45.0% — better when the human started closer, though the experiment assigns route advantage rather than involvement, so this is not a causal effect of involvement.

• The authors are explicit about the strength of the claim. The relationship between solo capability and selective allocation is associational across four models, not a demonstrated causal or continuous scaling relationship, and all comparisons are descriptive given four human participants.

Methodology in Plain English

The team built mazes as 5×5 grid graphs, generated with randomized depth-first search so every maze is a perfect maze with exactly one path between any two cells. The shared target sits at an endpoint of the maze diameter, and start cells are chosen so the two avatars begin at exactly known distances from it.

The model sees the full maze as a textual adjacency list plus both avatar positions and a running history of who did what; the human sees only a small local area around their own avatar and can follow or override any recommendation. The prompt never calls the partner a human, never states the oracle distances or Δ, and never discloses the remaining turn budget.

In the solo baseline, each model ran 50 matched-seed mazes with one avatar and full observability. In co-play, four of the paper's authors acted as human operators, each completing 120 episodes for 480 total. The design crossed model, participant, ten fixed maze seeds, and Δ ∈ {−2, 0, +2}, giving 40 episodes per model–advantage cell. The mazes have a shortest path of five moves, so the initial route pairs were (5,7), (5,5), and (7,5) for agent-favored, tied, and human-favored episodes, each with a ten-turn horizon.

The primary outcome is the first allocation decision, summarized by initial self-action, guidance, and deference rates. Secondary measures cover whole-game action composition, directional optimality, and team success and efficiency. The authors report 95% percentile intervals from 1,000 two-way cluster-bootstrap samples over participants and maze seeds.

Why This Matters

Impact on research. The paper argues that aggregate human-involvement rates are an uninterpretable metric on their own, and pairs two things that fix this: a solo capability baseline and a task where the partner's marginal value is controlled by construction. That recipe — measure what the agent does alone, then vary the partner's advantage geometrically — is portable to any shared task where the advantage can be manipulated, which gives collaboration evaluation a way to separate competence from deference.

Real-world applications:

• Agentic coding assistants deciding whether to write code themselves, suggest a fix to a developer, or hand the decision over entirely.
• Decision-support in high-stakes settings (medical triage, financial review) where the model should defer specifically when the human holds information or positioning it lacks.
• Human-in-the-loop robotics and shared autonomy, where control authority shifts between operator and system during a task.
• Workflow and office agents that must judge when a task genuinely needs a person versus when escalating is just noise.

Industry relevance. Teams deploying agents that can act autonomously need a defensible answer to "why did the agent stop asking for approval?" This paper suggests that question is only answerable alongside a capability measurement, and it offers a concrete harness for producing both numbers — relevant to anyone tuning escalation policies, approval thresholds, or handoff behavior in production agents.

Future Directions

• Broaden the model sample. With only four evaluated models, the capability–selectivity link is descriptive; a wider sweep, including intermediate-capability models, would test whether the relationship is graded or a threshold effect.
• Scale up and diversify the human side. Four participants drawn from the author list, all in one maze protocol, leave open whether the pattern holds across participant populations and interfaces.
• Move beyond navigation. The authors plan to test settings where the human's advantage comes from other sources, including cooperative game environments, everyday work tasks, and collaborative coding.
• Test causality directly. Because the experiment assigns route advantage rather than human involvement, the observed link between involvement and completion (notably Command R's improvement to 45.0% when the human started closer) is not identified as a causal effect.

Target Audience

Researchers in human–AI interaction and agent evaluation; practitioners designing delegation, escalation, or approval policies for LLM agents; and anyone who builds or audits shared-control systems and needs metrics that distinguish competent self-reliance from unhelpful handoff. Readers without an evaluation background can follow the maze framing, but the metric definitions (ISR, HFU, SSC, MDO, S@k) and the bootstrap design reward some familiarity with experimental methodology.

Authors’ abstract

As language-model agents grow more capable, they can carry more of a task without asking anyone for help. Falling reliance on a human partner is easy to read as a failure of collaboration when it may instead be a correct division of labor, and the rate of human involvement alone cannot tell the two apart. We study this systematically by placing an agent and a person under a shared objective while varying who holds the advantage: the human has a clear advantage, neither does, or the agent does. To obtain this control, we introduce a shared-control maze, in which an agent-controlled and a human-controlled avatar share the objective of reaching a common target in as few team turns as possible. The agent allocates each team turn between acting itself and handing the move to the person, and initial placement sets the human's shortest-path advantage while the task is otherwise unchanged. We evaluate GPT~5.6~Sol, Command~A+, Qwen~3.8~27B, and Command~R, first measuring solo capability with a single avatar and full-map observability, then running co-play with human participants across the three advantage conditions. Across the four models, stronger solo performance is associated with selective, route-conditioned human involvement: the models that solve the maze reliably on their own act for themselves almost without exception when the human holds no advantage, and involve the human on roughly $80\%$ of initial decisions when the human route is shorter. Command R, the model with substantially weaker solo performance, shows no such selectivity.