Loop-Back Authority in LLM Agent Teams: A Paired Experiment on Flat and Hierarchical Coordination

AI summary

Overview

• Research area: Multi-agent systems (LLM agent teams), organizational design, coordination, LLM-as-a-judge evaluation
• Technical level: Intermediate — readers should be comfortable with LLM agents, retrieval-augmented generation, and basic experimental statistics (paired t-tests, Cohen's d, mixed-effects models)
• Scope: A paired controlled experiment testing whether granting a Manager agent the authority to reject worker output and force revisions improves or degrades the quality of a five-agent LLM team on an open-ended business-intelligence reporting task.

What This Paper Is About

Most production multi-agent LLM frameworks default to a hierarchical setup where a Manager agent reviews worker output and can send it back for revision. Classical organizational theory predicts this authority speeds convergence on decisive output, but work on LLM sycophancy and Degeneration-of-Thought predicts the opposite. The paper isolates that single authority link in a controlled paired experiment — holding roles, prompts, tools, models, and data fixed — and finds that hierarchy hurts quality and costs more.

Key Contributions

1. A paired experimental design that isolates one authority link in an LLM agent team. Both organizational forms share the same five roles (Researcher, Analyst, Writer, Critic, Manager), prompts, tools, model pool, and data; the only difference is whether the Manager may reject output and oblige a revision (capped at two loops per run).

2. Evidence that the supervisory tier is pure cost in this setting. The hierarchical form consumes 51.5% more tokens, costs 40.5% more in generation and 20.2% more overall, and runs 34.3% slower — for no gain on any of the three quality metrics.

3. A mechanistic account locating the harm inside the revision loop. The hierarchical Writer's first draft is statistically indistinguishable from the flat report; the damage appears only after the Manager issues a loop-back directive, and it takes the form of hedging and padding rather than worse reasoning or content drift.

4. A hybrid evaluation protocol combining a five-model LLM judge panel (with a qualitative-first scoring protocol) and a deterministic specification check, with leave-one-judge-out re-estimation and judge-sensitivity reporting for every headline result.

Main Findings

• Flat beats hierarchical on Utility. Paired-sample effect d = 0.42, p = 0.009, robust to dropping any single judge. Strategic Depth (d = 0.48, p = 0.003) carries the effect and is the only per-dimension test surviving Holm correction across the six follow-ups. The flat form wins the Final Score in 28 of 43 products.

• Flat also beats hierarchical on Writing Clarity. d = 0.34, p = 0.030. The gap is carried entirely by Conciseness (flat 4.18 vs. hierarchical 3.98, d = 0.44, p = 0.007); Structure and Coherence do not differ. This result is judge-sensitive: it depends on Qwen, GLM, and Gemini, survives Holm correction but is borderline under a Wilcoxon test (p = 0.051).

• The reports are the same length — this is not a length effect. Flat and hierarchical reports average 1036 vs. 1046 words (p = 0.76). All effects hold when word-count difference is partialled out. The extra tokens are pipeline traffic, not report content.

• Hierarchical reports hedge 53% more. 5.03 vs. 3.30 hedges per 1000 words (d_z = 0.61, p < 0.001), spread across modal and evidential terms (possible, may, suggests, could) rather than one term. Lexical diversity is identical between forms (MTLD 186 vs. 189).

• The gap opens inside the revision loop, not before it. The hierarchical Writer's first draft matches the flat report on every measure (hedging d_z = 0.11, p = 0.46; length p = 0.20; lexical diversity p = 0.98). Within the 31 runs that revised, the second draft hedges 1.95 more per 1000 words (d_z = 0.83, p < 0.001) and is 38 words longer. Revision is not rewriting: the median revised draft keeps 88% of its predecessor as a common subsequence.

• Dose-response within the hierarchical arm. Writing Clarity falls monotonically with loop count (Spearman ρ = −0.40, p = 0.008); an OLS estimates −0.142 Writing Clarity points per loop (p < 0.001, R² = 0.41). This is correlational because the Manager chooses which drafts to send back.

• Unexercised authority does no measurable harm. In the 11 pairs where the Manager never intervened, the hierarchical report is numerically (but not significantly) higher than the flat one on all three metrics. The harm is a property of what the link does when exercised, not of the structure as such.

• Specification accuracy is at ceiling in both forms. On a repaired metric requiring exact hardware variants and scoring precision and recall over all listing specifications, mean recall and precision are 1.000 (flat) and 0.997 (hierarchical); 85 of 86 reports restate every listing spec correctly. The single error in the corpus is one hierarchical report naming a CPU model absent from the listing.

• Judge agreement is low but directionally reliable. Krippendorff's alpha is 0.08–0.17 for Writing Clarity and 0.22–0.30 for Utility, driven mainly by GPT-5.4's severity. Utility is robust to leave-one-judge-out; Writing Clarity is not.

• The mechanism is sycophantic compliance, not Degeneration-of-Thought. Told to revise, the Writer keeps the draft intact, softens its claims, and appends qualifying material. There is no drift into worse content, no vocabulary collapse, and no rewriting — the loop does not make the agent think worse, it makes the agent commit less.

Methodology in Plain English

The researchers built two versions of the same five-agent team using LangGraph, both operating over a shared blackboard (a running state that every agent reads and appends to). The team has a Researcher (retrieves product specs and customer reviews via two tools), an Analyst (finds patterns), a Writer (drafts the report), a Critic (checks claims against the vector store using a verify_claim tool), and a Manager.

The treatment is a single change to the Manager's prompt and permissions. In the flat form, the Manager is a peer dispatcher that enforces forward routing, reviews only the latest output, and may add non-binding commentary — its prompt explicitly states it has no authority to reject work or force revisions. In the hierarchical form, the Manager reviews the entire blackboard after every step and may issue a directive naming what must be fixed, injected into the target worker's context — an obligation, not a suggestion — capped at two loops per run. The authority link carries no sanction, so any compliance observed is the model's own.

Each agent role is filled by random assignment from a pool of five frontier models (GPT-5.4, Gemini-3.1-Pro, Qwen-3.5-122B, GLM-5, Mistral-Large-3), and the same role-to-model assignment is used in both forms for a given product, making model identity a within-pair constant. Temperature is 0 and endpoints are pinned.

The task is business-intelligence reporting on gaming laptops with RTX 4060 GPUs from Amazon US: 43 products with at least five reviews (540 reviews total, stratified by star rating) enter the paired analysis. Each product is run once under each form, giving 86 runs.

Reports are scored two ways. Subjective quality uses a five-model judge panel, each judging every report, with judges writing a qualitative analysis before assigning a 1–5 score. Writing Clarity averages Structure, Coherence, and Conciseness; Utility averages Actionability, Root Cause Analysis, and Strategic Depth. Objective quality uses a deterministic specification-extraction script (regex over GPU, RAM, CPU, storage), plus a repaired version requiring exact variant matching and scoring precision and recall. Analysis uses paired t-tests with Cohen's d, linear mixed-effects models with product as a random intercept, an OLS within the hierarchical arm for the loop dose-response, and a draft-level comparison of successive Writer drafts extracted from run logs.

Why This Matters

Impact on research. The paper gives the long-standing contingency claim of organizational design a mechanism specific to generative agents: for an RLHF-trained worker, an authority link is not a neutral control channel but also a conditioning signal, and unverifiable critique is treated as a social cue to comply with rather than information to act on. It also reconciles a contradictory literature — hierarchical advantage shows up on benchmarks with verifiable short answers (where a supervisor can cheaply check a candidate), flat advantage on open-ended synthesis (where it cannot) — and the paper falls exactly on that seam.

Real-world applications:

• Agent framework defaults. Teams building orchestrator/worker frameworks (MetaGPT, AutoGen-style systems) should reconsider whether the Manager holds loop-back authority by default, especially for open-ended synthesis.
• Enterprise report generation. Business-intelligence, market-research, and competitive-analysis pipelines that synthesize customer reviews and product data can cut token costs roughly in half by removing the supervisory review tier without losing quality.
• Quality-gating design. Revision should be triggered by a verifier flag (a failed claim check, a missing required section) rather than by managerial judgment of quality — turning the loop-back into a verification mechanism rather than an opinion mechanism.
• Cost planning for agent deployments. The 51.5% token overhead and 34.3% latency penalty give a concrete figure for what an oversight tier costs when it cannot verify anything.

Industry relevance. The finding directly challenges the practitioner intuition that oversight improves quality. As agent frameworks scale into production, the "supervisor who can only opine" pattern is common and, this paper argues, is a liability: a supervisory tier earns its cost when it can verify something and becomes a liability when it can only opine.

Future Directions

1. Make the revision trigger the manipulated variable. Compare verifier-fired loops (e.g., a failed verify_claim, a missing required section) against Manager-judged loops, single-pass review, and peer negotiation without authority. This would directly test the proposed design rule.

2. Introduce a sanction or a persistent stake for the worker. If compliance without deference disappears when the worker has something to lose, that would identify the boundary condition the classical theory's translation needs.

3. Measure deference from model internals rather than surface words. Concept probes on frozen models could separate calibrated uncertainty from evasion without relying on LLM judges, addressing the paper's own admitted weakness that its hedge lexicon cannot distinguish the two.

4. Extend the design rule beyond short-context synthesis. Open questions include whether long-context generation, code, tasks with richer verifiers, or the factuality of analytical claims (which this study did not measure) are equally insensitive to structure.

Target Audience

• Multi-agent systems researchers studying coordination topologies, orchestration, and agent team design
• Practitioners building LLM agent frameworks who need to decide whether to include a Manager with loop-back authority
• Organizational and coordination theorists interested in whether classical predictions transfer to generative agents
• LLM evaluation researchers working on LLM-as-a-judge reliability, sycophancy, and Degeneration-of-Thought
• Applied AI engineers designing report-generation, knowledge-synthesis, or quality-gating pipelines who want a concrete cost/quality trade-off for oversight tiers

The paper is most valuable to readers who already understand the multi-agent orchestration landscape and want a clean, controlled answer to a design question that is usually settled by intuition rather than evidence.

Authors’ abstract

Hierarchical orchestration, in which a Manager agent reviews worker output and can send it back for revision, is the default coordination pattern in production multi-agent LLM frameworks. Classical organizational theory predicts that the authority link speeds convergence on decisive output; work on sycophancy and Degeneration-of-Thought predicts that authoritative critique makes LLM output worse. Prior comparisons vary whole frameworks on tasks with checkable answers, leaving the authority link untested on open-ended work. We present a paired experiment that holds five LLM agents, their roles, prompts, tools, models, and data fixed and varies one link: whether the Manager may reject a worker's output and oblige a revision. Across 43 paired products and 86 runs of a business-intelligence reporting task, a five-model judge panel and a deterministic specification check score every report. The flat organization scores higher on Utility (d = 0.42, p = 0.009) and on Writing Clarity (d = 0.34, p = 0.030); the classical prediction fails. The reports are the same length, but hierarchical reports hedge 53% more, each revision loop is associated with a 0.14-point drop in Writing Clarity, and the hierarchical Writer's first draft is indistinguishable from the flat report: the gap opens inside the revision loop. Specification accuracy is at ceiling in both organizations, and the supervisory tier costs 51.5% more tokens for no quality gain. A supervisor pays for itself when it can verify and becomes a liability when it can only opine.