The Weight of Teamwork: Does Communication Structure Help LLM Scientists Search for Scientific Laws?

AI summary

Overview

Research area: Multi-agent LLM systems, communication topology, and AI-driven scientific discovery, evaluated in a symbolic-regression testbed.

Technical level: Intermediate. The design is conceptually simple (same budget, different communication policies), but the paper's argument depends on paired bootstrap intervals, Holm-corrected Wilcoxon tests, and the distinction between simple effects and interactions.

Scope in one sentence: Using LLM-SRBench symbolic regression with Qwen and Terra model layers, the paper asks whether peer-visibility and update-timing policies change the quality of the expressions an LLM team generates and the expression its selection rule ultimately returns — and finds that the answer depends on which of those two things you measure.

What This Paper Is About

Team performance is usually reported as one number, but that number comes from two stages: the system generates a pool of candidate answers, and a selection rule picks one of them. This paper asks whether the communication structure of an LLM "team of scientists" (who can see whose proposals, and when) actually improves the quality of the hypotheses generated, or only appears to because of how the final answer is chosen. The authors use symbolic regression — where every candidate equation can be logged and objectively scored — as an observable stand-in for scientific law discovery.

Key Contributions

1. Distributing a fixed call budget across more independent trajectories produced the largest measured gain in the testbed. Splitting 50 scheduled calls per problem across ten independent Qwen trajectories instead of one raised mean held-out Q by 0.70 (95% CI [0.41, 1.03]) — larger than any communication-policy effect reported.

2. The implemented serial Ring policy underperformed Independent under a common archive-wide selector, while Star and Complete remained unresolved. Ring reduced mean held-out Q by 0.33 (95% CI [−0.49, −0.18]) in Qwen and by 1.18 on the 41-problem Terra Transform layer; Star and Complete intervals spanned zero.

3. On frozen candidate archives, the measured gap between architectures changes with the rule used to return a final expression. Complete minus Independent was +0.10 under the common archive-wide rule but −0.75 under final-round selection — a change in the sign of the point estimate — and 9 of 20 prespecified architecture-by-delivery interactions survived Holm correction in Qwen (0 of 20 in Terra).

4. A reporting-practice argument. The authors argue that architecture rankings are not identified by communication structure alone, and recommend that papers state the selection rule, report the generated archive and the returned expression as separate endpoints, and either hold the selector fixed or cross architecture with selector.

Main Findings

• Ring loses under the common selector (Qwen). Mean held-out Q difference versus Independent was −0.33 (95% CI [−0.49, −0.18]; Holm p = 5.75 × 10⁻⁷), with median −0.01 and 21 wins / 75 losses / 4 ties out of 100 problems.

• Star and Complete are unresolved, not equivalent (Qwen). Star's mean difference was +0.03 ([−0.08, +0.13]; p = 0.91), median −5.64 × 10⁻⁷, 43/50/7. Complete's was +0.10 ([−0.08, +0.34]; p = 0.36), median +1.73 × 10⁻⁵, 53/42/5. The authors explicitly say the intervals allow both benefit and harm and do not establish equivalence.

• Ring's disadvantage holds within each Qwen task family. Mean differences were −0.30 on Transform ([−0.48, −0.14]) and −0.35 on Synth ([−0.63, −0.12]). Star's family-specific means were +0.11 and −0.06 (Holm p = 0.29 and 0.40); Complete's were +0.23 and −0.04 (p = 0.21 and 0.84).

• Terra shows a narrower, differently shaped result. Ring's mean difference was −1.18 ([−2.04, −0.48]; Holm p = 1.63 × 10⁻³). Star's mean was −0.57 with a zero-spanning interval ([−1.56, +0.40]) despite a Holm p of 0.03 on the signed-rank test; Complete's mean was −0.36 ([−1.43, +0.73]; p = 0.08). All three medians were zero, with 20–24 ties among 41 problems.

• More independent trajectories, more gain. At 50 calls per problem, N = 1 → 10 raised mean Q by 0.70 (median gain 0.11; 86/11/3; Holm p = 4.41 × 10⁻¹³). The N = 1 → 5 reference gave +0.43. Every adjacent step had a positive mean interval: +0.23 (1 → 2), +0.20 (2 → 5), +0.26 (5 → 10), with median gains of 6.30 × 10⁻⁵, 4.70 × 10⁻³, and 5.20 × 10⁻⁴.

• Mean gains are right-skewed. The mean–median gaps (e.g., 0.70 versus 0.11 for 1 → 10) indicate that large gains on relatively few problems drive the average, while typical per-problem gains are small.

• Paired gains do not mean good absolute fitting. Mean Q at N = 10 was +0.17 with a 95% interval spanning zero ([−0.08, +0.49]); 38/100 problems beat the constant-mean predictor at N = 10 versus 19/100 at N = 5, and the N = 10 median was −1.70 × 10⁻³.

• Allocation gains extend across task families and models. N = 1 → 5 gave +0.35 on Qwen Transform ([0.16, 0.56]) and +0.52 on Qwen Synth ([0.26, 0.84]; 44/6/0). Terra Transform gave +1.04 ([0.06, 2.07]) with median zero and 20 ties among 41 problems. The paired Terra Synth comparison was not completed.

• Selection rule changes the measured gap. For Complete minus Independent: d1 (archive-wide minimum training error) +0.10; d2 (final-round minimum training error) −0.75 ([−1.56, −0.25]); d3 (final-round plurality) −0.78 ([−1.59, −0.28]); d4 (archive-wide plurality) −0.59 ([−1.40, −0.10]). The d2 interaction had mean −0.84 ([−1.85, −0.26]; Holm p = 1.05 × 10⁻⁷). Because the d1 interval spans zero, the authors classify d2/d3 as point-estimate sign changes rather than confirmed reversals.

• Grouping sensitivity was checked. Complete's d3 interaction had Holm p = 6.14 × 10⁻⁸ with frozen skeletons and 8.10 × 10⁻⁸ under alternative functional canonicalization; its d4 interaction failed Holm correction under both conventions (p = 0.07 and 0.16). The d2 result is grouping-independent because that selector does not group expressions.

• Team-level concentration is not the same as within-agent repetition. A modal expression skeleton accounted for at least 80% of a problem's outputs in 81 Solo cases, 72 Ring, 56 Star, 46 Complete, and zero Independent cases; the agent-level proportion was 81% for Solo and 81.80% for Independent. Median distinct skeletons per problem were 2 (Solo), 2 (Ring), 5 (Star), 7 (Complete), and 8.50 (Independent) — with 3 at N = 2 and 15.50 at N = 10.

• Ring generated the fewest invalid candidates. Among five-agent conditions, Ring had 1 invalid candidate slot out of 5,000, versus 61 for Independent, 43 for Star, and 36 for Complete.

• A post-hoc diversity account did not hold up. Solo's best-so-far Q improved by 0.01 between calls 10 and 50, but an analysis relating within-problem changes in skeleton counts to changes in delivered Q found no consistent association.

Methodology in Plain English

The authors set up a controlled bake-off where an LLM team tries to infer a compact Python expression matching a hidden target from five training rows, and every candidate equation is logged and scored.

The testbed. They use LLM-SRBench. The Qwen layer contains 100 problems (50 LSR-Transform + 50 LSR-Synth); the Terra layer uses the common 41-problem Transform intersection. Evaluation uses the benchmark's official pregenerated held-out ID arrays, which the agents never see.

The metric. Normalized mean squared error (NMSE) converted to Q = −log₁₀(NMSE + 10⁻¹⁰). A constant-mean predictor gives NMSE = 1 and Q ≈ 0 for nondegenerate targets, so positive Q means beating that baseline.

The budget. Every condition gets 50 scheduled model calls per problem. The communication comparison fixes five agents and ten rounds; the allocation comparison varies (N, T) over (1, 50), (2, 25), (5, 10), and (10, 5).

The communication policies. Independent sees no peers. Ring reads the latest predecessor response in rotating serial order. Star has leaves read the latest hub response before the hub reads current leaf responses. Complete reads a previous-round snapshot from all four peers. Solo and Independent have self-history only. Crucially, the prompts anonymize colleagues and never name the topology, and the policies vary update timing alongside visibility — so graph connectivity alone does not define the treatment.

Two endpoints. First, a common rule applied to everyone: pick the valid expression with lowest training NMSE across the whole archive. Second, a crossed-delivery analysis that holds the frozen archives fixed and applies six prespecified selectors (archive minimum training error, final-round minimum training error, final-round plurality, archive-wide plurality, last message, and a complexity-penalized rule using NMSE + 0.05 × nodes/nodes_max).

The statistics. Primary estimates are mean paired differences with 95% percentile bootstrap intervals from 10,000 resamples of problems (seed 20260904), accompanied by medians, wins/losses/ties, signed-rank rank-biserial correlations, and Wilcoxon tests, with Holm correction within prespecified families (three pooled communication contrasts, five allocation contrasts, or 20 delivery interactions).

Why This Matters

Impact on research. The paper's core claim is methodological rather than architectural: a benchmark score for a multi-agent system conflates what the team generated with how the answer was picked. If the sign of an architecture comparison can flip when the selection rule changes, then reported rankings of communication topologies are not identified by topology alone. The paper also offers an unusually clean negative result — the paper's own exploratory hypothesis about diversity and repetition did not survive its post-hoc analysis.

Real-world applications (implications, not claims tested in the paper):

• Automated scientific discovery pipelines that propose and refine equations or hypotheses — the paper's finding argues for instrumenting such systems to log the full candidate pool, not just the returned answer.
• LLM agent orchestration frameworks deciding whether to run many cheap parallel attempts or a smaller number of chatty, peer-aware agents under a fixed API budget.
• Benchmark and evaluation design for agentic systems, where leaderboards currently compare architectures without fixing the answer-selection rule.
• Cost-aware deployment of LLM teams, since the strongest measured lever was simply spreading the same number of calls over more independent runs.

Industry relevance. The practical takeaway for teams building multi-agent LLM products is that peer visibility is not a free win: the implemented Ring policy measurably hurt under a shared selector, and Star and Complete showed no clear additional gain over Independent at N = 5. The allocation result — more independent trajectories, more gain — is cheap to adopt and does not require new communication infrastructure.

Future Directions

• Separate topology from update timing. The authors propose testing parallel scheduling under the same Ring adjacency, since the current policies vary visibility and timing together and therefore cannot isolate connectivity.
• Test richer messages. Peer messages currently carry expressions and discussion; the authors suggest testing peer messages that include explicit evaluations of candidate expressions.
• Extend to stronger backbones and broader task families, and complete the paired Terra Synth comparison that was left unfinished.
• Repeat conditions across independent samples to quantify run-to-run sampling variation, which the problem-level bootstrap intervals do not measure. Relatedly, a mechanism-level test would need to link exposure to a peer message with subsequent candidate changes and their quality.

Target Audience

Researchers and practitioners working on multi-agent LLM systems, automated scientific discovery, and agent benchmarking will get the most from this paper, particularly those who design evaluations where a "team score" is reported as a single number. It is also useful for statistically literate engineers choosing between parallel sampling and peer-communicating agents under a fixed API budget. Readers looking for a new solver or a recommended topology will not find one here — the authors are explicit that the study identifies neither a communication mechanism nor a universally preferred architecture.

Authors’ abstract

Team performance is often reported as one delivered score, even though that score is produced in two stages: a system generates candidate answers and a selection rule returns one of them. We use symbolic regression as an observable scientific-search setting and compare fixed-budget LLM teams that differ in peer visibility and update timing, evaluating both their frozen candidate archives and the expressions selected from those archives. Distributing the call budget across more independent trajectories produced the largest measured gain, improving mean held-out fitting score Q by 0.70 at the widest tested allocation. Under a common archive-wide selector, the implemented Ring policy reduced mean Q by 0.33 relative to Independent, while Star and Complete remained unresolved. Holding the generated archives fixed, changing the delivery rule changed the measured architecture gaps, including the sign of the Complete point estimate. These results support reporting candidate generation and final selection as separate endpoints. They do not identify a communication mechanism or a universally preferred topology; rather, they show that an architecture ranking is not identified by communication structure alone when the rule that returns the final answer is left unspecified.