Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures

AI summary

Overview

Research area: Autonomous AI research agents and neural architecture search, applied to linear attention (efficient sequence-modeling) architectures. The work sits at the intersection of multi-agent system design, LLM-driven scientific discovery, and language model architecture research.

Technical level: Intermediate. The organizational design is explained in largely conceptual terms, but following the architectural results requires familiarity with linear attention, recurrent state, and standard language-model evaluation metrics (perplexity, downstream accuracy).

Scope: The paper studies how to organize a team of 16 persistent, discipline-grounded LLM research agents — with private histories, shared evidence, and adaptive allocation of a limited experiment budget — and deploys that team to discover a new linear attention architecture, Context-First DeltaNet (CFDN), validated at 150M and 1.3B parameters.

What This Paper Is About

Autonomous agents can now propose ideas, write code, and run experiments, but a single agent or a fixed pipeline tends to explore a narrow region of a design space and stall in a local minimum. The authors ask how a team of AI researchers with different scientific backgrounds can build on one another's findings and sustain progress when only a limited number of costly experiments can be run. They answer this by constructing such a team and using it to search for improved linear attention architectures, then testing which organizational choices actually drive the discoveries.

Key Contributions

1. A framework for organizing autonomous research teams under limited compute. The design combines (a) 16 researchers each initialized with a distinct disciplinary profile, (b) private longitudinal memory of each researcher's own investigations, (c) a shared record of proposals, reviews, results, and lessons synthesized every 10 iterations, and (d) adaptive researcher selection via discounted upper confidence bound (UCB).

2. Discovered architectures under matched training compute. The team found Context-First DeltaNet (CFDN), which adds a recurrent vector to GDN2's matrix memory and updates the vector before the matrix, evaluated at approximately 150M non-embedding parameters and then scaled to 1.3B parameters.

3. Controlled ablations linking organizational choices to outcomes. Separate runs remove disciplinary backgrounds, remove private memory, and replace adaptive selection with uniform selection, holding the agent model, iteration allowance, per-candidate training budget, and search feedback fixed.

4. A traced account of how the discovery emerged. The paper follows CFDN's development across four researchers, showing how principles from different fields were applied to refine one another's mechanisms, including how a degraded intermediate result redirected the design.

Main Findings

• The discovered architecture: CFDN. CFDN augments GDN2's recurrent matrix memory with a recurrent vector. The vector contributes the same context information to every query, while the matrix supplies a query-dependent prediction; the vector is updated first and the matrix then fits the remaining error, giving shared information priority over query-specific correction. The change adds no notable compute over GDN2 and remains compatible with efficient parallel training.

• Search-time gains over GDN2 at 150M. On the reduced search suite, CFDN improves all three search targets relative to GDN2: LM perplexity falls from 12.21 to 11.90, CSR accuracy rises from 44.48% to 44.63%, and retrieval rises from 28.91% to 50.00%. Its aggregated utility is 1.04 versus 0.00 for GDN2, −0.59 for KDA, and −1.29 for GDN, while using the fewest training tokens (13.53B versus 15.00B for GDN and 14.82B for KDA).

• Full 150M evaluation is mixed on common-sense reasoning. CFDN has the best WikiText perplexity (33.84) and LAMBADA perplexity (52.33) and the best NIAH accuracy (29.76), but its CSR accuracy (43.29) is slightly below GDN2 (43.60) and above GDN (42.88) and KDA (41.94).

• The architecture holds up at 1.3B. CFDN achieves the best result in every evaluation category: WikiText perplexity 15.74 versus 16.05 for both GDN2 and KDA and 16.27 for GDN; LAMBADA perplexity 11.04 versus 11.22 for GDN2; CSR 54.53 versus 52.65; NIAH 67.29 versus 66.86; and real-world retrieval 33.41 versus 32.52. All models were trained under the GDN-100B reference FLOP budget.

• Disciplinary backgrounds prevent an early plateau. The full system's best utility is 1.04, compared with 0.54 without backgrounds, 0.56 with fresh memory (no private histories), and 0.76 with uniform selection. The main run contains 63 evaluated candidates versus 65–68 in the variants, so the gap is not explained by running more experiments. The no-background run reaches its best utility at iteration 22 and never improves, whereas the main run's strongest proposal appears at iteration 68.

• Private memory and adaptive allocation both add value. Fresh-memory and uniform-selection runs fall short of the full system despite having access to the same shared proposals, results, and lessons, indicating that individual continuity and non-uniform opportunity allocation contribute beyond shared evidence.

• The search objective steers what gets found. In an independent LM-only search (same profiles, memory, selection rule, and budget), the resulting model reached WikiText perplexity 34.59, LAMBADA 46.40, CSR 42.25, and NIAH 26.73, versus 33.84 / 52.33 / 43.29 / 29.76 for the three-target search. The LM-only model is substantially better on LAMBADA but weaker elsewhere, suggesting LM-only objectives may not transfer to all capabilities of interest.

• Discovery unfolded through cross-disciplinary revision. A researcher grounded in linear attention introduced the recurrent context vector, building on affine-memory precedent; another learned the address origin determining what the vector represents; a researcher with adaptive-filtering expertise normalized the correction by the translated address's energy. A later revision updated the matrix before the context vector — improving LM perplexity but weakening CSR and retrieval — and a researcher with a predictive-representations background used that tradeoff to reverse the update order, producing CFDN.

Methodology in Plain English

The team. Sixteen agents, all instances of the same language model (GPT-6 Astra with high reasoning effort, run through the Codex harness), share the same tools and the same description of the problem, baseline, and constraints. They differ in a disciplinary profile that supplies prior knowledge and mathematical tools, scientific insights (when those tools help and where intuitions fail), and research habits (how to decompose a problem, what to inspect, what would count as a counterexample). Example profiles include Bayesian filtering, dynamical systems, and streaming algorithms.

Two kinds of memory. Each researcher keeps its own session history and private notes across investigations, so it can revisit unresolved questions and revise earlier assumptions. Everyone also sees a shared record of proposals, reviews, experimental results, and conclusions, with accumulated findings synthesized into shared lessons and unresolved questions every 10 iterations.

One iteration, six steps. (1) Select a researcher using discounted UCB, which balances recent empirical contributions against opportunities for less-tested researchers; in the first 16 iterations every researcher is invoked exactly once. (2) The selected researcher proposes a hypothesis stating what to change, why it should help, and which outcome would challenge it, specifying the mathematical operations, parameterization, and a trainable parallel formulation. (3) Two independent reviewers check the proposal — first technical soundness and computational feasibility, then novelty and research value. (4) The author refines the design in response to critiques, or withdraws if the central mechanism cannot be saved. (5) A separate implementation agent writes the code and passes integration and speed checks, with per-step training time required to be under 1.2 times GDN2's. (6) The author reflects on the outcome, updates private notes, and shares generalizable lessons.

The reward. Every trained model is scored by a utility that averages normalized improvements over three metrics: language modeling (negative log-likelihood), common-sense reasoning (mean accuracy over eight datasets), and retrieval (a needle-in-a-haystack exact-continuation proxy). An iteration earns reward equal to its utility gain over the best model known when that iteration began, floored at zero, so failures are not penalized and do not discourage exploration.

The experiments. All models train from scratch on FineWeb-Edu with token counts adjusted to match the total FLOPs of training GDN (15B tokens at 150M scale, 100B at 1.3B scale), so more expensive architectures receive fewer tokens. The search space lets candidates change the recurrent mixer's memory representation, update rule, readout, and parameterization while retaining the common depth, residual width, and feed-forward network. The search run used a budget of 80 iterations with at most 8 researchers working in parallel, with the proposal stage forced to be sequential to prevent repetition.

Why This Matters

For research on AI-driven discovery, the paper argues that progress depends not just on stronger individual agents but on organizational design — who explores, what they remember, and how effort is allocated. The controlled ablations give direct evidence that disciplinary diversity, private memory, and adaptive selection each contribute, and that removing any one of them degrades the best discovery. The failure mode identified in the no-background run — plateauing at iteration 22 — is a concrete demonstration of the local-minimum problem that motivates heterogeneous teams.

For architecture research, the paper contributes CFDN, a mechanism that improves over GDN2 across two scales with no notable added compute, plus a methodological point: optimizing only for language-model quality produces architectures that do not transfer uniformly to retrieval and reasoning, which matters for systems guided by a single LM metric.

Potential real-world applications (not claims evaluated in this paper):

• Long-context assistants and document analysis, where retaining and retrieving information from context is the binding constraint.
• Retrieval-augmented and memory-intensive systems that need a fixed-size state rather than a growing key-value cache.
• Efficient inference on edge or on-device hardware, where linear-time sequence processing matters.
• Accelerating architecture exploration in industry labs where each training run is expensive and the number of ideas that can be tested is small.

Industry relevance: the design is a template for running many LLM agents against a fixed compute budget with an explicit exploration/exploitation policy, an enforced efficiency constraint on proposals, and a shared evidence trail — directly applicable to settings where automated search must operate under real training costs rather than unlimited trial.

Future Directions

• Extending beyond linear attention. The organizational design is demonstrated on one search space; whether the same combination of disciplinary profiles, private memory, and adaptive allocation transfers to other open-ended problems (optimizers, data mixtures, training recipes) is untested here.

• Budget scaling and cost. The team used 80 iterations and 63 evaluated candidates at 150M scale; how the design behaves at much larger or much smaller budgets, and whether the UCB exploration constant and discount factor need retuning, is not established.

• Guardrails against metric gaming. The ethics statement acknowledges that agents may generate incorrect claims, overlook prior work, or optimize evaluation metrics without improving broader capabilities. Stronger novelty checks and broader capability evaluation are open problems.

• Recovering from stalled search. The no-background run plateaued early while the full system kept improving to iteration 68; understanding which specific cross-disciplinary interactions produced late-stage gains, and whether they can be deliberately scheduled rather than left to adaptive selection, remains open.

Target Audience

Researchers and engineers working on autonomous research agents, multi-agent LLM systems, or LLM-driven architecture search will find the organizational ablations most directly useful. Architecture researchers interested in linear attention and recurrent memory design will benefit from the CFDN mechanism and its two-scale evaluation. Practitioners who run expensive training experiments and want to allocate a limited budget across many candidate ideas will find the selection and reward design practically relevant. Some background in language modeling and evaluation metrics is assumed for the results sections.

Authors’ abstract

Autonomous agents can increasingly propose ideas, modify code, and run experiments, creating an opportunity to accelerate scientific discovery using teams of collaborative AI researchers. To escape local minima in the discovery, we propose to encourage orthogonal exploration by constructing the team using agents with varying expertise. However, realizing this design requires more than diverse backgrounds. When experiments are costly, a team can test only a limited number of ideas, thereby creating the need to balance exploration and exploitation when selecting a subset of researchers to propose and execute new experiments. Furthermore, each result should shape what the team tries next. We therefore ask how AI researchers with different scientific backgrounds can build on one another's findings and sustain progress under a limited experimental budget. To address these challenges, we build a team of persistent AI researchers, each grounded in a different discipline, who keep private research histories, share experimental evidence, and receive research opportunities through adaptive allocation. We demonstrate the effectiveness of this design by deploying the team to search for linear attention architectures, where it outperforms GDN2 at 1.3B scale on language modeling, common-sense reasoning, and synthetic and real-world retrieval tasks. In controlled comparisons, removing any single organizational component from our design yields markedly weaker discoveries: without disciplinary backgrounds, the search plateaus early; without private histories, researchers make less progress despite identical shared evidence; and uniform selection falls short of adaptive allocation within the same budget. These results suggest that sustained autonomous discovery depends not only on stronger agents, but also on who explores, what they remember, and how research effort is allocated.