Agora: Git as Shared Memory for Collective AutoResearch

AI summary

Overview

Research area: Multi-agent AI systems and autonomous research agents, with a specific application to model weight transfer (initializing a language model from pretrained donors without training data).

Technical level: Advanced. The paper's institutional framing is accessible, but the winning method involves randomized SVD of a 50257 × 50257 bigram table, 96-dimensional hidden-state band routing, and sparse edits to attention, feed-forward, and state-space blocks.

Scope: The paper describes Agora, a Git-backed shared contribution graph for asynchronous human and AI researchers, and reports a nearly 12-day, 13-worker run on a weight-transfer task. Affiliation: NVIDIA.

What This Paper Is About

Research agents working in separate sessions cannot easily see what other agents have already tried, which results hold up, or where the frontier currently sits. The authors built Agora to store every contribution as an append-only Git commit with parent links, so a growing directed acyclic graph (DAG) preserves experiments, negative results, hypotheses, and verifications that outlive any single session. They then stress-tested it by letting 13 language-model workers, with no assigned tasks and no central planner, try to initialize a frozen hybrid language model from pretrained donor weights without training data or gradient updates.

Key Contributions

1. A Git-backed contribution DAG. Each contribution is a commit recording a result, insight, hypothesis, verification, or report, with explicit parent lineage, content-addressed artifacts, and an SQLite index that can be rebuilt from Git history. The prototype is a Go service with a CLI client and a Next.js web interface, 26 HTTP routes, and 15 CLI command groups.

2. A scoring and evidence scheme based on downstream reuse. A contribution's evidence score is the tag-weighted count of contributions by other accounts that build directly on it, with self-citation excluded and failed verifications excluded from descendant counts. Tag weights include setup +5, result +5, insight +5, hypothesis +5, report +5, verification +20/+10/−20, and endorsed/wip at 0.

3. Diversity-aware attention allocation. The system embeds contribution descriptions with Arctic-Embed 2.0 (568M parameters, 1,024-dimensional vectors), single-link clusters them at cosine similarity above 0.90, and reports top-cluster share and effective cluster count. Candidates are ranked by a diversity-aware upper-confidence bound with three presentation slots: exploit, explore known, and explore novel.

4. A documented collective run. 13 workers spanning 17 accounts produced 1,703 contributions, including 1,124 scored results, 284 insights, 203 hypotheses, 165 verifications, and one report, plus 165 reproductions across 95 distinct targets.

Main Findings

• Score reduction without training. The community's best transfer() initialized the frozen 119.6M-parameter target to 1.899044 bpb, against 3.3923 for random initialization and about 1.0 for a trained GPT-2 124M, closing 62% of that gap. No training corpus or gradient update on the target was used.

• The first day did almost all the work. The first 18 scored contributions account for about 98% of the total reduction, and the first eight improvements for roughly 70% of the total descent. The first scored attempt copied parameter slices from GPT-2 and Mamba into matching shapes and scored 4.6784, worse than random, and was published as a negative result. Thirty minutes later the same account reached 2.5151 with a unigram prior read from GPT-2 predictions, and within six hours four accounts had extended the idea to bigram statistics under 3, 6, 12, and 24 prefixes (1.9319).

• The winning recipe combines donor prediction statistics with sparse structural edits. Stage A queries six donors sharing the GPT-2 vocabulary (GPT-2 small and large, Cerebras-GPT 111M to 1.3B) over every vocabulary token under 28 contexts, blends their next-token log-softmaxes into a 50257 × 50257 bigram table, and factorizes it to rank d−1 = 671 with randomized SVD, producing the embedding and output head. Stage B re-enables sublayers with deterministic edits on 96-dimensional hidden-state bands.

• The recipe was assembled by many accounts, not one. The best contribution at cutoff has a 145-commit ancestry written by 15 of the 17 accounts; 115 of its 144 parent edges cross account boundaries. The method's code consists of 83 Python modules, each importing its parent and applying one change.

• Reproduction was broad and uniformly successful. Participants posted 165 verification contributions covering 95 distinct targets; each names its target, each verifier differs from the target's author, and none reports a failure. Same-hardware reproductions are bit-identical; A100 versus H100 runs differ by up to 1.3 × 10⁻³ bpb, inside the brief's tolerance. Forty of the winner's 144 scored ancestors were independently reproduced.

• The graph recorded failure as well as success. The window contains 53 contributions explicitly tagged as negative results. Doubling the prefix set to 48 made the recipe worse; flattening the singular-value spectrum, transplanting native Mamba blocks, copying GPT-2's embedding matrix directly, and building the prior from Pythia (a different tokenizer) all regressed and were published with scores.

• Search concentrated heavily on one lineage. At cutoff the graph has 1,703 nodes, 1,894 edges, 149 multi-parent nodes, and one component holding 98.9% of all nodes. Of 696 pairs of different accounts posting identical scores, 63% are within an hour of each other and 80% within six hours.

• One human intervention shifted exploration. On May 2, when more than a third of all activity belonged to a single semantic cluster and the leaderboard had stalled, the authors deployed the clustering, diversity summary, and diversity-aware UCB. On May 3 at 00:13 UTC a worker published the first SSM edit, scoring 1.9028 bpb, and selected milestones subsequently reached 1.8995 on May 5 and 1.8990 on May 8.

• The last gains are below measurement noise. The last recorded improvement of 9 × 10⁻⁶ bpb is below cross-hardware variation. The authors state they did not rerun the winning method; the primary result is archived evaluator output corroborated by the agents' cross-hardware reproductions.

• Causal claims about discovery efficiency are explicitly not made. The paper states that measuring the effect on discovery per unit of compute requires a matched comparison, and that establishing the effect on discovery efficiency requires matched comparisons across tasks.

Methodology in Plain English

The authors set up a shared record where every piece of work is a commit in a Git repository, and each commit points back to the commits it builds on. Because Git content-addresses commits and parent links cannot form loops, the result is an immutable, append-only DAG with provenance built in. A database layer turns that history into searchable views: recent activity, unverified claims, metric leaders, nodes with at most two descendants, and untouched leaves.

To keep workers from all piling onto the same best result, the server embeds each contribution's description into a 1,024-dimensional vector and groups similar descriptions into clusters once at least 50% of contributions have embeddings. It reports how concentrated activity is (top-cluster share, effective cluster count, evenness) and ranks candidate parents with a formula that balances quality percentile, an uncertainty bonus based on how few descendants a node has, and a term favoring descriptions that few others have used. Candidates are shown in three slots: refine the leader, extend a thin cluster, or inspect an untouched node.

For the research task, the authors assembled 141 open-weight donor models (534 GB) across 32 architecture families and defined a target hybrid that matches no donor's dimensions: 14 layers alternating attention and simplified Mamba-style state-space blocks, hidden size 672, seven attention heads, untied embeddings, 119,572,320 parameters. A submission is a Python file with a transfer(model, config) function; the evaluator seeds all random-number generators with 42, runs the function, and scores 200 FineWeb-Edu texts in non-overlapping 512-token chunks under the GPT-2 tokenizer. The loader raises an error if called from inside transfer(), and the rules forbid pretraining, fine-tuning, and editing the evaluator or target configuration. Random initialization scores 3.3923 bpb, and the project brief set an aspirational target below 2.5.

Each worker session ran in a container with one GPU, one Agora account credential, the Agora CLI, Git, PyTorch and Transformers, and read access to the donor zoo. The launcher gave each session a one-line prompt telling it to read the two-page brief and run agora analyze. Nothing named a method, assigned a role, or ranked participants. Sessions looped: analyze, pick a parent, check out that exact commit, make one change, evaluate, commit, push, post what was learned, and analyze again.

Why This Matters

Impact on research. Most multi-agent systems coordinate a team inside a single application, conversation, or episode. Agora targets asynchronous participants that share no conversation, manager, role graph, runtime, or filesystem, and preserves failed approaches and unverified claims alongside successes. The reported run shows that a shared record can carry provenance across 17 accounts and 145-commit ancestry, and it also shows the limits: visibility alone did not sustain broad exploration, and the authors had to add diversity views by hand to move workers into a neglected cluster.

Real-world applications (proposed directions rather than reported deployments):

• Distributed AI research automation, where independently scheduled agents contribute experiments to one durable record instead of isolated transcripts.
• Large-scale hyperparameter or architecture search where failures and negative results are as useful as successes.
• Reproducibility infrastructure for machine-learning claims, since each result carries code, artifacts, parent lineage, and independent verification verdicts.
• Weight or knowledge transfer between model families that share no architecture dimensions, avoiding a full pretraining or fine-tuning pass.

Industry relevance. The work is from NVIDIA and cites an ecosystem of agent frameworks, benchmarks, and evaluation suites. Its practical implications for industrial labs are an auditable record of what was tried, a stated gap between "a low-rank approximation of donor next-token behavior transferred successfully across architectures" and the initial parameter-copying attempt that performed worse than random, and a prototype design (Go service, SQLite index, Docker image, bearer-authenticated writes, rate limits) that is closer to deployable infrastructure than to a paper artifact.

Future Directions

1. A matched comparison to isolate discovery efficiency. The paper repeatedly states that the effect on discovery per unit of compute requires a matched comparison, and that establishing the effect on discovery efficiency requires matched comparisons across tasks. This is left open.

2. Independent testing of the community's own diagnoses. Several contributions converged near 1.90 bpb and attributed the plateau to a globally linear evaluator and underused target sublayers, and described the bigram recipe as a hard local optimum. The paper notes these explanations were proposed in the contributions but were not independently tested.

3. Verification above measurement noise. The last recorded improvement of 9 × 10⁻⁶ bpb is below cross-hardware variation (up to 1.3 × 10⁻³ bpb between A100 and H100). Determining how a shared record should treat improvements in this range is unresolved.

4. Scaling the institutional mechanisms. The run shows a narrow spine, short side branches, and heavy parallel rediscovery. Whether the diversity-aware views generalize beyond one project, and how the platform would behave across many tasks or with human participants mixed in, is not reported.

Target Audience

Researchers and engineers working on multi-agent LLM systems, autonomous research agents, and agent infrastructure; machine-learning practitioners interested in weight transfer and model initialization without training data; and teams building reproducibility, provenance, or experiment-tracking systems. Readers wanting only the platform design can follow the institutional framing; readers wanting to reproduce the method need the SVD, band-routing, and state-space details in Algorithm 1 and Table 3.

Authors’ abstract

Research agents working in separate sessions need to know what others have tried and which results they can build on. Agora stores their contributions as an append-only directed acyclic graph (DAG) in Git. Each commit records a result, insight, hypothesis, verification, or report and links it to prior work. Searchable views show leading results, neglected branches, and verification status; diversity-aware recommendations suggest experiments beyond the current leaders. We report a run of nearly 12 days in which 13 language-model workers, with no assigned tasks or central planner, used Agora to solve a weight-transfer problem. Given 141 pretrained donor models and a frozen 119.6M-parameter attention-SSM hybrid whose dimensions match no donor, the workers had to initialize the target without training data or gradient updates. They published 1,703 contributions and reduced the development evaluator score from 3.39 to 1.899 bits per byte, closing 62% of the gap to a trained GPT-2 124M. The best method compresses donor next-token statistics into the target's embedding and output head, then adds short-range context through sparse edits to attention, feed-forward, and state-space blocks. Its 145-commit ancestry spans 15 accounts. Participants also posted 165 independent reproductions across 95 targets, with no reported failures. The run documents how agents reused and verified shared work. Measuring the effect on discovery per unit of compute requires a matched comparison.