SplatLab: Autonomous Scientific Discovery for 3DGS Research

AI summary

Overview

Research area: Autonomous scientific discovery systems applied to 3D Gaussian Splatting (3DGS), a subfield of neural rendering and computer vision.

Technical level: Advanced. The paper assumes familiarity with 3DGS training pipelines (densification, opacity reset, spherical harmonics), LLM-agent orchestration, Elo-style ranking, and evolutionary program search.

Scope: SplatLab is an end-to-end system that takes a research prompt, a target-paper URL, or no input at all, reproduces a published 3DGS baseline, proposes and trains an extension, ablates it, and writes a manuscript in which every reported number is bound to a logged GPU run.

What This Paper Is About

Reproducing a published 3DGS paper takes weeks of expert engineering before any new extension can even be proposed, and prior autonomous-research systems have never been demonstrated end-to-end in a live computer-vision subfield with this much activity. The authors argue that the missing ingredients are a domain framework, a verified baseline, full-loop iterative refinement, and a binding from every written claim back to a measurement on disk. SplatLab is built to supply all four for 3DGS, turning the read–reproduce–extend–train–ablate–write cycle from months of graduate-student time into days of mostly autonomous compute.

Key Contributions

1. A verified-reproduction backbone. Every candidate idea must mutate a splatify plugin (built on nerfify) that has already been verified to compile, train, and match its target paper's published headline number, so any reported gain has a fixed external reference rather than an unverified LLM reimplementation.
2. A cascade-weighted, grounding-conditional Elo tournament across eight islands. Ideas and code variants are ranked on measured GPU PSNR from real ns-train runs, conditioned on a (benchmark ID, loss ID) pair, with ReEvo-style verbal-gradient feedback routed back to the ideator and five mutation operators chosen by a multi-armed bandit.
3. A standing falsifier coupled to selection. An in-loop adversarial falsifier attacks every survivor on edge scenes, sparse-view reduction, and hyperparameter perturbation; each failed attack subtracts 30 Elo, so stress-test survival becomes a selection signal rather than a post-hoc check, and the falsifier holds veto power before writing.
4. Claim-backed verification and a domain-aligned modification space. Six integrity audits (Provenance, Faithfulness, Novelty, Compute match, Cross-codebase port, Claim check) bind every number to an (experiment ID, commit, config) triple, and the run cannot terminate while any claim is unbacked; meanwhile a seven-category 3DGS taxonomy wraps EVOLVE-BLOCK markers in scaffolds that a cross-codebase auto-mapper can place in arbitrary 3DGS repositories.

Main Findings

• Consistent PSNR gains over reproduced targets: across the reported pairings, discovered methods beat their baselines by +0.18 to +3.78 dB PSNR, with the largest gain on astrophotographic nebula rendering.
• Nebula (no prior 3DGS method exists): vanilla 3DGS reaches 32.58 PSNR / 0.914 SSIM / 0.278 LPIPS; the discovered Nebula3DGS-v2 (a 3DGS-MCMC sampler with a saturation-driven star seeder) reaches 36.36 / 0.945 / 0.201.
• Bubble: 3DGS at 27.40 / 0.857 / 0.337 versus BubbleGS at 30.11 / 0.906 / 0.224, using a Snell/Fresnel two-pass conditional shell renderer.
• Mip-NeRF 360, FastGS target: 29.12 to 29.69 PSNR (TileFocalGS, rank-graded per-tile weights with EMA smoothing); SSIM 0.858 → 0.869, LPIPS 0.217 → 0.194.
• Mip-NeRF 360, 50-second wall-clock target: splatfacto at 20.53 versus View-Curriculum-GS at 23.36, by training the first 15 s on the 30% most pose-similar views.
• Mip-NeRF 360, MSAA target: 25.35 to 26.19 (MSAA-Bilat, bilateral grid contrast recovery plus 15K extra iterations).
• LLFF, DWTGS target: 18.68 to 21.09 (DWTGS-Patch, adding patch structure loss and opacity/normal/depth regularisers).
• Multi-seed robustness: GNS averaged 25.95 ± 0.07 dB and GNS-AA 26.18 ± 0.05 dB, a +0.23 ± 0.09 dB gain versus the originally reported +0.18 dB; gains inside the estimated noise floor are labeled marginal rather than verified.
• Run outcomes: 16 runs produced 198 experiment branches, yielding five verified positives, five marginal results with qualified claims, five non-PSNR contributions, and one recorded failure.
• Compute cost: 24–48-hour wall-clock budgets, averaging 113M API tokens over roughly 1,000–1,500 LLM calls, 2–36 GPU-hours per run, and an estimated $150–250 in LLM API costs.
• Ablations isolate each mechanism: removing the verified-plugin contract makes 9/16 winners non-reproducible (greater than 1.0 dB error) versus 0/16; removing the claim-backed audit gate leaves 16/16 manuscripts with unbacked numbers versus 0/16; removing the falsifier lets 5/16 winners fail held-out attacks versus 0/16; removing verbal gradients takes 5 generations and 2.4× the tokens versus 2 generations; removing per-(B,L) Elo costs −1.6 dB on Nebula versus a specialist.
• Elo pre-selection saves GPU time: 24 GPU evaluations to reach a winner without it versus 6 with it (20 ideas rejected pre-GPU).
• Reviewer re-experimentation raises simulated scores: average simulated acceptance of 2.2/6 without it versus 3.6/6 with it.
• Idea quality near human level: under the blind expert protocol with 10 experts scoring 40 ideas on 1–10 Likert, SplatLab's 20 Stage-2 ideas scored 5.9 novelty, 5.6 excitement, 6.2 feasibility, 6.1 effectiveness, 5.8 overall — versus PhD students at 5.2 / 5.0 / 6.4 / 5.7 / 5.1 and accepted preprints at 5.8 / 5.5 / 6.0 / 5.8 / 6.0.
• Manuscript quality close to human ground truth: on the 0–100 multi-agent judge, the published human paper scored 93 overall, SplatLab 90, PaperOrchestra 79, SingleAgent 38, AI Scientist-v2 32, and GPT-5 ET 17; human re-scoring agreed with the LLM at Spearman ρ = 0.87 (weighted κ = 0.84), with 54/60 rationales marked supported by the manuscript text.
• Peer review splits by judge type: the multi-agent reviewer favored SplatLab slightly (3.6 vs. 3.5 overall on the 1–6 scale), while the blind human reviewer gave published papers 4.1 overall (clear accept) and SplatLab 3.5 (borderline), with SplatLab higher only on Novelty (3.5 vs. 3.1).
• Real acceptance: three generated manuscripts — AttDis & Friends, NeedleSwap, and AuditSplat — were accepted as posters at a CVPR 2026 workshop.
• Taxonomy helps efficiency, not executability: on the nebula target, taxonomy-guided and taxonomy-free runs gained +3.78 and +3.45 dB respectively, while removing the taxonomy raised token use from 47.5M to 85.5M (80%).
• Context for the gap: PaperBench reports the strongest evaluated agent at 21.0% average replication across 20 ICML 2024 Spotlight and Oral papers, and on a 30-paper 3DGS reproduction benchmark frontier models succeed on roughly 5% of inputs.

Methodology in Plain English

The system runs as a six-stage pipeline under a three-level hierarchy of 36 agents (Professor, PhD coordinators, and undergraduate leaf agents) with no fixed dependency graph, so the master can revisit earlier stages on a falsifier veto, audit failure, or a reviewer's demand for a new experiment.

1. Grounding. One thread reads the literature via arXiv, Semantic Scholar, and Google Scholar plus a persistent cross-run paper store, and distills deduplicated limitation strings (L-IDs) with citation sets. A parallel thread trains the verified plugin on representative scenes and profiles loss, PSNR trajectory, memory, and gradient norms to produce measurement-anchored bottleneck strings (B-IDs). These form the thesis spine that everything downstream conditions on.
2. Ideation without training. An Ideator proposes candidates, each naming a mechanism, one of seven 3DGS novelty categories (or "open"), an L-ID, a B-ID, and a predicted delta. Candidates must pass precedent, novelty, saturation, and orthogonality checks, then enter an Elo bracket (initial rating 1200) judged on novelty, specificity, implementability, and delta plausibility.
3. Implementation. Each surviving idea is compiled as a structured diff against the verified plugin, restricted by a context-free grammar and guided by a Graph-of-Thoughts controller, then smoke-tested for 100 iterations. Build failures return to the implementer with the log as feedback.
4. Tournament. Candidates climb four rungs — smoke, probe, multi-scene, full benchmark — with Elo updates weighted 0.2, 0.5, and 1.0 so a cheap smoke win cannot drown out a multi-scene loss (K = 32). Survivors spread over eight islands, evolve via Refine, Crossover, Single-Parent, Tweak, and Rewrite, and are promoted on Pareto improvement over (B, L) rather than scalar dominance.
5. Audit. Six independent checks run between champion and draft; any failure aborts the run and it is archived as a research log rather than a paper.
6. Writing. Five writers draft in parallel, a VLM checks figures, and three reviewer agents under an Editor can demand new training runs, re-entering Stage 4 with the comment as a verbal-gradient seed; the loop ends only when no reviewer raises a new-experiment concern.

Experiments use Mip-NeRF 360 (nine unbounded scenes) and LLFF, hold out every 8th image for testing, and train for 30k iterations on NVIDIA A6000 GPUs. Three out-of-distribution scenes captured for this work — nebula, bubble, and diamond — test discovery where no prior 3DGS method exists.

Why This Matters

Impact on research. The paper reframes the bottleneck on autonomous research as missing infrastructure rather than missing model capability: once a verified baseline, a structured modification space, full-loop refinement, and claim-to-measurement binding are in place, the discovery rate is set by GPU throughput rather than researcher time. The claim-backed audit is the most consequential piece — without it, 16/16 ablated manuscripts contained numbers that traced to no log on disk, which closes the hallucination failure mode that earlier AI Scientist-style drafts leave open.

Real-world applications:

• Astrophotographic and scientific imaging, where SplatLab's nebula result (+3.78 dB) addresses a regime 3DGS had not previously been applied to.
• Rapid prototyping on top of published baselines for small labs that cannot afford weeks of reimplementation engineering.
• Automated reproduction and stress-testing of existing methods, since every candidate is anchored to a reproduced target and attacked by a standing falsifier.
• Transparent, ledger-backed technical writing, where each reported figure carries its experiment ID, commit, and config.

Industry relevance. Teams maintaining 3D reconstruction, AR/VR capture, or neural-rendering pipelines can treat SplatLab as a design-space sweep tool that runs at 2–36 GPU-hours and $150–250 in API cost per run. The portability machinery — a cross-codebase auto-mapper plus a port audit that flags codebase-specific gains — is directly relevant to organizations with proprietary 3DGS codebases. The authors are explicit that this is a research instrument, not a paper-mill backend, and recommend derivative submissions be labelled and ship their experiment ledger.

Future Directions

• Extending the verified-reproduction backbone beyond 3DGS. The authors note that per-field backbone cost biases capability toward pre-engineered subfields and well-resourced labs; how cheaply a new subfield's backbone can be built is left open.
• Reducing the taxonomy's expert cost. Stage 2's taxonomy requires expert effort to adapt, and the single-target nebula comparison (+3.78 vs. +3.45 dB, 47.5M vs. 85.5M tokens) is the only evidence on its value — broader study of taxonomy-free operation is flagged for the Appendix.
• Closing the writing-quality gap. Human reviewers scored SplatLab lower on Soundness (3.2 vs. 3.6) and Presentation (3.0 vs. 3.8) than published papers, so improving prose and presentation without losing the novelty advantage remains open.
• Community norms for volume and attribution. The authors call for labelling, ledger release, and discouraging volume submissions, and suggest AI research mentors and Mode A logs as teaching material to offset reduced entry-level research opportunities.

Target Audience

Researchers and engineers working on 3D Gaussian Splatting and neural rendering who want to understand what an automated extension of their own baseline would look like; builders of LLM-agent systems and autonomous-research pipelines interested in verification, grounding, and claim-to-log binding; and research infrastructure teams evaluating whether automated experimentation can realistically slot into a live subfield. Readers should be comfortable with 3DGS training internals and multi-agent system design, since the paper is written at an advanced technical level.

Authors’ abstract

Reproducing a published 3DGS paper takes weeks of expert engineering before any extension can be proposed. Prior autonomous-research systems read papers, draft hypotheses, and write code, but are limited by short fixed templates, low executability, and weak traceability, and none has been demonstrated end-to-end in a live computer-vision subfield with intense activity. SplatLab is an autonomous research system for 3D Gaussian Splatting that addresses these limitations. Given a research prompt, a target-paper URL, or no input at all, it reproduces a published baseline, proposes an extension, trains and ablates across Mip-NeRF 360 and LLFF, writes the manuscript, and binds every reported number to a logged training run. Five design choices distinguish it: (i) candidates mutate plugins from a verified-reproduction backbone matched to published baselines; (ii) a cascade-weighted Elo tournament across eight islands scores candidates on measured GPU PSNR with verbal-gradient feedback; (iii) a standing falsifier subtracts Elo on failure, integrating stress-test survival into selection; (iv) every reported number is bound to a logged training run, and the run cannot terminate while any claim is unbacked; (v) a seven-category 3DGS taxonomy wraps EVOLVE-BLOCK markers in domain-aligned scaffolds transferable across 3DGS repositories. On PSNR-targeted runs, SplatLab's manuscripts beat their reproduced targets by +0.18 to +3.78 dB, including +3.78 dB on astrophotographic nebula rendering, for which no prior 3DGS method exists. The research cycle compresses from months of graduate-student time to days of mostly autonomous compute. Code, all manuscripts, experiment ledgers, and the cross-run paper store are available on the project page.