SPLATIFY: Reproduce, Discover, Innovate! From Papers and Ideas to Trainable 3DGS Code

AI summary

Overview

• Research area: Computer vision / neural rendering, specifically 3D Gaussian Splatting (3DGS), combined with LLM-based multi-agent code synthesis and automated research reproduction.
• Technical level: Advanced. The paper assumes familiarity with 3DGS, differentiable rasterization, gsplat/Nerfstudio plugin architecture, context-free grammars, and retrieval-augmented agent pipelines.
• Scope (one sentence): The paper presents Splatify, a multi-agent framework that turns 3DGS papers into trainable gsplat implementations, then composes and invents new methods beyond the source papers, evaluated on a new 30-paper benchmark called Splatify-Bench.

What This Paper Is About

3DGS research has grown to over 1,500 follow-up papers in under three years, roughly three new arXiv submissions per day, yet over 40% of tracked 3DGS papers release no code, and those that do often publish isolated forks of the INRIA codebase with incompatible assumptions. Reimplementing a single paper before building on it typically costs weeks of manual work. The authors build a multi-agent system that reads a 3DGS paper and produces a compiling, trainable gsplat plugin in 10–30 minutes, and then goes further by autonomously combining published techniques to beat the reproduced result and by importing physics from outside the 3DGS literature to handle scene types no prior 3DGS paper has addressed.

Key Contributions

1. A context-free grammar for gsplat plugins built over a modular method template with nine extension points covering losses, densification, rendering, optimization, and per-Gaussian attributes. The grammar appears in every agent's system prompt and doubles as a verification contract, so generated repositories are valid derivations by construction and satisfy gsplat's architectural invariants.
2. Architectural elements for faithful reproduction: fork-aware citation recovery that diffs a cited paper's repository against its parent (3DGS, Mip-Splatting, 2DGS, Scaffold-GS) and labels changed functions at function-level granularity; Graph-of-Thought synthesis in topological dependency order; RAG-guided in-context example selection seeded with 20 manually validated paper–code pairs; and visual feedback combining PSNR-guided regeneration, Gaussian-level structural checks, and VLM-driven patching.
3. Knowledge-driven compositional improvement and interdisciplinary method discovery: an evolutionary search over a structured technique space (densification strategy, composable losses, anti-aliasing, appearance models, schedule parameters) with 8 explorer agents on separate islands, plus a mode that retrieves physical priors from outside the 3DGS literature to synthesize physics-informed modifications.
4. Splatify-Bench: a 30-paper benchmark split into three sets of 10 — never-implemented papers with commissioned expert reference implementations, papers with public code, and novelty-coverage papers used for component-level analysis.

Main Findings

• Set 1 (no public code): Splatify matches expert implementations within ±0.5 dB PSNR and ±0.02 SSIM on average, while every baseline (Paper2Code, AutoP2C, GPT-5.2, Gemini 3.1 Pro, DeepSeek 3.2, Nerfify) failed to produce trainable code. Experts needed one to two weeks per paper; Splatify took 10–30 minutes.
• Set 2 (public code): Against original author repositories trained under identical conditions, Splatify shows average deviations under 0.3 dB PSNR while standardizing each method into a gsplat plugin within Nerfstudio for fair cross-method comparison.
• Novelty preservation: On the 10 novelty-coverage papers, Splatify reaches weighted semantic scores above 0.94 on all papers (TrickGS 0.94, 2DGS 0.95, Scaffold-GS 1.00, GaussianShader 1.00, Pixel-GS 1.00), versus notably lower scores for Paper2Code, Gemini 3.1 Pro, DeepSeek 3.2, and GPT-5.2 Thinking. Judge–human agreement was κ=0.87 on correct/incorrect/missing labels and Spearman ρ=0.92 on semantic score.
• Compositional discovery improves on reproductions by up to +2.4 dB PSNR with no human guidance. The largest single jump was DWTGS from 18.68 dB to 21.09 dB (patch structure loss plus opacity/normal/depth regularization); MSAA 3DGS rose from 25.35 to 26.19 dB, Opti3DGS from 27.71 to 28.05 dB, GNS from 25.97 to 26.15 dB, and CompGS from 28.28 to 28.50 dB. Random sampling over the same search space used 8 times as many tokens and reached only a quarter of the PSNR gain.
• NebulaGS (volumetric nebula rendering): 32.52 dB PSNR and 0.914 SSIM versus 31.19 dB and 0.862 for vanilla 3DGS (+1.33 dB, +0.052 SSIM), with per-view standard deviation dropping from 6.29 to 2.62 and rendering speed unaffected. Evaluated on 157 training and 23 test views with six autonomous modifications, including an opacity ceiling loss (α < 0.95), a per-Gaussian emission parameter, and a 2D FFT log-amplitude spectral loss.
• BubbleGS (iridescent thin-film rendering): 27.81 dB PSNR and 0.869 SSIM versus 26.97 dB and 0.858 for vanilla 3DGS, LPIPS improving from 0.264 to 0.244, per-view standard deviation dropping from 8.23 to 4.85, and speed falling from 99.1 to 69.1 FPS due to a second thickness rasterization pass. Six modifications included a per-Gaussian film thickness parameter, a thin-film chrominance loss, and an opacity ceiling at α = 0.45.
• Ablation (10 Set 1 papers, full system scores 0.97 with 100% trainability): removing CFG constraints drops the score to 0.91 and trainability to 70%; removing in-context examples drops to 0.73; removing fork-aware recovery drops to 0.64 and leaves 30% of novelties unimplemented; removing smoke tests causes the largest trainability drop (50%); removing Gaussian structural checks gives 0.89 and 80%; removing VLM feedback or PSNR curve monitoring each gives 0.94; one-shot generation without Graph-of-Thought collapses to 0.49 with 60% trainability.
• Ecosystem diagnosis: roughly 70% of 3DGS papers release code as forks of the INRIA repository. The running example, Trick-GS, assembles 8 techniques from 7 papers, and fork-chaining expands its dependency graph to 12 papers; GaussianPro spans 8 dependencies, and Scaffold-GS and PGSR each span 7.
• Training-curve feedback is 3DGS-specific: because 3DGS trains in minutes, agents can monitor the full PSNR trajectory. Densification terminates around 15k iterations, after which pruning or loss-schedule faults appear as late-stage collapse (PSNR drops sharply post-15k) or stalled plateau (under 0.1 dB gain over any 5k-iteration window after warm-up).

Methodology in Plain English

The authors treat a 3DGS paper as a specification and a gsplat plugin as a fill-in-the-blanks template. They studied over 40 3DGS papers and found that modifications cluster into nine places in the code — things like adding a loss term, changing how Gaussians split and prune, or attaching a new per-Gaussian attribute. They wrote a formal grammar describing which combinations of those nine slots are legal, plus a working template class (GSTemplateModel) that already inherits the rasterizer, density control, learning-rate schedules, and optimizer setup. An agent therefore never writes a whole repository from scratch; it only decides which slots a paper's contributions map to and writes the code for those hooks, and a master agent checks after each file that the repository is still a legal derivation.

Because most 3DGS code lives in forks, text-level citations are not enough. For each cited paper, the agent checks whether the repository forks a known base, computes a structured diff against the parent, and tags each changed function with a semantic label such as loss/depth_distortion. Chaining these diffs across transitive fork edges recovers the full modification stack, with a fallback to PDF-level extraction when no repository exists.

Generation proceeds in dependency order, with interface freezing and smoke tests. Afterwards the system runs 3k iterations and inspects the result on four fronts: per-window PSNR/SSIM error maps, cross-view geometry checks for floaters and ghosting, a vision-language model producing structured diagnoses with candidate patches, and the PSNR training curve. Faults get localized to a specific extension point and patched, looping until feedback stops, budget runs out, or the paper's reported PSNR is matched.

For going beyond reproduction, eight "explorer" agents each mutate one axis of a structured search space (mutually exclusive densification strategies among ADC, AbsGrad, and MCMC; additively composable losses; anti-aliasing, appearance models, and schedule parameters), converting each proposal into runnable code and evaluating it in a 2k → 7k → 30k iteration cascade that eliminates the bottom half at each stage. Proposals are ranked by Pareto sorting on 0.4·PSNR + 0.3·(1−LPIPS) + 0.3·SSIM, and the loop stops when no agent improves for three consecutive rounds. For cross-domain invention, agents additionally retrieve physics from outside the field and propose losses that appear in no existing 3DGS paper.

Why This Matters

The paper reframes paper-to-code from a translation task into an infrastructure problem: reproduction alone is not the bottleneck, discoverability and composability of the accumulated literature are. By making implementations uniform gsplat plugins inside Nerfstudio, results from different papers become directly comparable under identical training conditions — something the fork-dominated ecosystem currently prevents. The demonstration that agents can autonomously find a better combination of existing techniques than any individual paper (up to +2.4 dB) suggests a practical route to exploring a composition space too large for manual search.

Real-world applications:

• Astronomical and scientific visualization: NebulaGS shows 3DGS-style reconstruction for optically thin emission media, where standard alpha compositing produces opaque shell artifacts and Kolmogorov turbulence places detail at spatial frequencies pixel-space losses under-weight.
• Thin-film and material appearance capture: BubbleGS handles soap bubbles and oil slicks via Airy-function reflectance, where peak reflectance is only about 8% and standard spherical harmonics cannot represent the rapid color variation.
• Rapid prototyping for rendering researchers: getting a newly published method into a runnable, comparable form in minutes rather than one to two weeks per paper.
• Microscopy and other domains the authors name as natural fits for Gaussian primitives but currently untouched because the engineering barrier is too high.

Industry relevance: teams building 3D reconstruction, AR/VR content pipelines, or simulation stacks can absorb new research faster and compare variants under one harness. The broader lesson — domain-specific grammar plus verified in-context examples plus closed-loop visual feedback — transfers to any research area with a stable framework and a modular extension surface.

Future Directions

• Porting the architecture to other domains. The authors state that adapting Splatify requires only defining a CFG over a new framework's extension points and curating a seed set of verified paper–code pairs, naming 3D reconstruction pipelines and robotics simulation stacks as candidates. Whether that transfer holds empirically is untested.
• Scaling the knowledge base and search. The in-context library is seeded with 20 manually validated implementations and grows as Splatify succeeds, and the technique space is drawn from the knowledge base; how far compositional discovery improves as both grow, and whether quality plateaus, is an open question.
• Validation of invented methods. Cross-domain success is measured by improvement over vanilla 3DGS, cross-view consistency, and physical plausibility, with human verification at the code level. The paper does not report a physical ground-truth comparison for NebulaGS or BubbleGS, leaving rigorous validation of physics-informed inventions unresolved.
• Contamination and generalization limits of the benchmark. Set 1 was designed to rule out pretraining contamination by using papers with no public code, but Set 2 papers may have appeared in LLM training data — the authors note baselines still fail despite this advantage. Extending evaluation to a larger, continuously refreshed paper pool is a natural next step.

Target Audience

Most valuable to 3DGS and neural-rendering researchers who want to build on recent papers without reimplementing them, and to LLM-agent and automated-research-reproduction researchers interested in grammar-constrained code synthesis with domain-specific feedback loops. It is also relevant to graphics and vision engineers maintaining reconstruction pipelines who need cross-paper comparability, and to scientifically oriented graphics researchers exploring physics-informed Gaussian representations. Readers without background in Gaussian Splatting, differentiable rasterization, and plugin architectures will find the technical sections dense.

Authors’ abstract

The rapid growth of 3D Gaussian Splatting (3DGS) research demands significant effort to reimplement papers before building on them. We introduce SPLATIFY, a multi-agent framework that converts 3DGS papers into trainable gsplat-based implementations, where generic paper-to-code methods and frontier models fail. SPLATIFY achieves this through five innovations: (1) A context-free grammar for gsplat over a modular method template with extension points for losses, densification, rendering, and optimization, constraining synthesis so generated code satisfies gsplat's architectural invariants by construction. (2) Architectural elements for faithful reproduction: fork-aware citation recovery retrieving component-level code at function-level granularity, Graph-of-Thought synthesis in topological dependency order, RAG-guided in-context example selection from over 20 verified implementations, and visual feedback combining PSNR-guided regeneration, Gaussian-level structural checks, and VLM-driven patching. (3) Knowledge-driven compositional improvement that autonomously finds weaknesses and composes complementary regularizers, losses, and densification strategies to improve upon original results. (4) Interdisciplinary method discovery where agents retrieve physical priors from outside the 3DGS literature and compose them with rendering knowledge to produce methods for previously unaddressed scene types. (5) SPLATIFY-Bench, an evaluation framework across 30 diverse 3DGS papers. On papers without public code, SPLATIFY matches expert implementations while reducing development time from weeks to minutes, and through compositional discovery further improves PSNR by up to 2.4 dB. We additionally demonstrate novel methods for volumetric nebula rendering and other scientific domains, synthesized entirely by SPLATIFY.