NERFIFY: A Multi-Agent Framework for Turning NeRF Papers into Code

AI summary

Overview

Research area: Automated code synthesis for computer vision research — specifically, converting neural radiance field (NeRF) papers into runnable training code.

Technical level: Advanced. The paper assumes familiarity with NeRF rendering pipelines, multi-agent LLM systems, and 3D scene reconstruction metrics (PSNR, SSIM, LPIPS).

Scope: Nerfify is a multi-agent LLM framework that converts NeRF research papers into fully trainable Nerfstudio plugins, evaluated against expert human implementations across a 30-paper benchmark called Nerfify-Bench.

What This Paper Is About

Since the original NeRF paper appeared in 2020 it has spawned over 1,000 follow-ups, but most of them have no publicly available code, so anyone building on them must first reimplement them from scratch. Nerfify takes a NeRF paper PDF as input and automatically produces a trainable NeRF implementation in the Nerfstudio framework, targeting visual quality comparable to expert human reimplementations while cutting implementation time from weeks to minutes. The paper's central claim is that generic paper-to-code systems and frontier models like GPT-5 usually fail here, and that domain-specific design is what makes the difference.

Key Contributions

1. Nerfstudio as a context-free grammar (CFG). The paper formalizes Nerfstudio's architectural patterns and interface contracts as a CFG that constrains LLM synthesis, so generated code satisfies architectural invariants by construction rather than by post-hoc repair.
2. Graph-of-Thought (GoT) multi-agent code synthesis. A master agent orchestrates specialized file-agents that build multi-file repositories in topological dependency order — DAG construction, interface freezing, implementation, and integration testing — validating type signatures, tensor shapes, and circular dependencies at each node.
3. Compositional citation recovery. Agents traverse citation graphs of cited papers to retrieve implicit dependencies that the target paper only mentions in passing, such as proposal networks (Mip-NeRF 360), hash encoders (Instant-NGP), and factorization schemes (TensoRF). For K-Planes, the paper reports this requires 7 direct dependencies and 12 total papers including transitive dependencies.
4. Visual-driven feedback and a benchmark. A critique agent diagnoses artifacts using PSNR-minima ROI analysis, cross-view geometric validation, and Qwen3 VLM-guided patching; the paper also introduces Nerfify-Bench, described as the first specialized benchmark for NeRF paper-to-code synthesis, containing 30 papers across four categories.

Note: the abstract lists six innovations (adding a knowledge-enhancement option and the benchmark), while Section 1 states "five key technical innovations."

Main Findings

• Baselines almost never produce trainable code. Table 2 shows Paper2Code, AutoP2C, GPT-5, and R1 all fail to compile/train, fail training stability, and fail to converge to paper results. Paper2Code fails to resolve imports; the other three resolve imports but remain untrainable. The paper states generic baselines fail to produce trainable code in 95% of cases, versus 100% executability for Nerfify.
• Nerfify matches expert human quality on papers with no public code. On Set 1 (never-implemented papers), Table 1 reports KeyNeRF at PSNR 26.12 / SSIM 0.90 / LPIPS 0.09 for Nerfify versus 25.70 / 0.89 / 0.12 for the human implementation; mi-MLP NerF at 22.85 / 0.87 / 0.15 versus 22.64 / 0.87 / 0.15; ERS at 27.02 / 0.90 / 0.12 versus 26.87 / 0.90 / 0.12; and TVNeRF at 27.30 / 0.92 / 0.10 versus 26.81 / 0.92 / 0.12. The abstract and Section 1 state this is within ±0.5 dB PSNR and ±0.2 SSIM; Section 4.3.1 states "within 0.5 dB PSNR and 0.02 SSIM on average," so the SSIM tolerance is reported inconsistently across sections.
• Nerfify reproduces official code exactly where it exists. For Vanilla NeRF (31.36 PSNR / 0.95 SSIM / 0.04 LPIPS) and Nerfacto (20.36 / 0.82 / 0.22), Nerfify's numbers are identical to the original repositories; the paper notes LLMs may have seen these codebases during pretraining. For author-provided implementations, Nerfify produces standardized Nerfstudio-compatible code: ℓ0 Sampler goes from 29.21 / not reported / 0.04 to 30.13 / 0.97 / 0.03, while InfoNeRF drops from 18.27 / 0.81 / 0.23 to 17.87 / 0.69 / 0.44.
• Novelty coverage is near-perfect. Across 10 Set 4 papers (Table 4), Nerfify reports correct implementation rate C=1.00 and missing-component rate M=0.00 on every paper, with overall LLM scores of 0.95–1.00. Baselines show large omissions, with M ranging broadly across methods (the paper summarizes this as M=0.12–0.90 for most methods).
• Component ablations quantify what matters. Table 5, averaged over 10 Nerfify-Bench papers: full Nerfify scores 0.98 with 100% trainability and C=1.00. Removing in-context examples drops the score to 0.71 (90% trainable, C=1.00). Removing citation recovery drops it to 0.68 with C=0.65. Removing both yields 0.58 (90% trainable, C=0.65). Removing smoke tests drops trainability to 60% with score 0.69 and C=0.85. Removing VLM feedback raises the score slightly to 0.99 (still 100% trainable, C=1.00). Replacing GoT with one-shot generation collapses the score to 0.45 with 70% trainability.
• Context on the difficulty of the task. The paper cites that O1 achieves 26.6% accuracy on complex papers versus 41.4% for human experts, and that Claude 3.5 Sonnet reaches only 21% on PaperBench versus 41.4% for human researchers on ICML papers. The original NeRF training requires 100k–300k iterations, taking 24–48 hours on high-end GPUs, and expert developers needed 1–2 weeks per paper to create reference implementations for Set 1.

Methodology in Plain English

Nerfify runs in four stages.

Stage 1 — Paper understanding and grounding. MinerU converts the PDF to structured markdown preserving equations, tables, figures, and references. A cleaning agent strips extended introductions, related-work discussion, and redundant references while preserving all equations, pseudocode, diagrams, and citation relationships, and checks that every key technical component named in the abstract survives. Curated NeRF paper–code pairs are stored as in-context examples alongside the formalized Nerfstudio grammar.

Stage 2 — Dependency recovery. A paper extractor agent builds a citation dependency graph and recursively walks it. For each cited paper it identifies which components are borrowed (architectural modules, loss functions, training protocols), retrieves those papers, and recurses until no unresolved dependencies remain and all interface contracts of the target paper can be satisfied.

Stage 3 — Code generation. A master agent coordinates specialized file-agents over a repository DAG. Agents first freeze minimal public APIs in topological order, then implement each file, validating contracts and shapes locally, then run end-to-end smoke tests that trigger automated critique and repair. The paper argues this graph-native structure gives component-level fault localization and faster convergence than generating the whole repository monolithically.

Stage 4 — Visual refinement. After synthesis, the system smoke-trains for 3k iterations, renders from multiple viewpoints, and sends images to a critique agent that runs three branches: a metric branch computing local-window PSNR and SSIM error maps with morphological operations to find worst regions; a geometry branch implementing Cross-View Artifact Consensus to flag view-inconsistent floaters and ghosting; and a semantics branch using Qwen3 to diagnose artifact triplets and propose patches. The master agent applies patches revertibly, looping until the critique agent produces no feedback, the iteration limit is reached, or PSNR targets from the original paper are met. An optional human-in-the-loop mode exists but was not used in the reported experiments.

Evaluation setup. All experiments ran on NVIDIA A6000 GPUs with 48GB memory, training scenes for 100k iterations on Blender and DTU datasets.

Why This Matters

Impact on research. The paper's central argument is that depth of specialization, not model scale alone, unlocks reliable translation of scientific ideas into working code. If it holds, papers that would otherwise remain purely theoretical become usable artifacts, and compositional research — building on prior methods rather than reimplementing them — becomes far cheaper. It also directly attacks a reproducibility bottleneck: over 1,000 NeRF follow-ups, most without code.

Real-world applications (the paper does not evaluate these directly, but they follow from the NeRF domain):

• Novel view synthesis and photogrammetry pipelines that need fast prototypes of new radiance-field methods.
• 3D content creation for AR/VR, where new NeRF variants could be tested without a specialist writing a new implementation.
• Robotic perception and simulation, where scene reconstruction methods need to be swapped and compared quickly.
• Scientific and industrial 3D scanning, where method papers currently lag their usable implementations by a long stretch.

Industry relevance. Teams that want to evaluate a new NeRF paper before committing engineering time can generate a trainable baseline in minutes instead of weeks. More broadly, the paper frames domain-specific grammars plus agentic reasoning as a template that could be applied to other vision subfields and, eventually, to broader paper-to-experiment pipelines.

Future Directions

• Generalize beyond NeRF. The authors explicitly state future work will extend Nerfify to further NeRF variants and NeRF-based methods, to other areas of computer vision research, and to broader paper-to-experiment frameworks.
• Close the human-in-the-loop gap. An optional user-driven feedback mode exists expressly for domain experts but was disabled for fair evaluation; understanding how much expert steering adds to quality is untested.
• Reconcile reproduction with improvement. Nerfify can adaptively modify hyperparameters through visual feedback, and the paper notes this can achieve better training dynamics than the original paper specifies. How to balance fidelity to a paper against practical convergence remains an open design question.
• Address benchmark bias from pretraining. The paper relies on Set 1 (papers with no public implementation) to avoid LLM training-data contamination, and concedes that for Set 2 and Set 3 papers the LLMs may have seen the code. Scaling contamination-free evaluation is unresolved.

Target Audience

Researchers and engineers working on NeRF and neural rendering who want to build on papers without public code; practitioners building multi-agent LLM code-synthesis systems; and reproducibility-focused groups in computer vision and machine learning. The paper is also relevant to research-software maintainers evaluating whether domain-specific grammars are a viable alternative to scaling generic coding agents.

The paper states that code, data, Nerfify-Bench, and generated implementations will be publicly released, though the truncated content does not include a URL.

Authors’ abstract

The proliferation of neural radiance field (NeRF) research requires significant efforts to reimplement papers before building upon them. We introduce NERFIFY, a multi-agent framework that reliably converts NeRF research papers into trainable Nerfstudio plugins, in contrast to generic paper-to-code methods and frontier models like GPT-5 that usually fail to produce runnable code. NERFIFY achieves domain-specific executability through six key innovations: (1) Context-free grammar (CFG): LLM synthesis is constrained by Nerfstudio formalized as a CFG, ensuring generated code satisfies architectural invariants. (2) Graph-of-Thought code synthesis: Specialized multi-file-agents generate repositories in topological dependency order, validating contracts and errors at each node. (3) Compositional citation recovery: Agents automatically retrieve and integrate components (samplers, encoders, proposal networks) from citation graphs of references. (4) Visual feedback: Artifacts are diagnosed through PSNR-minima ROI analysis, cross-view geometric validation, and VLM-guided patching to iteratively improve quality. (5) Knowledge enhancement: Beyond reproduction, methods can be improved with novel optimizations. (6) Benchmarking: An evaluation framework is designed for NeRF paper-to-code synthesis across 30 diverse papers. On papers without public implementations, NERFIFY achieves visual quality matching expert human code (+/-0.5 dB PSNR, +/-0.2 SSIM) while reducing implementation time from weeks to minutes. NERFIFY demonstrates that a domain-aware design enables code translation for complex vision papers, potentiating accelerated and democratized reproducible research. Code, data and implementations will be publicly released.