PaperCompiler: Faithful Paper-to-Code Generation via Repository-Level Specification Compilation

AI summary

Overview

Research area: Natural language processing / automated code generation — specifically paper-to-code synthesis, where a machine learning paper is converted into a multi-file repository implementation. The work sits at the intersection of LLM-based software engineering agents, repository-level code generation, and research reproducibility.

Technical level: Advanced. The paper assumes familiarity with LLM agent pipelines, staged code-generation workflows, repository-level benchmarks, and evaluation protocols that compare generated code against author implementations.

One-sentence scope: The paper proposes PaperCompiler, a framework that compiles a research paper into explicit repository-level implementation specifications — with ownership assignments, cross-file dependencies, and non-degradation constraints — before generating code, and evaluates it on the 90-paper Paper2CodeBench benchmark.

What This Paper Is About

Research papers usually describe methods at a high level, leaving implementation-critical details (data preprocessing, initialization, evaluation conventions) implicit. Existing paper-to-code systems pass paper-derived knowledge between stages as free-form plans or summaries, which downstream coding agents can ignore, reinterpret, or compress, leading to algorithmic simplification and inconsistent repository structure.

PaperCompiler's goal is to make those intermediate representations explicit and binding: instead of a summary, it produces a compiled specification that tracks what the paper supports, what must not be degraded, which file owns each requirement, and how artifacts flow between files.

Key Contributions

1. Specification compilation as the framing for paper-to-code generation. The authors recast repository synthesis as a controlled information transformation, separating three questions that prior workflows conflate: what is supported by the paper, which requirements must be preserved or remain unresolved, and where each requirement belongs in the repository.

2. A three-phase framework (PaperCompiler). Paper Grounding builds an implementation blueprint and a reference registry; Specification Compilation reconciles requirements and produces ownership and file-level specifications; Constraint-Guided Repository Generation generates files in dependency order under those specifications.

3. Evidence typing and provenance preservation. Each atomic implementation item is recorded as z = (x, ℓ, τ, r), where the evidence-status tag τ distinguishes paper-supported, externally delegated, inferred, and unresolved information, and the locator ℓ points to a section, table, equation, algorithm, appendix, or external reference. Long or format-sensitive material (prompt templates, output schemas, algorithm listings, benchmark-specific evaluation formats) is copied verbatim into a reference registry Q rather than summarized.

4. Non-degradation constraints and ownership structure. Reconciled requirements include a forbid field identifying substitutions that would weaken the intended method, and a primary ownership function ω maps core requirements to the files responsible for them, alongside producers π(a) and consumers Γ(a) for tracked cross-file artifacts.

Main Findings

• Largest gains under reference-based evaluation. On 90 papers from Paper2CodeBench (three 30-paper subsets from ICLR, ICML, and NeurIPS 2024), PaperCompiler improves reference-based fidelity from 3.647 to 4.152, a 13.8% relative improvement — the largest of the three protocols.

• Consistent but smaller improvements in reference-free settings. Reference-free scores improve from 4.562 to 4.777 (4.7% relative) and P2C-Ex scores from 4.535 to 4.728 (4.3% relative).

• Per-subset results. ICLR 2024: reference-free 4.500 → 4.804 (+6.8%), P2C-Ex 4.479 → 4.771 (+6.5%), reference-based 3.667 → 4.179 (+14.0%). NeurIPS 2024: 4.596 → 4.779 (+4.0%), 4.579 → 4.700 (+2.6%), 3.617 → 4.125 (+14.0%). ICML 2024: 4.591 → 4.746 (+3.4%), 4.547 → 4.711 (+3.6%), 3.659 → 4.151 (+13.4%).

• Gains are broadly distributed, not driven by outliers. Per-paper win ratios against PaperCoder are higher for PaperCompiler under all three evaluation protocols across all three conference subsets, with ties excluded.

• General multi-agent baselines score much lower. ChatDEV averages 4.043 reference-free and 2.877 reference-based; MetaGPT averages 3.580 reference-free and 2.727 reference-based. Both show relatively large standard deviations across conference subsets.

• Fewer severe semantic failures. Across roughly 2.6K critique items per method, algorithmic degradation drops from 28.0% to 24.6%, missing core components from 12.3% to 6.8%, and evaluation mismatches from 13.4% to 8.4%. High-severity critiques fall from 13.2% to 6.1%, and high-or-medium-severity critiques from 54.2% to 37.9%.

• One failure category worsens. API/schema mismatches rise modestly from 2.3% to 4.0%, which the authors flag as a remaining interface-alignment challenge. External protocol hallucination rises from 3.0% to 3.7%, and the no-clear-failure label rises from 6.7% to 13.5% (noted as sensitive to evaluator phrasing).

• Comparison with end-to-end systems on a ten-paper subset. PaperCompiler reaches 4.813 reference-free, 4.850 P2C-Ex, and 4.263 reference-based, versus PaperCoder at 4.450 / 4.513 / 3.825, AutoReproduce at 3.375 / 3.013 / 2.650, and AutoP2C at 2.700 / 2.700 / 2.088.

• Token cost. PaperCompiler uses 1.71M tokens per repository on average versus 0.98M for PaperCoder, corresponding to approximately $1.88–$7.51 per generated repository at current o3-mini API rates. AutoP2C uses a comparable 1.68M tokens yet scores substantially lower, so the authors argue the gap is not explained by token volume alone. AutoReproduce's aggregate token usage is not reported.

• Ablations identify Reconciliation and Contracting as most important. Removing Requirement Reconciliation drops reference-based performance from 4.38 to 3.86 (−0.51); removing File-Level Contracting drops it to 3.92 (−0.46); removing Context Slicing drops it to 4.11 (−0.26).

• Reference-free scores can be misleading under ablation. Removing Context Slicing actually improves reference-free scores by +0.15 and P2C-Ex by +0.32 while reducing reference-based fidelity by 0.26, which the authors read as evidence that reference-free evaluation can reward surface-complete repositories.

• Qualitative failure cases. On VDC without Contracting, the multimodal inference module becomes an InstructBlipModel placeholder with an unimplemented forward path; full PaperCompiler preserves a functional Instruct-BLIP interface, yielding a 2.25-point reference-based advantage. On INTR without Reconciliation, the decoder retains only cross-attention while omitting the self-attention and feed-forward refinement required by the method.

• Motivating case study. On Universal Neural Functionals (Zhou et al., 2024), PaperCoder keeps only a single partition candidate, removing most basis elements, whereas PaperCompiler enumerates valid partitions and assembles the corresponding basis blocks.

Methodology in Plain English

The authors treat repository generation as a compilation problem: paper text goes in, an explicit specification comes out, and code is generated from that specification rather than from a loose summary. The pipeline has three phases.

Paper Grounding. The parsed paper (converted to Markdown with MinerU) is turned into a compact implementation blueprint using a structured LLM prompt with a fixed output schema. The blueprint identifies the implementation scope M (main method versus baselines, optional analyses, and ablation-only components) and a set of atomic implementation items Z. Each item records what the detail is, where it came from, whether it is paper-supported, externally delegated, inferred, or unresolved, and what implementation role it serves. Separately, a reference-extraction step copies material that is too long or format-sensitive to compress — prompt templates, output schemas, algorithm listings, benchmark-specific evaluation formats — verbatim into a reference registry.

Specification Compilation. Requirement Reconciliation checks the grounded items against their evidence, groups related items into method-level requirements, and represents each as k = (id, role, src, req, bdry, forbid), where req is the behavior to preserve, bdry records semantic or runtime boundaries, and forbid names substitutions that would weaken the method. Ownership-Guided Architecture Synthesis then maps the core requirements onto concrete files through the ownership function ω, assigns a producer and consumers to each tracked cross-file artifact, and derives the repository graph G = (F, E, ω, π, Γ), whose generation dependencies are kept acyclic. File-Level Contracting slices out the requirements, interfaces, artifact relations, dependencies, unresolved cases, and reference materials relevant to each file, producing a per-file specification S_i = (I_i, A_i, R_i, H_i, D_i) covering public interfaces, the implementation recipe, produced and consumed artifacts, cross-file handoff requirements, and non-degradation or unresolved constraints.

Constraint-Guided Repository Generation. Files are generated in topological order. Each file is generated using its own specification as the primary instruction, previously generated code as committed context (treated as fixed in terms of paths, public APIs, schemas, artifact names, and externally visible behavior), and downstream file specifications as compatibility constraints. When the specification contains an unresolved dependency or incompatible boundary, the system preserves the specified interfaces and method-specific requirements while making the limitation explicit. Local engineering choices not fixed by the paper remain flexible.

Evaluation setup. All controlled comparisons use the same MinerU-parsed Markdown inputs, o3-mini for generation, and o3-mini-high for evaluation, with scores on a 1–5 scale aggregating multiple independently sampled judge outputs. The ablation study samples nine papers (INTL, AutoVP, TransformerCompression, RECOMBINER, WassersteinSSL, INTR, Auto-J, VONet, VDC), generating each paper–variant pair once with o3-mini to yield 27 ablated repositories plus nine PaperCompiler repositories, averaging eight independent judgments per repository. The ten-paper subset used for the end-to-end comparison consists of INTL, AutoVP, TransformerCompression, RECOMBINER, CAML, CARE, GGS, SEABO, iTransformer, and SparseFormer.

Why This Matters

Impact on research. Reproducibility in machine learning depends on implementation details that papers frequently leave implicit. This work argues that the bottleneck is not only LLM capability but the representation and propagation of paper-derived information, and it shows that stronger reference-based fidelity — the protocol that compares against author repositories — can be achieved by controlling that representation rather than by scaling generation budget alone.

Real-world applications.

• Reproducing published methods when official code is unavailable or incomplete, which the authors identify as a central motivation.
• Accelerating the path from a paper to a working reference implementation for researchers exploring a new method.
• Benchmark and evaluation harness reconstruction, where the paper's evaluation conventions and output formats must be preserved rather than approximated.
• Auditing how faithfully a generated repository implements a claimed method, using the failure-label taxonomy from algorithmic degradation through producer-consumer breaks.

Industry relevance. Teams that reimplement research methods for products face exactly the failure modes catalogued here — dropping a component, substituting a generic approximation, or mismatching interfaces between files. The reported cost of roughly $1.88–$7.51 per generated repository is small against the fidelity gains, though the 1.71M-token budget and the observed increase in API/schema mismatches indicate that the approach still requires review rather than blind adoption.

Future Directions

1. Multimodal paper understanding. The authors state that PaperCompiler relies primarily on text-parsed paper content, limiting recovery of implementation-critical information conveyed through architecture diagrams, complex figures, and visual examples. They propose incorporating multimodal document understanding as future work.

2. Retrieval and tool interaction for external dependencies. Papers that rely on specialized external tools, proprietary APIs, simulators, undocumented benchmark conventions, or auxiliary resources remain difficult; the authors suggest targeted retrieval and executable tool interaction.

3. Fine-grained API consistency. API/schema mismatches increased from 2.3% to 4.0%, which the authors call a remaining interface-alignment challenge and an important direction for future work.

4. Validation, execution testing, and human verification. Because the compiled requirements and file-level specifications are generation-time guidance rather than formal correctness guarantees, the authors propose integrating automated validation, execution-based testing, and independent or human verification — noting that the current evaluation does not establish full reproduction of reported experimental results or executability across all generated repositories, generation seeds, and model families.

Target Audience

Researchers and engineers working on LLM-based code generation, automated software engineering agents, and research reproducibility tooling will get the most from this paper. It will also interest benchmark designers concerned with the gap between reference-free and reference-based evaluation, since the ablation results show reference-free scores can rise while alignment with author implementations falls. Readers should be comfortable with staged agent pipelines and repository-level evaluation protocols; practitioners evaluating paper-to-code systems for adoption will find the failure taxonomy and token-cost comparison directly useful.

Authors’ abstract

Faithfully translating research papers into repository-level implementations remains challenging because papers often describe methods at a high level, leave implementation assumptions implicit, and require generated repositories to preserve method logic, evaluation protocols, and cross-file consistency. Despite recent advances in paper-to-code agents, their intermediate outputs are often presented as free-form plans or summaries that downstream coding agents may ignore, reinterpret, or compress, leading to algorithmic simplification and inconsistent repository structure. To address these challenges, we introduce PaperCompiler, a paper-to-code generation framework that compiles paper-grounded evidence into explicit repository-level implementation specifications. PaperCompiler grounds implementation-relevant evidence while preserving source provenance and distinguishing paper-supported, inferred, externally delegated, and unresolved information. The resulting specifications encode non-degradation requirements, ownership assignments, cross-file dependencies, and file-level constraints. Repository generation proceeds under these compiled specifications while retaining flexibility over local engineering choices not fixed by the paper. PaperCompiler outperforms strong baselines on Paper2CodeBench, achieving a 13.8% relative improvement in reference-based fidelity (from 3.64 to 4.15) and reducing high-severity evaluator critiques (from 13.2% to 6.1%).