Compile by Training: Turning Natural-Language Specifications into Local Neural Functions

AI summary

Overview

Research area: Natural language processing, specifically compiling natural-language specifications into small, reusable local neural functions; it sits at the intersection of LLM-based synthetic data generation, parameter-efficient finetuning, and practical ML systems/inference serving.

Technical level: Intermediate. The paper assumes familiarity with LoRA adapters, frozen base models, gradient-descent finetuning, and inference serving concepts, but it explains its pipeline and deployment architecture in largely plain terms.

Scope: The paper introduces "compile by training," a method that spends roughly a minute of compile-time teacher synthesis and adapter finetuning to produce a small local program from a natural-language specification, evaluates it on FuzzyBench-Hard, and demonstrates it in a deployed compiler service plus three applications.

What This Paper Is About

Many recurring text functions, such as routing an email to "immediate" versus "wait," are easy to describe in words but tedious or impossible to write as hand-coded rules, while calling a large remote model for every input adds repeated cost, latency, and provider dependence. The paper's goal is to make a natural-language description behave like ordinary software: compile it once, at build time, into a reusable local neural function that then runs without the teacher models.

Key Contributions

1. The compile-by-training method. Teacher models synthesize a task-specific dataset from a natural-language specification, and those examples finetune a lightweight LoRA adapter for a shared frozen interpreter. The amortized Program-as-Weights (PAW) compiler provides the starting adapter parameters and run-time scaffold, which are then refined by gradient descent, and the adapter, scaffold, specification, and interpreter metadata are packaged into a single reusable program.
2. An interactive minute-scale compilation service. The deployed service overlaps teacher synthesis with training via a streaming compile path, maintains persistent job records, dispatches jobs from a shared queue to GPU workers, reuses cached teacher outputs, and stores completed artifacts centrally so compilation behaves like a background software build rather than a blocking form submission.
3. An evaluation with a semantic-correctness metric on a hard benchmark. The paper reports LLM Exact Match (LEM) on FuzzyBench-Hard, a subset of specifications on which the PAW fast compiler produced no exact matches, comparing compile-by-training against the fast amortized compiler.
4. Three deployed demonstrations of composition. Compiled functions are used in a multi-site website helper (30 compiled programs), a language-controlled 3D avatar that emits an executable action DSL, and a bidirectional English-to-Claudish translator, alongside a public compiler playground and a public service.

Main Findings

• Large correctness gain on hard tasks: On FuzzyBench-Hard, compile by training improves mean LEM by 0.612 absolute, from 0.224 to 0.836. The paper notes the fast compiler still has a nonzero LEM because some predictions are semantically correct even without exact matches.
• The gain costs compile time: Compile by training takes 50.9 seconds, versus 3.5 seconds for PAW's fast amortized compiler, adding a point to PAW's speed–accuracy tradeoff.
• Teacher mixture helps: Using a 2:1 mixture of GPT-5.4-mini and GPT-5.5 supervision (2400/1200), with 3600 unique pairs repeated to form a 6400-example training set, raises mean LEM from 0.746 to 0.851 relative to GPT-5.4-mini alone (3600/0).
• More data helps modestly, with saturation: In the data-scaling sweep, mean LEM is 0.821 with 1440 unique pairs, 0.836 with 2400, remains 0.836 with 3600, and reaches 0.866 with 7200.
• The grader is validated: The selected GPT-5.5 LEM judge reaches 0.977 accuracy and Cohen's κ = 0.946 against 128 author labels (FPR 0.025, FNR 0.023), compared with 0.938 accuracy, FPR 0.05, FNR 0.068, and κ = 0.858 for the GPT-5.4-mini grader.
• Latency is usable interactively: A cold compile of the same representative specification took 50.9 s on a B300, 68.2 s on an H200, and 99.2 s on an RTX GPU, measured at service launch in May 2026. In a load test with four concurrent compile jobs, all four completed with a mean queue wait of 1.01 s and even utilization across workers.
• Deployed applications work: The website helper's content pack contains 30 compiled programs, 28 of which participate in live routing while two support evaluation and backward compatibility, serving four websites from one backend. The avatar program produced the expected action structure on 43 of 44 hand-authored validation instructions. Between its public launch on August 22 and September 2, 2026, the Claudish translation demo completed 100,747 successful translation requests.

Methodology in Plain English

A developer writes a description of the desired text function in ordinary language. The system sends that specification to teacher models through a structured JSON request template, and the teachers generate many input/output example pairs illustrating the behavior. The compiler validates each returned batch and rejects malformed or incomplete ones before the examples reach the trainer.

Training a separate full model per function would be too expensive, so every compiled function instead shares one frozen interpreter: a quantized Qwen3-0.6B model. Each function is represented by a small LoRA adapter (rank 64, alpha 16) plus a scaffold, which is a compiler-generated prompt template that encodes the specification as structured instructions and examples with a placeholder for the new input. The existing PAW amortized compiler provides an initial adapter and scaffold in seconds as a warm start, and the synthesized examples then refine the adapter by minimizing the negative log-likelihood of the reference outputs given the scaffolded inputs. The result is packaged into a .paw artifact.

To keep this minute-scale build usable, the deployed service starts teacher requests, model loading, and training concurrently rather than sequentially, so training begins as soon as the examples for its first batch exist and only blocks if it catches up with synthesis. A scheduler assigns incoming examples to the open slots needed by earlier batches first. The API keeps a persistent job record, a shared queue dispatches jobs to available GPU workers, workers reuse cached teacher outputs before requesting new ones, and finished programs go to a shared artifact store linked back to the job record. Users see queue position and training progress, and that progress survives page navigation and reloads.

For evaluation, because multiple outputs can be valid for fuzzy tasks, the paper reports LLM Exact Match: the fraction of predictions an LLM judge deems semantically correct given the specification, input, reference output, and prediction. The judge prompt tolerates cosmetic differences such as whitespace, JSON formatting, key order, bullet style, and output container choice, while rejecting differences in values, item count, order, structure, logic, and inappropriate refusals.

Why This Matters

Research impact: The paper frames model adaptation as a software build step, with large models acting as tool builders rather than run-time dependencies. It extends Program-as-Weights by adding an explicit accuracy point to the speed–accuracy tradeoff: a fast single-forward-pass compiler versus a slower specification-specific training build, using the same program format and run-time interface.

Real-world applications demonstrated in the paper:

• An email-triage style function that maps "Signature needed by EOD" to immediate and a newsletter to wait, used as the running example and shown in the public playground.
• A multi-site website helper that answers page-aware questions across the first author's personal website, the University of Waterloo CS 486 course site, NeuralOS, and the PAW website, combining compiled fuzzy decisions with deterministic BM25 retrieval, caching, and branch control.
• A language-controlled 3D avatar that turns instructions such as "jump twice, then dance" into a validated action DSL supporting sequences, durations, repetition, and compatible parallel motions.
• A bidirectional English–Claudish translator whose two separately finetuned programs power a live service and can also be downloaded and run locally.

Industry relevance: The approach targets recurring, narrow text functions that are too fuzzy for conventional rules but too frequent to justify a remote LLM call on every invocation. Compiled programs can be stored, versioned, cached, and composed like ordinary software artifacts, while local SDK execution does not send future inputs to the PAW service or the teachers — a meaningful property for cost, latency, and dependency management. The paper also reports an intermediate design: compilation is hosted, so the specification itself is sent to the PAW service and teacher APIs.

Future Directions

• Addressing teacher error inheritance: The limitations section notes that synthetic supervision may inherit teacher errors and that applications requiring guaranteed correctness should validate outputs or retain deterministic control paths. Reducing or detecting such errors is an open problem.
• Systematic user studies: The paper states that its application evidence focuses on composition and structured execution, and that systematic user studies are future work.
• Reconciling the data-scaling results: The data-scaling sweep shows LEM rising from 0.821 at 1440 unique pairs to 0.866 at 7200, with a plateau at 0.836 for both 2400 and 3600, leaving the returns from further scale and the best teacher-mixture/data-size combination as open questions.
• Extending composition: The paper demonstrates compiled functions used as components inside larger applications. How far this composition model extends across many interacting compiled functions and ordinary code is left as a direction implied by the demos rather than a settled result.

Target Audience

This paper is most useful to NLP and ML systems researchers working on efficient adaptation, synthetic supervision, and deployed inference; to engineers who need reliable, locally runnable text functions without per-call remote model dependence; and to practitioners building LLM-supported applications that combine fuzzy neural decisions with deterministic code. It is also relevant to readers interested in the operational side of "compilation," including build queues, caching, artifact packaging, and interactive latency, since those details are reported concretely rather than treated as an afterthought.

Authors’ abstract

Many recurring text functions are easy to describe but difficult to implement with rules, while calling a large remote model for every input introduces repeated cost, latency, and dependency on a provider. We present compile by training, which turns a natural-language specification into a reusable neural function. At compile time, teacher models generate task-specific examples that are used to train a small adapter for a compact interpreter. The resulting function runs without the teachers and can be stored, versioned, and composed like ordinary software. On FuzzyBench-Hard, a subset on which the Program-as-Weights fast compiler produced no exact matches, compile by training reaches 83.6% semantic accuracy. This higher accuracy comes with a higher compile-time cost: roughly a minute rather than seconds for the fast compiler. We deploy the compiler in a public interactive service and demonstrate compiled functions in a multi-site website helper, a language-controlled 3D avatar, and a bidirectional English-Claudish translator.