LLM Agents Implement an NLG System from Scratch: Building Interpretable Rule-Based RDF-to-Text Generators

AI summary

Overview

Research area: Natural language generation (NLG), specifically RDF-to-text generation, neurosymbolic systems, LLM-based program synthesis, and knowledge graph verbalisation.

Technical level: Intermediate. The core ideas are accessible without deep machine learning background, but familiarity with RDF triples, NLG evaluation metrics, and basic Python helps.

One-sentence scope: The paper describes a framework in which five cooperating LLM agents write a complete, interpretable, rule-based RDF-to-text generator in pure Python from knowledge graph triples alone, with no human reference texts, and no LLM used at inference time.

What This Paper Is About

RDF-to-text systems usually fall into two camps: rule-based methods that are precise and controllable but brittle, or neural methods that are fluent but opaque, prone to hallucination, and computationally heavy. This paper asks whether LLM coding ability can be used during a "training" phase to write a rule-based NLG system from scratch, so that the deployed system is transparent Python code that runs fast on a single CPU. The goal is a generator that needs no supervised reference texts, can be read and edited by humans, and produces fewer unsupported facts than neural alternatives.

Key Contributions

1. A neurosymbolic training paradigm. A "training" procedure in which five LLM agents (Software Architect, Software Engineer, Evaluator, Test Engineer, Code Analyst) collaborate to iteratively design, implement, test, and revise a rule-based NLG system as a single Python file, using only in-domain RDF triples and no human reference texts.

2. Test-driven, reference-less construction. The Test Engineer agent synthesises unit tests (50 input-output example pairs per request, repeated until every predicate is covered by at least three unit tests) from the knowledge graph, which the Evaluator then uses to accept or reject the written program.

3. An interpretable, LLM-free inference system. The output is a single self-contained Python file (168 lines of code on average) that can be understood and modified by human engineers and needs no neural component at inference time.

4. Evaluation across five datasets and four training LLMs, including reference-based metrics, reference-less LLM-as-a-Judge evaluation, ablations, a small human evaluation, two software-engineering tasks measuring interpretability and editability, and inference time comparisons.

Main Findings

• Reference-based results on WebNLG. The system trained by GPT-4.1 agents reached the highest METEOR (0.7069 on all examples, 0.7124 on out-of-domain) and BLEURT (0.1841 all, 0.1483 out-of-domain) scores. Fine-tuned BART scored higher on overall BLEU (0.4352 vs. 0.3934) and BERTScore (0.9308 vs. 0.9291), but the rule-based system beat BART on both of these metrics on the out-of-domain subset (BLEU 0.3615 vs. 0.3052; BERTScore 0.9251 vs. 0.9183). All rule-based variants outperformed prompted Llama 3.3 70B.

• Largest open model approaches the proprietary one. Qwen 3 235B produced results close to GPT-4.1, scoring slightly better on BLEU (0.3939 all, 0.3772 out-of-domain) and on out-of-domain examples generally. Systems trained with Qwen 2.5 72B (BLEU 0.3309) and Llama 3.3 70B (BLEU 0.2858) performed less well, suggesting that stronger LLMs may be needed to implement a complete NLG system.

• OpenDialKG favours the fine-tuned model on surface metrics. Fine-tuned BART obtained the highest scores (BLEU 0.9372, METEOR 0.9849, BERTScore 0.9973, BLEURT 0.9340), which the authors attribute to the importance of original training data for producing expected sentence structures. Nonetheless, every rule-based variant beat the prompted Llama 3.3 70B baseline on all four metrics (e.g. GPT-4.1: BLEU 0.3144 vs. 0.2040).

• Fewer hallucinations in reference-less evaluation. On the WebNLG test set and the three GEM 2024 sets (factual, counterfactual, fictional), the GPT-4.1-trained system was more grammatical and added fewer unsupported facts than fine-tuned BART on all four sets (e.g. WebNLG test set: grammaticality 0.734 vs. 0.692; additions 0.029 vs. 0.510). Llama 3.3 achieved the highest grammaticality (0.752 and up to 0.984 on GEM factual), but added more unsupported facts than most rule-based variants.

• Strong showing on OpenDialKG reference-less metrics. The GPT-4.1-trained system produced significantly fewer additions (0.013) and omissions (0.022) than both fine-tuned BART (0.052, 0.139) and Llama 3.3 (0.030, 0.063), with statistical significance reported via a t-test at α = 5%. Its grammaticality (0.923) exceeded BART's (0.502) but was below Llama 3.3's (0.985).

• Large speed advantage. On the WebNLG test set, the generated system took 7 seconds on CPU, versus 249 seconds (GPU) / 1910 seconds (CPU) for fine-tuned BART and 6360 seconds (GPU) for prompted Llama 3.3 70B — a 35x speedup on CPU versus BART on GPU and 272x when both run on CPU.

• Both ablations hurt. Replacing the agent-generated design with a static human-written design (Abl. 1) lowered every metric (BLEU 0.323, METEOR 0.611, BERTScore 0.912, BLEURT -0.010 versus 0.331 / 0.653 / 0.922 / 0.119 for the full system). Using original WebNLG training examples instead of generated unit tests (Abl. 2) also gave slightly worse results (0.309 / 0.638 / 0.919 / 0.071).

• Human evaluation found no hallucinations for the authors' system. Across 100 randomly selected WebNLG test instances annotated by six NLP experts (300 outputs total, interannotator agreement 0.8288 Cohen's Kappa), the GPT-4.1-trained system had 0.00 minor and 0.00 major hallucinations, versus 0.22 and 0.40 for BART and 0.07 and 0.05 for Llama 3. It had the lowest disfluency rate (0.19).

• The generated code is genuinely interpretable. Two Python engineers passed all interpretability tests, averaging 9.6 seconds per instance to trace a highlighted word back to its source line, and completed code modifications to remove omissions in almost all cases within five minutes. Both reported the code was fully understandable, though it contained unused parts and could be refactored.

• Program characteristics. Generated programs average 168 lines of code, typically group triples by subject, add modifiers, form a clause, then refine it into a sentence. No runtime errors appeared on WebNLG, but GEM evaluations produced errors because the programs were not robust to differences in date formatting between datasets, reducing performance there.

Methodology in Plain English

The researchers treat code generation as a substitute for gradient-based training. Starting from a knowledge graph, a Test Engineer agent inspects every predicate, samples a triple per predicate, and asks an LLM to invent example input-output pairs, which become unit tests. A Software Architect agent designs the system's function structure. A Software Engineer agent implements each function one at a time. A Python interpreter then runs the program on the unit tests, with each instance in a separate process and a timeout; errors and timeouts count as failures, and evaluation stops early once five tests fail. Outputs that run successfully are passed to an LLM judge that answers a yes/no question about whether the text correctly reflects the input triples. A Code Analyst agent then diagnoses failures and decides whether a single function needs rewriting or the whole design needs revisiting — routing work back to the Software Engineer or the Software Architect accordingly. This loop ends when all unit tests pass or an iteration limit is hit.

Experiments used five datasets: the WebNLG test set plus the three GEM 2024 test sets (counterfactual, fictional, factual), and OpenDialKG. Baselines were a fine-tuned BART-base model, trained with AdamW, learning rate 2e-5, β = (0.9, 0.997), ε = 1e-9, a polynomial schedule with 10% warmup, 20 epochs, early stopping with patience 10, batch size 8, and 0.1 label smoothing; and prompted Llama 3.3 70B with post-processing. Training used GPT-4.1, Qwen 3 235B, Qwen 2.5 72B, and Llama 3.3 70B (the open-source models in 4-bit quantisation via ollama), up to 25 iterations (10 for GPT-4.1), repeated three times, with the best run chosen by number of unit tests passed. Because the WebNLG graph is large, the system was trained separately for each thematic category, and the Evaluator always used Llama 3.3 for comparability. Evaluation combined reference-based metrics (BLEU, METEOR, BERTScore, BLEURT — not applied to the GEM sets, which lack references), reference-less LLM-as-a-Judge scoring of grammaticality, additions, and omissions, and a small in-house human study.

Why This Matters

Impact on research. The paper offers an alternative to the assumption that building a good NLG system requires supervised data and a neural model at inference time. It connects program synthesis, agentic LLM workflows, and traditional rule-based NLG, and it shows that generated code can be competitive on out-of-domain inputs and hallucination measures even where it trails on surface-form metrics. It also provides a concrete case study of applying multi-agent code generation to a language-processing task, which the authors state has not been done before.

Real-world applications:

• Verbalising knowledge graphs and structured databases in public-sector, scientific, or enterprise settings where transparency and auditability of the generation process are required.
• Robustness-critical deployments such as counterfactual or fictional data scenarios (the GEM 2024 conditions), where rule enforcement can prevent inventing facts.
• High-throughput or resource-constrained environments, since the system runs on a single CPU with no GPU and produces text in seconds rather than minutes or hours.
• Systems that must be maintained and edited by domain experts, since the output is readable Python that engineers successfully modified to fix omissions.

Industry relevance. The combination of fast CPU inference, no dependency on an LLM at serving time, full control over phrasing, and no need for annotated training corpora addresses several common objections to deploying neural NLG. The authors note that generated programs are not robust to schema-level differences such as date formatting between datasets, which is a deployment consideration.

Future Directions

• Improving robustness to schema and formatting differences across datasets, since the generated programs failed on GEM largely due to date-format mismatches.
• Reducing dependence on the strongest proprietary models, as smaller open-source LLMs produced noticeably weaker systems, and determining what capabilities a model needs to write a complete NLG system.
• Making the generated code cleaner and more maintainable, since the engineers who inspected it found unused parts and refactoring opportunities.
• Expanding beyond the two domains tested, and clarifying the guarantee of generalisation, which the paper limits to unseen data that follows the same schema with predicates defined consistently with the training graph.
• Further study of hallucination behaviour, given the limitation that the approach may still generate non-factual outputs and that the authors recommend thorough testing before deployment.

Target Audience

Researchers and practitioners in natural language generation, knowledge graph verbalisation, and neurosymbolic AI; engineers interested in agentic LLM code-generation workflows and in systems that avoid neural components at inference; and industry teams that need controllable, auditable, low-cost data-to-text generation. Evaluation and NLP engineering readers will also find the reference-less judging setup, ablations, human study, and interpretability tasks useful as a methodological template.

Authors’ abstract

We present a novel neurosymbolic framework for RDF-to-text generation, in which the model is "trained" through collaborative interactions among multiple LLM agents rather than traditional backpropagation. The LLM agents produce rule-based Python code for a generator for the given domain, based on RDF triples only, with no in-domain human reference texts. The resulting system is fully interpretable, requires no supervised training data, and generates text nearly instantaneously using only a single CPU. Our experiments on the WebNLG and OpenDialKG data show that outputs produced by our approach reduce hallucination, with only slight fluency penalties compared to finetuned or prompted language models