Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data

AI summary

Overview

Research area: natural language processing, specifically data-to-text (D2T) generation, cross-domain transfer, and knowledge distillation from large language models into small models. Technical level: Advanced. Scope: This paper asks whether a small (~1–2B parameter) language model can learn to generate faithful text from five very different kinds of structured data — time series, triples/dictionaries, tables, and charts — when no in-domain training texts and no test references exist, by distilling synthetic target-domain data from a larger teacher model.

What This Paper Is About

Structured data comes in many shapes (tables, knowledge graphs, charts, time series) and turning it into text can mean different tasks, from weather reporting to game summarisation to chart captioning. Most prior D2T work either trains on task-specific data or relies on zero-shot prompting of large models, and prior work by Kasner and Dusek (2024) showed that 76 to 86% of texts generated by several open-source 7B models across five domains contain at least one factual or semantic error. This paper's goal is to make small models reliably generate faithful English text across five such domains without any in-domain reference texts for training or evaluation, using out-of-domain supervision, synthetic data, and structural data augmentation.

Key Contributions

1. The authors investigate data-driven knowledge distillation (DDKD) approaches for cross-domain data-to-text generation in a setting with no target-domain reference text.
2. They introduce structure-preserving target-domain input augmentations — structural subsampling and noise-based structural perturbation — intended to improve robustness to schema and format heterogeneity.
3. They construct QUINTD-5, a fivefold extension of QUINTD-1 that raises real target-domain inputs from 100 to 500 per domain, and use it to show that structure-preserving augmentation is more effective and more cost-efficient than simply scaling real target-domain inputs.
4. They provide a reference-free evaluation combining error-taxonomy LLM-as-a-Judge assessment with two independent judges (GPT-5.1 and Gemini-2.5-Pro), human validation and agreement analysis, and a content-coverage check showing that DDKD improves faithfulness without relying on conservative under-generation.

Main Findings

• Constant-size distillation wins: At constant model size (1.7B parameters), DDKD consistently outperforms both fine-tuning on out-of-domain WebNLG data and zero-shot inference across all five target domains, for both the Qwen3 and Gemma3 backbones.
• Small models beat large ones on some domains: The abstract reports that the small distilled models outperform a much larger finetuned model on two of the five domains and achieve comparable performance on the remaining three. Section 6.1 separately reports that a compact distilled model outperforms much larger LLMs (Qwen3-32B and GPT-4.1) on four of the five domains (Wikidata, Ice Hockey, GSM Arena, and OWID).
• Best augmentation configuration: Under the WebNLG-SFT teacher, the mixed strategy gave the best small-model scores per domain (Wikidata 0.10, Ice Hockey 0.03, OpenWeather 4.47, GSM Arena 0.87, OWID 0.36) and the best NormAvg of 0.10, versus 0.47 for the unaugmented Base (100) setting.
• Augmentation beats real-data scaling: NormAvg is 0.42 when training on 500 real instances versus 0.27 using perturbed instances under the zero-shot teacher, and 0.20 versus 0.10 for the mixed setting under the WebNLG-SFT teacher.
• Weather is hardest: Across all models, the OpenWeather domain consistently yields the worst scores, which the authors attribute to the complexity of summarising weather data spanning 15 data points.
• Teacher choice shifts domain ranking: For zero-shot inference and DDKD with a ZS teacher, the best-performing domains in descending order are Wikidata, GSM Arena, Ice Hockey, OWID; for SFT and DDKD with an SFT teacher, the ranking shifts to Ice Hockey, Wikidata, OWID, GSM Arena.
• Fewer hallucinations, not less content: Among 1.7B models, the best distilled model (DDKD from SFT) hallucinates substantially less (Not Checkable error rate 0.05) than the zero-shot (0.53) and finetuned (0.70) models, while DDKD also achieves higher content coverage than same-size zero-shot and source-domain SFT baselines, remaining close to the selected best teacher.
• Large models remain more faithful overall: The three large models show lower Incorrect error counts (0.14, 0.50, 0.41) than the best distilled model (1.13), which the authors say indicates distillation needs further improvement in conditioning on input structure.
• Judge agreement is high: Inter-judge agreement is very high at the system level (r > 0.95 for both score- and ratio-based judgments; p < 0.001 when pooling all domain-system pairs).
• Human evaluation corroborates: Human annotators labelled 240 outputs (60 inputs, 12 per domain); inter-annotator agreement was high for Incorrect Fact (α = 0.810) and Other (0.871) but lower and even negative for Not Checkable and Misleading. Agreement between human evaluation and GPT-5.1 was high at the system level (Pearson r > 0.90).
• Gains are not explained by conservative under-generation: The coverage sanity check shows DDKD improves faithfulness without relying on conservative omission.

Methodology in Plain English

The starting point is the QUINTD-1 benchmark, which provides five domains (Wikidata, Ice Hockey, OpenWeather, GSM Arena, OWID), each with 100 development inputs and 100 test inputs and no reference texts at all. The authors treat the development inputs as seeds and reserve the test inputs purely for evaluation.

Four approaches are compared. First, zero-shot prompting, where an LLM is simply asked to verbalise the structured input. Second, supervised fine-tuning of the model on out-of-domain data — 40K knowledge graph/English text pairs from WebNLG, whose graphs are deterministically re-written into the JSON, Markdown, and CSV formats that appear in the target domains, so the model is not confused by format mismatch. Third and fourth, data-driven knowledge distillation (DDKD), where a large teacher model writes synthetic target-domain texts for the seed inputs and a small student model is trained on those (input, synthetic text) pairs with standard maximum-likelihood estimation. The teacher is either a plain zero-shot LLM or one already fine-tuned on WebNLG.

The distinctive ingredient is augmentation of the seed inputs. Structural subsampling exploits the compositional structure of the inputs: given a central entity and a set of attribute-value pairs, the method takes non-empty subsets of those pairs and asks the teacher to verbalise them, creating many simpler but still faithful training examples. Noise-based structural perturbation then randomly modifies values within instances (for example birth rates in OWID) at a fixed perturbation ratio of 20% of selected instances, to reduce overfitting. A mixed setting combines both and shuffles them together. Separately, the authors build QUINTD-5 by collecting five times as many real structured inputs (500 per domain) using the same pipeline, APIs, and preprocessing, as a control for whether simply adding real data would do as well.

Because no reference texts exist, evaluation uses an LLM-as-a-Judge protocol with an error taxonomy of four types — Incorrect Fact, Not Checkable, Misleading, and Other — scoring the average number of errors per output, with GPT-5.1 as the primary judge and Gemini-2.5-Pro as an independent second judge. A content-coverage check guards against the possibility that models score well merely by saying less. Human annotators then label a stratified diagnostic subset to validate the automatic judges.

Why This Matters

Research impact: the paper shows that cross-domain generalisation for D2T generation can be transferred into compact models by using large LLMs as intermediate generators rather than as final deployment targets, and that augmentation plus distillation can substitute for absent in-domain training data. It also demonstrates that structure-preserving augmentation is more data-efficient than collecting more real inputs, and supplies QUINTD-5 as a scaling control plus a reference-free evaluation recipe combining two LLM judges, human validation, and a coverage check.

Real-world applications, drawn from the domains the paper discusses:

• Generating weather forecasts from time series data.
• Verbalising knowledge graphs, records, and structured entity descriptions.
• Producing summaries of sports games from statistics.
• Generating captions for chart data, including health data from OWID.

Industry relevance: the paper's cost table shows that a 1.7B DDKD model trained on OWID took 74.8 GB peak memory and 1.2 hours on a single A100 80 GB GPU, compared with 20.3 hours for fine-tuning Qwen3-32B-SFT and 10.4 hours for Qwen3-8B-SFT on WebNLG; inference for small distilled models over the full 100-example test set completes within 5 minutes across all domains, while generating synthetic supervision with a large teacher can take up to 12 hours per target domain. This makes small, long-context models attractive where fine-tuning and inference at long sequence lengths would otherwise be computationally prohibitive.

Future Directions

• Extending the study to additional open-weight model families and scales, since the current work focuses on Qwen3 and Gemma3 in the 1B–2B range and transferability of the gains to significantly larger architectures is not yet fully established.
• Exploring how comparable evaluation results can be obtained with open evaluators, given that GPT-4.1 and GPT-5.1 are used only for comparison and judging and are closed-source.
• Reducing the reliance on heuristic choices in structural augmentation, such as defining atomic units and enforcing exchangeability constraints, which carry minimal structural assumptions about input data and may not be optimal for all domains.
• Improving distillation so that student models condition better on input structure, since the best distilled 1.7B model still shows a higher Incorrect error count (1.13) than the large models (0.14, 0.50, 0.41).

Target Audience

Researchers and practitioners in natural language generation, data-to-text generation, and knowledge distillation who work with heterogeneous structured inputs; engineers who need to deploy small, long-context models on tasks that lack in-domain labelled data or reference texts; and evaluators interested in reference-free assessment using LLM-as-a-Judge protocols with multi-judge and human validation.

Authors’ abstract

Structured data exists in many forms (tables, knowledge graphs, charts, and time series), and converting it into text may involve different generation tasks. However, most prior work on data-to-text (D2T) generation has focused on specific tasks and datasets, relying either on task-specific training data or on the zero-shot capabilities of large language models. We study cross-domain D2T generation in a setting where neither in-domain training text nor test references are available, and where domains, generation goals, and input structures vary substantially. We compare data-driven knowledge distillation (DDKD) against zero-shot inference and fine-tuning on out-of-domain D2T data, and introduce structure-preserving augmentation via structural subsampling and perturbation. Experiments on five benchmarks show that, at constant model size (1.7B parameters), DDKD consistently outperforms both fine-tuning and zero-shot inference. Moreover, the resulting small models outperform a much larger finetuned model on two of the five domains, achieving comparable performance on the remaining three. We further construct QUINTD-5, a fivefold extension of QUINTD-1, and show that simply scaling real target-domain inputs yields only modest gains, whereas our augmentation strategy remains more effective and more cost-efficient for cross-domain distillation.