Instruction Tuning of Large Language Models for Tabular Data Generation-in One Day

AI summary

Overview

Research area: Instruction tuning of large language models (LLMs) applied to synthetic tabular data generation — a conditional generation task where a model produces new rows of a table that match the structure and statistical properties of an input table.

Technical level: Intermediate. The paper is written in an accessible style and requires familiarity with LLM fine-tuning concepts (instruction tuning, LoRA-free full fine-tuning, DeepSpeed ZeRO stages) and with standard tabular synthesis evaluation metrics (Shape, Trend, TSTR), but it does not introduce new mathematical machinery.

Scope: A single empirical study demonstrating that a compact open-source LLM instruction-tuned on 7K high-quality instructions for under 6 hours on one A100 80GB GPU can generate tabular data at a quality comparable to GPT-4o. (Note: the arXiv listing assigns this paper to cs.CV, although its content is tabular data generation and instruction tuning.)

What This Paper Is About

Tabular instruction tuning has become a popular way to improve how LLMs understand structured data, but existing work almost exclusively targets question answering and reasoning over tables — not the generation of new tabular data. Generating realistic, domain-relevant tabular data matters because such data is scarce in many scientific and industrial settings, and LLMs are poorly aligned with tabular structure due to the mismatch between the bi-dimensional, relational nature of tables and the unidimensional autoregressive training objective of LLMs.

The obstacle the authors attack is cost: prior state-of-the-art tabular instruction tuning (TableLlama) used roughly 2 million tabular instructions and 48 A100 GPUs. This paper asks whether tabular data generation capability can be improved with limited data and limited compute.

Key Contributions

1. First exploration of instruction tuning for tabular data generation. The authors state that, to the best of their knowledge, no prior work has studied instruction tuning as a way to improve LLM tabular data generation; prior tabular instruction tuning work (TableLLM, TableLlama, TAMA) addresses QA, reasoning, or Pandas code generation only.

2. A high-quality instruction dataset for conditional tabular data generation. The dataset contains 10K instructions. It is built from 20 publicly available tabular datasets spanning 10 topics, split into 14 tables for training and in-domain evaluation and 6 held-out tables for out-of-domain evaluation. Each instance contains an instruction, an input table, table metadata (a general description plus column-wise descriptions including column names and data types), and an expected output table.

3. ITT-Gen (Instruction Tuning for Tabular data Generation). The method instruction-tunes Llama3.1-8B-Instruct on 7,000 mixed and shuffled training instructions using a single A100 80GB GPU, a learning rate of 2e-5, a batch size of 3, 2 epochs, and DeepSpeed ZeRO-2 for training efficiency — completing in less than 6 hours.

4. Evidence that the approach is model-agnostic and resource-efficient. The paper reports that ITT-Gen reaches fidelity and utility performance on par with GPT-4o, and shows in an appendix that the same recipe also improves TableLlama, a state-of-the-art open-source model for table-based tasks.

Main Findings

• The untuned base LLM largely fails the generation task. Llama3.1-8B-Instruct frequently produces unrelated instructions instead of tables; in the reported behavior, only about 20% of its generated output is usable tabular data while roughly 80% is discarded. Only for some instructions does it generate partial rows rather than the 20 rows requested. The authors note they collected and filtered all generated tabular data specifically so that fidelity and utility metrics could be reported for the base LLM at all.

• ITT-Gen matches GPT-4o on fidelity. Across the datasets in Table 1, the instruction-tuned model reaches parity overall with GPT-4o, though the two models trade wins on individual datasets. Examples: on bank, ITT-Gen scores Shape 85.57 / Trends 86.34 versus GPT-4o's 93.42 / 91.7 and the base LLM's 75.63 / 65.08; on breast_cancer, ITT-Gen scores 84.12 / 69.36 versus GPT-4o's 78.65 / 64.16 and the base LLM's 55.31 / 37.07; on california_housing, ITT-Gen scores 73.29 / 80.06 versus GPT-4o's 96.27 / 97.84 and the base LLM's 88.7 / 90.52.

• Caveat on the base LLM's fidelity numbers. The paper explicitly flags that the base LLM's apparently competitive fidelity scores are computed only on the tabular portion of its output (about 20%), with the remaining irrelevant generation discarded.

• ITT-Gen matches GPT-4o on utility. Under the Train-on-Synthetic, Test-on-Real (TSTR) framework with linear, random forest, and XGBoost models, ITT-Gen approaches GPT-4o and clearly beats the base LLM on most reported datasets. Examples from Table 2: adult (real 0.8796; base 0.655867; ITT-Gen 0.826533; GPT-4o 0.873200), bank (real 0.800720; base 0.353441; ITT-Gen 0.616246; GPT-4o 0.819928), and bestseller (real 0.781972; base not usable, marked "–"; ITT-Gen 0.743701; GPT-4o 0.710766).

• Several datasets refuse to yield usable base-LLM output. On bestseller, breast_cancer, and iris, the base LLM's output cannot be used to train a machine learning model (marked "–") and is therefore excluded from those utility comparisons.

• Tabular data generation is a distinct capability from table QA and reasoning. TableLlama — pretrained on a large set of table-based tasks — produces output that does not follow tabular structure at all on the four evaluated datasets, so fidelity and utility cannot even be computed for it (marked "–"). After instruction tuning on the authors' dataset, its fidelity improves substantially (for example, California 78.57 / 79.75; Boston 75.84 / 75.63), yet a considerable gap to GPT-4o remains, which the authors attribute to TableLlama's relatively outdated Llama 2 backbone.

• Metadata quality matters. The authors report that preliminary experiments indicated high-quality metadata is important for steering LLMs toward proper tabular data generation, which motivated using GPT-4o to draft table descriptions followed by manual review.

• Row-based supervision beats next-token prediction. Providing a set of rows as the expected output during instruction tuning gave better results than the next-token prediction approach used in previous work, according to the authors' empirical results.

Methodology in Plain English

1. Collect tables. Sample 20 publicly available tabular datasets covering 10 topics, ranging from 150 rows and 5 columns (iris) to 45,211 rows and 17 columns (Bank Marketing) and 48,842 rows and 15 columns (Adult Income). Fourteen tables are kept for training and in-domain evaluation; six are held out for out-of-domain evaluation.

2. Build instructions. For each dataset in the training set, construct 500 training instances and 100 evaluation instances; evaluation-only datasets contribute 100 evaluation instances each. Each instance pairs a manually designed instruction describing the generation task with an input table, metadata, and an expected output table. Input and output tables are formed by randomly selecting N = 20 rows.

3. Generate metadata with GPT-4o, then verify. Metadata consists of a general table description (topic, structure, applications) and column-wise descriptions (column name and data type — numerical, categorical, or textual). A template prompt, seeded with a manually written example description, is used to obtain descriptions for all tables; the authors then manually review every generated description for quality and accuracy.

4. Fine-tune. Mix and randomly shuffle the 7,000 training instructions (500 × 14 datasets) and fine-tune Llama3.1-8B-Instruct with the Hugging Face transformers library, learning rate 2e-5, batch size 3, 2 epochs, DeepSpeed ZeRO-2, on one A100 80GB GPU, in less than 6 hours.

5. Evaluate. Compare against two baselines: the untuned Llama3.1-8B-Instruct and GPT-4o. Fidelity is measured with Shape (per-column marginal distribution similarity) and Trend (capture of inter-column correlation), where higher is better. Utility is measured with Train-on-Synthetic, Test-on-Real (TSTR), training on synthetic data and testing on held-out real data using linear, random forest, and XGBoost models; averaged AUC is reported for classification datasets and R² for regression datasets in the main utility table.

Why This Matters

Impact on research. The paper reframes tabular data generation as a distinct instruction-following capability rather than a byproduct of table understanding, and it shows that the field's assumption of needing millions of instructions and dozens of GPUs may not hold for this task. It also provides a reusable dataset construction recipe — table snapshot plus general and column-wise metadata — and reports that TableLlama, a state-of-the-art table model, fails at generation despite extensive table-task pretraining. A small, well-curated dataset and a single GPU suffice for competitive results, which lowers the barrier to entry for academic and small-lab research.

Real-world applications:

• Augmenting limited real-world datasets in data-scarce scientific and industrial domains to make downstream machine learning more feasible.
• Generating synthetic tabular data for privacy-conscious sharing scenarios, where the goal is data that follows the input table's column distributions and inter-column relationships without reproducing it.
• Supporting benchmarking, testing, and prototyping pipelines where realistic tabular data is needed but real data is restricted.
• Enabling downstream classification and regression workflows in domains such as finance and credit risk, healthcare and medical research, housing economics, and customer analytics — the topical areas covered by the paper's datasets.

Industry relevance. The authors are affiliated with Betterdata AI in Singapore alongside the Singapore Institute of Technology and the National University of Singapore, indicating direct commercial interest in practical, low-cost synthetic tabular data. The headline cost profile — one A100 80GB GPU, under 6 hours, 7K instructions — is within reach of most organizations that could not previously contemplate tabular instruction tuning at the scale of 48 A100 GPUs.

Future Directions

• Scaling the instruction dataset. The paper does not test whether adding instructions beyond 7K, or drawing on more than 20 datasets and 10 topics, yields further gains toward or beyond GPT-4o.
• Testing more and larger base models. The two base LLMs studied are Llama3.1-8B-Instruct and TableLlama; the authors note the TableLlama gap stems from its outdated Llama 2 backbone, leaving open how much a newer or larger backbone would close the remaining distance to GPT-4o.
• Handling the hard cases. Several datasets remain weak for all methods, including job_posting, Players2024, and california_housing under fidelity, and the TableLlama-based utility results on Credit and Diabetes trail GPT-4o. The paper does not explain why, or how to fix it.
• Closing the gap between metric scores and practical usability. The paper reports that the untuned base LLM's fidelity scores look competitive only because roughly 80% of its output is discarded; a follow-up question is how to measure instruction-following reliability directly rather than post-filtering output.
• Privacy guarantees for generated tables. The paper frames generation around matching distributions and inter-column relationships but does not report any privacy metric, leaving the privacy properties of ITT-Gen-generated data an open question for real data-sharing use.

Target Audience

Researchers and practitioners working on tabular data generation, synthetic data, and LLM instruction tuning; machine learning engineers who need realistic tabular data but have limited GPU budgets; and applied teams in finance, healthcare, insurance, and analytics who cannot afford the data-and-compute regime of prior tabular instruction tuning work. Readers should have some familiarity with LLM fine-tuning and standard synthetic data evaluation, but the paper is approachable for those new to tabular instruction tuning specifically.

Authors’ abstract

Tabular instruction tuning has emerged as a promising research direction for improving LLMs understanding of tabular data. However, the majority of existing works only consider question-answering and reasoning tasks over tabular data, leaving tabular data generation largely unnoticed. In this work, for the first time, we explore the efficacy of instruction tuning in improving LLMs tabular data generation capabilities. More specifically, given the high data and computation requirements of tabular instruction tuning, we aim to address the possibility of instruction tuning for tabular data generation with limited data and computational resources. To achieve this, we first create a high-quality instruction dataset for tabular data, enabling efficient LLM comprehension. We then instruction-tune an open-source LLM (Llama3.1-8B-Instruct) on the training set of this dataset to improve its tabular data generation performance. Our experimental results show that by using our high-quality dataset and instruction-tuning on only 7K instructions with an A100 GPU, for less than 6 hours, we achieve tabular data generation performance on par with the most capable commercial LLM, GPT-4o.