Skeletons Matter: Dynamic Data Augmentation for Text-to-Query

AI summary

Overview

Research area: Natural Language Processing — semantic parsing, specifically LLM-based translation of natural language questions into formal query languages (SQL, Cypher, nGQL), with a focus on data augmentation for fine-tuning.

Technical level: Intermediate. The paper is accessible to readers familiar with LLM fine-tuning and basic database concepts, though the structural-similarity machinery (AST edit distance, query skeleton extraction) assumes some technical background.

Scope: This paper unifies Text-to-SQL, Text-to-Cypher, and Text-to-nGQL under a single "Text-to-Query" paradigm and proposes a dynamic, skeleton-aware data augmentation framework that diagnoses a model's structural weaknesses and synthesizes targeted training data to fix them.

What This Paper Is About

Existing work on translating questions into database queries treats each query language separately, producing methods that do not transfer across SQL, Cypher, or nGQL. Data augmentation approaches are also static — they generate more examples indiscriminately, often duplicating question types the model already handles correctly, which wastes data and training compute.

The authors argue that underneath their surface differences, queries in all these languages share abstract structural patterns — "query skeletons" — and that failures in these skeletons are the dominant source of model errors. The paper's goal is to build one general framework that dynamically identifies which skeletons a specific model fails on and synthesizes targeted question-query pairs to correct exactly those weaknesses.

Key Contributions

1. A unified task formulation. The authors formally define the Text-to-Query paradigm as f(S, q) → Q, mapping a database schema S and natural-language question q to a query Q in any language, covering relational and graph databases under one formulation.

2. Query skeletons as a shared optimization target. They introduce query skeletons — queries with instance-specific tables, columns, and literals replaced by placeholders — as a language-agnostic abstraction for diagnosing model behavior, analyzing errors, and guiding augmentation.

3. A dynamic data augmentation framework with three components: (a) a diagnosis step that identifies model-specific error-prone skeletons via structural similarity measurement, (b) a Skeleton Generalizer that produces novel but plausible unseen skeletons, and (c) a skeleton-guided backward-forward synthesis pipeline that instantiates skeletons, back-translates them into questions, and verifies consistency.

4. State-of-the-art results with minimal data. Their models, Skeletron 7B and Skeletron 14B, achieve top performance on four benchmarks (Spider, BIRD, Text2Cypher-Exec, NL2GQL) using only 10,000 synthesized examples — roughly 1/250 of the data used by the closest competitor. Code is released publicly.

Main Findings

• Best-in-class Text-to-SQL performance. Skeletron 14B reaches 87.3% EX and 82.0% TS on Spider dev, 86.6% EX on Spider test, and 65.1% EX on BIRD dev, surpassing its own teacher model (Qwen2.5-72B-Instruct) and all fine-tuning baselines including CODES and OmniSQL.

• Large gains on graph query languages. On Text-to-Cypher and Text-to-nGQL, Skeletron 14B achieves 58.6% and 45.1% EX, outperforming Qwen2.5-Coder-32B-Instruct by 14.4 and 18.2 percentage points respectively — despite the baseline being a much larger model specialized for code.

• Extreme data efficiency. OmniSQL uses 2.5 million synthetic examples; Skeletron uses 10,000 yet still beats it by +0.6% TS on Spider dev and +0.9% EX on BIRD dev, and wins by up to 9.5% when compared under matched data budgets.

• Improvements scale with question difficulty. Over the base model, gains on BIRD grow from 15.1% on simple questions to 22.8% on challenging ones, because the diagnosis step naturally surfaces the harder skeletons the model fails on.

• Skeleton errors dominate model failures. On BIRD dev, Qwen2.5-Coder-32B-Instruct produces incorrect query skeletons in 26.3% of predictions, and these account for 73.4% of all its errors. Skeletron 14B cuts the skeleton error rate to 24.0%, confirming that targeting skeletons addresses the largest slice of the error distribution.

• Every component matters. Ablations show the largest drop comes from removing synthetic data entirely (BIRD dev falls from 61.4 to 57.5 EX). Removing dynamic diagnosis, the Skeleton Generalizer, or forward verification each causes measurable degradation, with forward verification most important for semantic consistency.

• Robust to teacher model choice. Substituting smaller teachers (Qwen2.5-14B or 32B) for the 72B teacher causes only a modest performance decline (58.2–58.7 vs. 61.4 EX on BIRD), showing the pipeline does not depend on a top-tier teacher.

• Threshold tuning matters. A structural-distance threshold of 2 works best. Too low a threshold flags semantically equivalent queries as errors and injects noise; too high a threshold misses genuine weaknesses.

Methodology in Plain English

The pipeline runs in three phases.

Diagnosis. The authors take the target LLM and run cross-validation on the dataset's training split. For each question, they extract the skeleton from both the model's predicted query and the gold query — replacing concrete table and column names with placeholders. They then compare the two skeletons structurally, either by parsing to abstract syntax trees and counting the minimum number of edit operations needed to turn one into the other (using SQLGlot's Change Distiller implementation), or, for languages without mature parsers like nGQL, by computing token-level edit distance. If the difference exceeds a threshold of 2, the prediction is flagged as a skeleton error. Skeletons with an error rate above 20% form the "error-prone skeleton set."

Generalization. Fine-tuning a model only on known failure cases risks overfitting to them. So the authors fine-tune Qwen2.5-Coder-14B on the error-prone skeletons using only the assistant prefix of the instruction template as a prompt, turning it into a Skeleton Generalizer that emits novel skeletons in the same structural style. These new skeletons are merged with the error-prone ones to form a larger skeleton pool.

Synthesis and training. For each database in the target dataset, a skeleton is sampled from the pool and a teacher LLM (Qwen2.5-72B-Instruct) fills in appropriate schema elements to instantiate a concrete query. Rule-based checks verify executability, syntax, and foreign-key constraints. The teacher then works backwards, writing a natural-language question that the query answers — an easier direction than going forward, since formal queries are unambiguous. Finally, the teacher performs forward verification, using chain-of-thought reasoning to check that the question and query are semantically consistent and to correct mismatches or hallucinations. The resulting pairs are combined with the original training data, and the base model is fine-tuned on the mix, producing the Skeletron models.

Why This Matters

Impact on research. The paper reframes a fragmented collection of tasks as one problem, and offers query skeletons as a diagnostic tool that works across query languages. This gives the community a shared abstraction for error analysis and a template for transferable augmentation methods, moving beyond the current pattern of building bespoke, language-specific pipelines.

Real-world applications:

• Business intelligence and analytics tools that let non-technical staff ask questions in plain English and receive correct SQL against a warehouse schema.

• Graph database interfaces for Neo4j, NebulaGraph, and similar systems, where Text-to-Cypher and Text-to-nGQL tooling is far less mature than its SQL counterpart.

• Customer support and internal data access portals, where a small, locally fine-tuned model can answer data questions without routing sensitive queries through a proprietary API.

• Low-resource query language support, since the token-based structural distance fallback lets the method work even when no parser exists for a language.

Industry relevance. The method is cheap. It avoids the multi-million-example synthetic datasets that dominate current state-of-the-art pipelines while matching or beating them, which directly cuts data generation and training costs. It also relies on open-source models end-to-end, sidestepping the privacy and inference-cost concerns that accompany prompting proprietary LLMs with schema and data. For organizations running niche graph query languages, this is one of the few augmentation approaches that applies at all.

Future Directions

• Better benchmarks for non-SQL query languages. The authors acknowledge that Text-to-Cypher and Text-to-nGQL lack high-quality datasets and standardized evaluation protocols, which limits how rigorously their cross-language claims can be verified.

• A single multi-language model. Currently the augmentation is run separately per task, so the resulting model handles one query language at a time. Training one model to handle SQL, Cypher, and nGQL simultaneously remains open.

• Cross-language skeleton transfer. Since skeletons are abstract structural patterns, it is worth investigating whether skeletons diagnosed as errors in SQL could inform augmentation for structurally similar Cypher patterns, or vice versa.

• Principled threshold selection. The structural-distance threshold was set empirically to 2. A more adaptive or language-aware criterion for declaring a skeleton error could further reduce noise in the diagnosis step.

• Broader query language coverage. The framework in principle extends to SPARQL, MongoDB aggregation, GraphQL, and other languages, but this has not been tested.

Target Audience

This paper is most useful to NLP researchers working on semantic parsing, Text-to-SQL, and LLM fine-tuning, particularly those interested in data augmentation and error analysis. Practitioners building natural-language database interfaces — including data engineers, analytics platform developers, and graph database vendors — will find the framework directly actionable. It also suits readers studying the general problem of how to spend a limited fine-tuning data budget, since the dynamic diagnosis idea applies beyond query generation to any structured-output task with recurring compositional patterns.

Authors’ abstract

The task of translating natural language questions into query languages has long been a central focus in semantic parsing. Recent advancements in Large Language Models (LLMs) have significantly accelerated progress in this field. However, existing studies typically focus on a single query language, resulting in methods with limited generalizability across different languages. In this paper, we formally define the Text-to-Query task paradigm, unifying semantic parsing tasks across various query languages. We identify query skeletons as a shared optimization target of Text-to-Query tasks, and propose a general dynamic data augmentation framework that explicitly diagnoses model-specific weaknesses in handling these skeletons to synthesize targeted training data. Experiments on four Text-to-Query benchmarks demonstrate that our method achieves state-of-the-art performance using only a small amount of synthesized data, highlighting the efficiency and generality of our approach and laying a solid foundation for unified research on Text-to-Query tasks. We release our code at https://github.com/jjjycaptain/Skeletron.