FALCON: A Model and Dataset Agnostic Framework for Synthetic Data Generation for NL2SQL Pairs

AI summary

Overview

Research area: Natural Language Processing, specifically NL-to-SQL (natural language to SQL) data generation for training language models to query relational databases.

Technical level: Intermediate to Advanced. The paper assumes familiarity with text-to-SQL benchmarks, fine-tuning, chain-of-thought prompting, and evaluation metrics such as exact match and execution accuracy.

Scope: The paper introduces FALCON, a model- and database-agnostic pipeline that uses compact open models to synthesize ambiguity-aware, complex NL-to-SQL training pairs, and evaluates the resulting data through human annotation, ablations, and difficulty-stratified downstream training experiments.

What This Paper Is About

Relational databases hold structured knowledge that non-experts want to query in natural language, but existing synthetic data generators either rely on expensive frontier-model APIs or produce oversimplified queries that miss real-world complexity and ambiguity. FALCON is a generation and filtering pipeline that aims to produce NL-SQL pairs matching the complexity of challenging human-authored benchmarks such as BIRD, while using only compact open models (for example Qwen-2.5-3B) so organizations can generate domain-specific data locally. The framework targets the complex end of the query distribution and is designed to complement, rather than replace, existing datasets.

Key Contributions

1. A model- and database-agnostic generation pipeline. FALCON combines reserved-word SQL seeding (target counts sampled from BIRD's reserved-word distribution) and persona-based prompting to produce structurally complex SQL and linguistically rich questions, with executability filtering at the SQL stage and LLM-as-a-judge selection among four candidate NL questions.

2. Explicit ambiguity handling rather than ambiguity removal. A DistilBERT model fine-tuned on Ambrosia data assigns each example a continuous ambiguity score; examples scoring above 7 trigger multi-interpretation generation, producing alternative valid SQL queries (up to three attempts) instead of discarding the example. The pipeline also introduces a "Broadness" ambiguity category alongside Scope, Attachment, and Entity Vagueness.

3. An alignment-based filtering model. Trained on 9,000+ BIRD-derived examples (50% systematically corrupted) using a hybrid metric combining semantic similarity (40%) and schema entity matching (60%) with corruption penalties, this ModernBERT-based scorer achieves MSE 0.0203 and MAE 0.1129 and correlates with human judgments at Pearson r = 0.70.

4. Cost and privacy analysis plus downstream difficulty-stratified validation. FALCON requires no external API calls, and the paper reports that generating 5,500 examples costs roughly $20 on a commodity GPU versus an estimated $326 at GPT-5.5 pricing, an over 16x reduction. Downstream training shows FALCON-trained models increasingly outperform Spider-trained models as query complexity rises.

Main Findings

• Generated data exceeds existing benchmarks in complexity and richness. Spider-seeded FALCON scored 0.43 SQL complexity and 0.39 NL richness; WikiSQL-seeded FALCON scored 0.28 and 0.39. For comparison, BIRD scored 0.42 and 0.31, Spider 0.38 and 0.29, and WikiSQL 0.20 and 0.28 — the only exception being the WikiSQL-seeded pipeline's SQL complexity, which falls below BIRD.

• Human evaluation is consistently high across model sizes. Four annotators each scored 100 disjoint examples. SQL, NL, and alignment scores all exceeded 0.92: Qwen-3B scored 0.96 / 0.95 / 0.94; Llama-8B 0.92 / 0.94 / 0.95; Qwen-32B 0.98 / 0.97 / 0.93; Llama-70B 0.93 / 0.97 / 0.94.

• The reward-model filter outperforms alternatives. At threshold 0.6 the reward model reached F1 93.73, specificity 71.43, and recall 92.81, reducing misaligned examples from 16% to 4% while retaining 92.81% of correct examples — described as 8-9x higher recall than competing methods. The prompt filter reached F1 91.54, specificity 28.57, recall 95.42; the LLaMA teacher filter reached 17.65 / 92.86 / 9.80; and Source2Synth reached 7.55 / 100.00 / 3.92.

• Ambiguity detection is deliberately conservative. On 100 human-annotated examples, the end-to-end ambiguity pipeline achieved 94.0% accuracy, 66.7% precision, 28.6% recall, 40.0% F1, and 98.9% specificity. The authors state the low recall is intentional: false positives create noisy multi-interpretation pairs, while false negatives merely omit optional enhancements.

• Design ablations favor SQL-then-NL with both seedings and a Compare Judge. On n=1000 examples with Qwen-2.5-3B, SQL-then-NL scored 0.42 SQL complexity and 0.31 NL richness versus 0.40 and 0.21 for NL-then-SQL. Reserved words raised SQL complexity to 0.48; personas raised NL richness to 0.34; the Compare Judge produced 0.46 and 0.36; the Score Judge produced 0.46 and 0.28.

• Cross-dataset transfer shows complementary strengths. With Qwen 2.5-3B, Spider-on-Spider reached 35.95 EM, 0.89 FS, and 0.42 Soft-EX; Spider-on-Pipe reached 0.05 EM, 0.69 FS, 0.18 Soft-EX; FALCON-on-Spider reached 0.80 EM, 0.68 FS, 0.32 Soft-EX; FALCON-on-Pipe reached 13.80 EM, 0.83 FS, 0.48 Soft-EX. Mixed training (95% synthetic, 5% Spider) reached 27.10 EM and 0.84 FS on Spider and 13.60 EM and 0.83 FS on Pipe.

• Advantage grows with query difficulty. Difficulty-stratified analysis shows FALCON-trained models increasingly outperform Spider-trained models as complexity increases, with the largest gaps in the hardest quartiles. Real-data seeding primarily benefits the easiest queries (first octile), while FALCON-only training maintains advantages on the hardest (last octile).

• Error analysis identifies one-third of failures as acceptable variations. Of 100 sampled incorrect Qwen 2.5-3B predictions on the Spider test set (model fine-tuned on FALCON data), 33.0% were acceptable variation, 20.0% spurious predicate hallucination, 18.0% result boundary error, 10.0% schema entity confusion, and 4.0% each for logical negation failure, degenerate join structure, cross-row constraint flattening, and hardcoded quantifier substitution, with 3.0% NL misinterpretation.

• Reserved-word adherence is moderate. The model reaches Pearson correlation r = 0.6 with target reserved word counts and a median deviation of 3 words, yet ablations confirm reserved word seeding still improves SQL complexity.

• Filtering does not collapse difficulty. The paper reports that filtered examples' SQL and NL complexity metrics match BIRD benchmarks, and complexity remains consistent across models without declining after filtering — in fact, it improves.

Methodology in Plain English

FALCON generates data rather than collecting it. It starts from existing table schemas (WikiSQL, Spider) and runs a five-stage pipeline:

1. SQL generation with seeding. Before generating each query, the pipeline injects either a target count of SQL reserved words sampled from BIRD's distribution or a generated user persona (sampled to be either a common professional type or a creative niche type, with goals and example questions). Chain-of-thought prompting produces the query, which must execute against the database within up to three attempts.

2. NL generation with judging. Four candidate questions are generated from the validated SQL and schema, and an LLM-as-a-judge Compare Judge picks the best one.

3. Ambiguity detection and multi-interpretation. A DistilBERT scorer trained on Ambrosia data gives each pair a 0-10 ambiguity score. Scores above 7 enter a branch where a prompt filter confirms genuine ambiguity, the ambiguity type is identified, and a second valid SQL interpretation is generated and validated.

4. Alignment filtering. A ModernBERT-based reward model scores how well the question grounds onto the schema, retaining examples at or above 0.6. Ambiguous examples bypass this model and use prompt-based filters instead.

5. Final prompt filter. A last prompt checks that the SQL validly interprets the NL, that the SQL is sensible and non-redundant, and that the NL sounds like something a human would ask.

Evaluation combines purpose-built complexity metrics (SQL complexity via SQLGlot parsing of words, columns, joins, sub-queries, operators, and splits; NL richness from tokens before the main verb, constituents per word, subordinate and coordinate clauses), human annotation with a rubric grading SQL, NL, and alignment separately, and downstream fine-tuning of Qwen 2.5 3B and 1.5B on 5,500 synthetic examples, 5,500 Spider examples, or a mixed set (95% synthetic, 5% Spider), tested on a 2,000-example synthetic test set and the 2,147-example Spider test set using Exact Match, Fuzzy Score, and Soft Execution Accuracy.

Why This Matters

Impact on research. The paper argues that synthetic NL-to-SQL generation has been stuck in a trade-off between expensive frontier-model APIs and oversimplified queries, and that ambiguity has been largely ignored by synthetic datasets. FALCON offers a unified alternative and a difficulty-stratified evaluation protocol showing that trained models' advantage grows with query complexity, which reframes evaluation from average accuracy toward performance on hard queries.

Real-world applications:

• Enterprises generating domain-specific NL-to-SQL training data for internal schemas without sending proprietary schemas to third-party APIs.
• Organizations deploying natural-language database interfaces for non-SQL experts, where ambiguous or underspecified questions are common.
• Teams that need a small fine-tuned model rather than a continuously prompted frontier model, given the reported cost and privacy advantages.
• Multilingual or domain-adapted variants built on the database-agnostic pipeline described in the paper.

Industry relevance. The paper explicitly targets the academic-industry gap: it cites cost barriers to continuous deployment, privacy concerns around transmission of proprietary schemas, and production-level database complexity as reasons enterprises cannot simply adopt benchmark-trained systems. Its reported local generation cost (roughly $20 versus an estimated $326, an over 16x reduction) and its requirement that only compact open models be used speak directly to local, on-premise deployment.

Future Directions

• Close the simple-query gap. Models trained solely on FALCON data achieve only 0.8% EM on Spider, and the paper states 5-10% real data supplementation is needed for broad applicability; the limitations section also cites underrepresentation of simpler queries.
• Improve ambiguity recall. Ambiguity detection achieves 98.9% specificity but only 28.6% recall, so many genuinely ambiguous queries are missed; raising recall without generating noisy multi-interpretation pairs is an open problem.
• Extend beyond SQLite. FALCON currently supports SQLite only; other dialects would require dialect-specific reserved word distributions and syntax constraints.
• Broaden evaluation and data coverage. The paper notes that evaluation is limited to public benchmarks (Spider, WikiSQL, BIRD), that production databases may exhibit complexity dimensions not captured academically, and that generated data inherits domain coverage from seed databases.

Target Audience

Researchers and engineers working on text-to-SQL, synthetic data generation, and fine-tuning of compact language models will benefit most. The paper is also relevant to practitioners in industry who need privacy-preserving, locally generated training data for enterprise database schemas, and to those studying ambiguity handling in natural language interfaces to structured knowledge. Readers without background in NL-to-SQL benchmarks, execution accuracy metrics, or prompt-based filtering will need to consult the cited prior work for context.

Authors’ abstract

Relational databases are among the most widely deployed forms of structured knowledge, and natural language access to them requires grounding language onto schema entities and relations while handling the ambiguity inherent in how people phrase requests. Existing synthetic NL-to-SQL data generation methods largely ignore this ambiguity and produce oversimplified queries that fail to prepare models for the complexity of real-world structured knowledge access. We present FALCON, a framework that generates realistic, ambiguity-aware NL-to-SQL data matching the complexity of challenging real-world benchmarks, at low cost using compact open models. Our approach combines reserved-word SQL seeding and persona-based prompting to generate structurally complex queries, while alignment-based filtering preserves difficulty by distinguishing genuinely incorrect examples from complex but valid queries. Human evaluation confirms consistent high quality across model sizes, and our generated data exceeds existing benchmarks in both SQL complexity and natural language richness. Difficulty-stratified analysis shows models trained on FALCON data increasingly outperform baseline-trained models as query complexity increases, validating our pipeline's success in generating challenging training data. When combined with a small proportion of existing benchmark data, mixed training recovers performance on simpler queries while preserving these advantages on complex ones. The model- and database-agnostic design enables organizations to generate high-complexity NL-to-SQL training data locally without external APIs.