SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps

AI summary

Overview

Research area: Natural Language Processing — text-to-SQL (NL2SQL) evaluation, model robustness, and interpretable dataset representation. The work sits at the intersection of LLM evaluation and benchmark analysis.

Technical level: Intermediate. Readers should be comfortable with concepts such as execution accuracy, K-means clustering, UMAP projection, random forests, and prompt-based feature extraction with LLMs.

One-sentence scope: The paper introduces SQLSpace, a pipeline that converts any text-to-SQL example into a compact, human-interpretable binary feature vector (187 features spanning five aspects), and demonstrates three uses of those vectors: comparing benchmark composition, locating model "blind spots," and rewriting natural language questions to boost accuracy.

What This Paper Is About

Text-to-SQL systems are typically ranked by overall execution accuracy on benchmarks such as BIRD (Bird-Bench) or Spider, but a single accuracy number hides why one model beats another, which kinds of questions every model fails on, and how a benchmark's composition differs from a practitioner's own data. This paper builds SQLSpace, a general-purpose representation of NL2SQL examples made of interpretable natural language features discovered with minimal human intervention, and uses it to expose and then partially close those robustness gaps. The paper reports comparative analysis of three datasets, per-cluster performance evaluations of 13 instruction-tuned models, and a proof-of-concept rewriting experiment on Bird-Dev.

Key Contributions

1. A representation-construction pipeline that produces |P|-dimensional (187-dimensional) binary feature vectors for any NL2SQL example, with four steps: aspect-based description generation, feature discovery, feature deduplication, and example representation construction. The representations are dataset-agnostic and require minimal human intervention.
2. A benchmark-composition analysis that statistically compares Spider-Dev, Bird-Dev, and Spider-Realistic, showing that SQLSpace surfaces dimensions along which datasets differ (using chi-square tests over feature proportions) and that these differences align with how the datasets were constructed.
3. A blind-spot discovery method that clusters example feature vectors across benchmarks (K-means, k=14) and reports per-cluster execution accuracy for 13 models, identifying clusters where models systematically underperform and clusters where cheaper or open-source models match expensive or proprietary ones.
4. A proof-of-concept correctness-estimation and rewriting application in which a random forest estimator predicts likely failures and an LLM rewriter removes question features associated with those failures, improving Bird-Dev execution accuracy for two open-source models.

Main Findings

• Feature set size and composition: Deduplication and manual cleanup yield 187 general-purpose predicates: Syntax 74 (40%), SQL 41 (22%), Example Syntax 27 (14%), Pragmatics 33 (18%), and Database Reasoning 17 (9%).
• Evaluator reliability: Human spot checks over 250 example-predicate pairs (50 per aspect) gave an average Evaluator accuracy of 73%, with inter-annotator agreement of Cohen's kappa = 60.2, described as substantial agreement.
• Benchmarks differ measurably: Bird-Dev contains more syntactically complex examples than Spider-Realistic — 20% of examples include mixed use of symbols and words, compared to 5% in Spider-Dev and 6% in Spider-Realistic. Spider-Realistic also shows the lowest presence of a question-schema alignment feature, consistent with its construction by paraphrasing away column-name mentions, and the highest proportion of examples trading the Gricean maxim of quantity for brevity.
• Two universal blind spots: Clusters C8 and C10 were difficult for all models, including the strongest model overall, gpt-4o. C8 contains examples with complex conditional expressions and technical jargon; C1 contains single-column filters and aliases.
• Two universally easy clusters: All models performed well above their overall accuracy on C1 and C12.
• Cheaper models can match expensive ones on some clusters: granite-code-3b performs on par with deepseek-coder-7b on C4 and exceeds it on C1; deepseek-coder-1.3b (the smallest model tested) outperforms granite-code-3b on C13 and performs on par on C14; code-gemma-7b performs on par with gpt-4o on C1.
• General-purpose models can beat code models on some clusters: gemma-7b exceeds code-gemma-7b on C4 and C5, while performing lower on all other clusters, suggesting some clusters require linguistic or reasoning skills rather than code-generation ability.
• Target data can land in weak clusters: Running K-means inference on Spider-Realistic (the stand-in for a practitioner's own data), only 2.6% of examples mapped to C11, the highest-performing cluster, while the majority mapped to C6, where many models struggle.
• Rewriting improves accuracy: On Bird-Dev, rewriting questions with the top three negative features raised execution accuracy from 31.2 to 37.7 for code-gemma-7b, and from 23.5 to 30.8 for deepseek-coder-7b. Accuracy with the top single negative feature was 31.9 and 24.3 respectively.
• Dataset-level accuracy varies widely: Reported execution accuracy figures place gpt-4o at 43.7 on Bird-Dev and 76.1 on Spider-Dev, while llama-2-7b scores 3.1 and 21.5 respectively — the same model's accuracy differs substantially across benchmarks.
• Overlap between benchmarks is concentrated in "simple" examples: Regions of high overlap between Bird-Dev and Spider-Dev largely correspond to Bird-Dev examples annotated as "simple" in its released difficulty metadata.

Methodology in Plain English

The researchers wanted a way to describe a text-to-SQL example that is readable by a human, applies to any dataset, and does not require an army of annotators.

They start with a large mixed collection of examples: 10,697 examples drawn from the development portion of the Unite corpus, which itself collates Spider, Squall, Spider-Syn, Criteria2SQL, SparC, CoSQL, Spider-DK, ParaphraseBench, KaggleDBQA, ACL-SQL, SEOSS-Queries, and FIBEN.

Then they run a four-step pipeline. First, an LLM (gpt-4o-2024-05-13, called the Describer) writes five free-text descriptions of each example, one per aspect: the syntax of the natural language question, the syntax and complexity of the SQL query, the semantics linking question to query, pragmatics such as speech acts and ambiguity, and the reasoning needed to map the question onto the database schema. Second, other LLMs (gpt-3.5-turbo-0125 and gpt-4o-2024-08-06, called Proposers) read those descriptions in batches and propose candidate yes/no predicates — statements like "contains a subquery" — generating 40 predicates per iteration over 5 iterations, then repeating across aspects. Third, duplicates such as "contains a nested JOIN" versus "uses nested JOINS" are removed using Levenshtein-based token-set similarity (thefuzz, threshold epsilon = 70) plus a manual pass. Fourth, an Evaluator LLM (gpt-4o-2024-08-05) checks each of the 187 remaining predicates against each example, turning every example into a binary vector.

With those vectors in hand, the authors do three things. They build vectors for Spider-Dev (1,034 examples), Bird-Dev (1,534 examples), and Spider-Realistic (508 examples), project them with UMAP (n_neighbors = 50, min_dist = 0.01), and use chi-square tests on per-feature proportions to find where datasets genuinely differ. They pool Spider-Dev and Bird-Dev, run K-means with k = 14 chosen by the elbow method, and measure each of 13 models' execution accuracy per cluster, using a random forest with 100 estimators and mean decrease in impurity to characterize what makes each cluster distinctive. Finally, they train a random forest correctness estimator (200 estimators, trained on 90% of Unite-Dev, feature importance measured on the remaining 10%) using only the 121 question- and schema-derived features, use permutation importance on negative examples to rank failure-associated features, and have gpt-4o-2024-08-06 rewrite questions that the estimator flags.

All 13 models — gemma-7b, code-gemma-7b, llama-2-7b, llama-2-13b, granite-code-3b, granite-code-8b, deepseek-coder-1.3b, deepseek-coder-7b, code-llama-7b, code-llama-13b, gpt-4o, gpt-4o-mini, and gpt-3.5-turbo — were run zero-shot on both development sets, with the schema plus three example rows of values provided in the prompt.

Why This Matters

Impact on research. SQLSpace offers a shared coordinate system in which benchmarks and models can be compared on interpretable dimensions rather than single accuracy numbers. It also reframes robustness benchmarking: instead of hand-designing perturbations from human priors, researchers can let discovered feature clusters reveal where models fail, including failures on combinations of properties that targeted perturbation benchmarks may miss. The authors release the 187-predicate set and plan to release the Bird-Dev and Spider-Dev clusters as community artifacts.

Real-world applications.

• Model selection and cost control: A team with its own dataset can map that data onto the discovered clusters and pick the cheapest model that performs well on the clusters it actually resembles — since deepseek-coder-1.3b and granite-code-3b beat or match larger models on specific clusters.
• Diagnosing puzzling accuracy drops: When the same model scores very differently on two benchmarks, the composition analysis shows which features are over-represented, giving a concrete explanation instead of a leaderboard mystery.
• Inference-time intervention: The correctness estimator plus rewriter can warn a user that a question is likely to fail and suggest a paraphrase that avoids the problematic feature.
• Challenge-split construction and data augmentation: Low-performing clusters can be turned into hard test splits, and benchmarks resembling a target dataset can be identified for augmentation.

Industry relevance. Any company deploying a natural-language-to-SQL interface (analytics tools, BI copilots, database assistants) inherits exactly the failure modes this paper surfaces: brittle handling of nested conditionals, jargon, and language that departs from exact column names. The per-cluster cost analysis is directly actionable for teams balancing inference cost against accuracy.

Future Directions

• Varying the Describer. The authors note SQLSpace can run with any Describer, open-source or proprietary, and suggest future work explore how different Describers affect the discovered features.
• Cheaper, distilled evaluators. The paper evaluates predicates with a large closed-source model for proof-of-concept and proposes fine-tuning lighter-weight models for the predicate-evaluation task.
• Extending to adversarial test-set generation. The authors argue blind spots could be used to generate new challenge examples expressing combinations of properties that models systematically struggle with, rather than only analyzing existing ones.
• Deeper study of feature modulation and rewriting cost. The paper points to an appendix discussion of feature modulation and the cost of rewriting, leaving the economics and the limits of rewriting as open territory.
• Open question on cluster stability. Because clusters are dataset-dependent and produced with inference data, how well the proposed Bird-Dev and Spider-Dev clusters transfer to arbitrary new datasets remains to be established.

Target Audience

NL2SQL and text-to-SQL researchers, benchmark designers working on robustness and challenge sets, LLM evaluation scientists interested in interpretable representations beyond embeddings, and applied engineers or product teams deciding which model to deploy for a natural-language database interface. Readers focused purely on model architecture or training methods will find less here; the paper's value is in evaluation, analysis, and inference-time intervention.

Authors’ abstract

We introduce SQLSpace, a human-interpretable, generalizable, compact representation for text-to-SQL examples derived with minimal human intervention. We demonstrate the utility of these representations in evaluation with three use cases: (i) closely comparing and contrasting the composition of popular text-to-SQL benchmarks to identify unique dimensions of examples they evaluate, (ii) understanding model performance at a granular level beyond overall accuracy scores, and (iii) improving model performance through targeted query rewriting based on learned correctness estimation. We show that SQLSpace enables analysis that would be difficult with raw examples alone: it reveals compositional differences between benchmarks, exposes performance patterns obscured by accuracy alone, and supports modeling of query success.