ARCS: Towards Precise Text-to-SQL via Structured Disambiguation

AI summary

Overview

Research area: Natural Language Processing / Text-to-SQL, specifically ambiguity handling and human–AI interaction design for database query systems. Technical level: Advanced — the paper assumes familiarity with SQL, database schemas, LLM benchmarking, and execution-accuracy metrics. Scope: The paper proposes "structured disambiguation" (resolving ambiguous questions through multiple-choice selections and value sliders rather than free-form dialogue) and introduces ARCS, the first text-to-SQL benchmark built from naturally occurring ambiguities over unmodified real-world databases.

What This Paper Is About

Text-to-SQL systems look strong on existing benchmarks, but those benchmarks use unambiguous, often over-specified questions that do not resemble how real users ask questions. In practice, questions are subtly ambiguous in domain- and data-specific ways ("How many GitHub issues are opened in 2025?" — newly created only, or also reopened?), and these ambiguities silently push system outputs away from user intent. The paper's goal is to make that ambiguity explicit and cheap to resolve, and to provide a benchmark that measures it rigorously.

Key Contributions

1. Structured disambiguation: a new interaction paradigm in which ambiguity is resolved by constrained recognition — multiple-choice selection when interpretations are finite, and sliders plus operator choice (e.g., > vs. ≥) when the interpretation space is unbounded — drawing on the HCI principle of "recognition rather than recall."
2. ARCS (Ambiguity Resolution Corpus for SQL): a benchmark of 311 realistic disambiguation tasks over six real databases, with complete annotations of all valid ambiguity points, all interpretations, and their corresponding SQL queries.
3. A two-dimensional ambiguity taxonomy: a linguistic dimension (semantic vs. syntactic ambiguity) and an orthogonal database dimension (column, table, value, computation ambiguity), designed to separate "parsing errors" from "schema alignment errors."
4. Extensive evaluation: 23 LLMs (8 open-source, 15 proprietary) under both an end-to-end setting with an LLM user simulator and a fine-grained setting that disentangles disambiguation from SQL generation.

Main Findings

• Ambiguity remains a major challenge: the best end-to-end execution accuracy is 57.88% (gpt-5, medium reasoning effort), while gpt-4.1 reaches only 29.90%; gpt-6-sol reaches 51.77%. The best open-source model, gptoss-120b, reaches 26.05%, and no open-source model exceeds 27%.
• Disambiguation, not SQL writing, is the bottleneck: across all models, disambiguation scores are substantially lower than SQL generation scores. All models except gpt-5 and gpt-6-sol identify all ambiguity points and interpretations for at most 30% of tasks, and no model exceeds a Perfect Disambiguation score of 14%.
• Similar SQL skill, very different disambiguation skill: gpt-5, claude-opus-4.5 and gemini-3-pro have comparable execution accuracy given ground-truth disambiguation (65.27%–68.81%), but their Full Recall spans 22.51%–61.09% — suggesting disambiguation is underrepresented in LLM training objectives.
• Reasoning effort helps, with saturation: gpt-5 improves from 37.62% (minimal) to 57.88% (medium) with no further gains at higher settings; o4-mini improves from 36.01% (low) to 44.05% (high). The effect is more pronounced for disambiguation than for SQL generation.
• Structured disambiguation is more efficient: with o4-mini (medium), structured disambiguation reaches 42.44% EX at user effort 18 and cost $0.03, versus conversational disambiguation at 43.73% EX with effort 34 and cost $0.03, and unstructured interpretation generation at 28.94% EX with effort 39 and cost $0.04. Lowering user patience in the conversational setting (patience = 1, 3, 5 → 31.51%, 39.55%, 39.87% EX) degrades it below structured disambiguation.
• Unstructured enumeration degrades with multiple ambiguities: it performs comparably on single-ambiguity questions but markedly worse when several ambiguity points interact.
• ARCS resists prompt heuristics: supplying the taxonomy raises o4-mini from 52.25% to 80.00% on Ambrosia (a 400-sample subset) but only from 42.44% to 46.30% on ARCS; gpt-4.1 goes from 48.75% to 63.25% on Ambrosia while dropping from 29.90% to 28.30% on ARCS.
• Dominant errors are unresolved ambiguity and wrong SQL formulation: qwen3-coder-480b produces invalid outputs on 18% of tasks, but for all three analyzed models most errors are executable queries that match no ground truth, or match a ground truth other than the user-intended one. Named failure modes include spurious ambiguity, failure to use database context to bound the ambiguity, and over-fragmentation of equivalent interpretations.
• Computation ambiguity is hardest to detect: all three analyzed models perform poorly on it; weaker models also miss syntactic ambiguities more often than semantic ones.
• User study agreement: in a NASA-TLX study, participants reported lower mental demand, effort and frustration with structured disambiguation, and rated it more transparent.

Methodology in Plain English

The authors started from real databases rather than inventing toy ones: five databases from BIRD and one from Spider 2.0, converted to SQLite, totalling 8.3 GB and spanning banking, retail, student activities, NBA, Stack Exchange and GitHub data. They used gpt-4.1, prompted with the schema, the taxonomy and a few hand-written exemplars, to draft 180 questions per database at temperature 0.7; the model was free to invent ambiguities beyond the taxonomy. The authors themselves then acted as annotators — a task requiring both linguistic and database expertise — selecting realistic, genuinely ambiguous questions, marking every ambiguity point, enumerating all valid interpretations, and writing a parameterized SQL query for every interpretation combination. Annotating one question took about 2.5 hours. A first annotation round had errors in over 60% of questions; after a second round with finalized guidelines and a final verification pass, fewer than 2% of questions contained errors.

To evaluate systems without a human in the loop, they built an LLM user simulator exposing an ask_user interface that supports free text, multiple-choice and value selection. Because a naive simulator leaked information about other ambiguity points, they used a two-stage design: first identify which ambiguity points the question concerns, then answer using only those. This produced 331 task instances from 101 unique questions, with 413 unique SQL queries (the contributions section and appendix report 311 tasks). Evaluation runs in two modes: end-to-end, where any agent architecture may interact with the database and simulator freely before emitting SQL, and fine-grained, where disambiguation quality (Full Recall, Perfect) and SQL generation quality given gold disambiguation (EX_disambiguated) are scored separately. Predicted queries are allowed to return extra columns, so a model that answers all interpretations in one table scores as correct with zero user effort.

Why This Matters

• Research impact: ARCS reframes text-to-SQL ambiguity as a recognition task with a measurable interface, and its fully annotated ambiguity points let researchers separate disambiguation errors from SQL errors — something prior ambiguous benchmarks (NoisySP, AmbiQT, Ambrosia) do not support, since they inject ambiguity into modified or synthesized databases and cap questions at one ambiguity point.
• Business intelligence and reporting: "total sales revenue" genuinely differs depending on how refunded, canceled or partially returned orders are counted — exactly the kind of ambiguity that silently corrupts budget-setting decisions.
• Engineering and developer analytics: counting "issues opened in 2025" (created only vs. also reopened) or interpreting "created pull requests" changes reported metrics materially.
• Customer-facing data assistants: multiple-choice panels and sliders resolve ambiguity faster than a clarification dialogue, which matters for users who cannot articulate schema constraints.
• Industry relevance: the benchmark spans enterprise-scale schemas (average of 133.83 tables, 1205 columns and 1,969,774 rows per database), and the disambiguation and SQL-generation components are explicitly implementation-agnostic, so any text-to-SQL system can be swapped in.

Future Directions

• Closing the disambiguation gap: since frontier models differ widely in Full Recall (22.51%–61.09%) despite similar SQL accuracy, the authors suggest disambiguation is an underrepresented training objective; how to train for it is left open.
• Handling computation ambiguity: it is the weakest category for every analyzed model, and no targeted remedy is proposed.
• Improving ambiguity detection quality: the identified failure modes — spurious ambiguity, ignoring database context to bound interpretation sets, and splitting semantically identical interpretations — point to concrete error classes needing better modeling.
• Validating with real users at scale: the reported human evidence is a NASA-TLX user study and an LLM simulator; whether structured disambiguation holds up in deployed workflows, and how to scale annotation beyond 101 unique questions given 2.5 hours per question, remain open.

Target Audience

Researchers and practitioners working on text-to-SQL, LLM agents and tool use, and natural-language interfaces to databases; benchmark designers interested in ambiguity, underspecification and human-in-the-loop evaluation; and product engineers building enterprise analytics assistants who need to decide between clarification dialogue and constrained UI-based disambiguation. Readers should already be comfortable with SQL semantics, schema grounding, and execution-accuracy evaluation.

Authors’ abstract

As text-to-SQL systems move beyond demonstrations toward real-world deployment, ambiguity in user questions becomes a primary source of errors. Such ambiguities are often subtle, domain- or data-specific, and can silently cause system outputs to deviate from the user's true intent. Ambiguity is traditionally addressed through conversational clarification, which is often inefficient, cognitively demanding, and poorly aligned with real-world user workflows. We propose structured disambiguation, a new paradigm in which ambiguity is resolved through explicit, constrained interactions rather than free-form dialogue. We construct ARCS (Ambiguity Resolution Corpus for SQL), the first text-to-SQL benchmark featuring naturally occurring, unconstrained ambiguities over real-world databases, with complete annotations of all valid ambiguity points, interpretations, and SQL queries. Experimental results show that text-to-SQL remains challenging in the presence of ambiguity: gpt-6-sol achieves only 51% end-to-end execution accuracy, and no open-source model exceeds 27%.