BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation

AI summary

Overview

• Research area: Natural language processing, specifically text-to-SQL generation, human-in-the-loop annotation tooling, and enterprise data systems.
• Technical level: Intermediate. The paper is readable without deep database internals, but familiarity with text-to-SQL benchmarks, retrieval-augmented generation (RAG), and basic evaluation metrics helps.
• Scope: The paper presents BenchPress, a system that helps organizations turn raw enterprise SQL logs into natural language annotations so they can build their own domain-specific text-to-SQL benchmarks, and reports a controlled 18-participant user study comparing it against manual annotation and a plain off-the-shelf LLM.

What This Paper Is About

Large language models score well on public text-to-SQL benchmarks such as Spider, Bird, and Fiben, but their execution accuracy drops sharply on private enterprise data, as shown by the authors' earlier Beaver benchmark. Building a comparable benchmark for a company's own warehouse requires pairing each logged SQL query with a natural language question, and asking highly trained database administrators to write and validate those questions is costly and slow. BenchPress addresses that bottleneck by having LLMs propose candidate natural language descriptions for a given SQL query while human experts select, rank, or edit them.

Key Contributions

1. The BenchPress system itself. A human-in-the-loop annotation interface that combines retrieval-augmented generation (RAG), prompt-based LLM output, and iterative feedback to produce natural language descriptions from SQL queries. It supports a one-time setup phase and a repeating annotation loop, with optional decomposition into Common Table Expressions (CTEs) and recomposition for nested queries.

2. A concrete workflow and design toolkit. Key design features include dense-vector retrieval of semantically similar prior annotations (via Sentence-BERT), schema retrieval through SQL parsing (e.g., sqlglot) or embedding search, structured prompt templates that annotators can refine, and a privacy-aware setup where the API key stays in the user's browser storage while large SQL logs and schemas are stored server-side for RAG.

3. A controlled user study of annotation performance. Eighteen participants, stratified into advanced and non-advanced SQL users, were randomly assigned to one of three conditions — BenchPress, Manual, or Vanilla LLM — using a balanced Latin square design, and each annotated the same 30 SQL queries sampled from Beaver and Bird.

4. A quantification of enterprise-vs-public benchmark difficulty. Tables 1 and 2 contrast query-level and data-level complexity metrics for Beaver's Datawarehouse (DW) against Spider, Fiben, and Bird, documenting why annotation and model performance differ in enterprise settings.

Main Findings

• BenchPress produced the most accurate annotations. Overall average accuracy was 93.0% for BenchPress, 83.1% for the Vanilla LLM condition, and 73.9% for Manual. On Beaver the gap widened: 86.1% (BenchPress), 66.2% (Vanilla LLM), 60.1% (Manual). On Bird, both BenchPress and the Vanilla LLM reached 100.0%, with Manual at 87.8%.

• BenchPress was also the fastest. Total average annotation latency was 28.1 minutes for BenchPress, 32.0 minutes for Vanilla LLM, and 183.9 minutes for Manual. On Beaver specifically: 16.1 minutes (BenchPress), 16.2 minutes (Vanilla LLM), and 102.1 minutes (Manual); on Bird: 12.0, 15.8, and 82.8 minutes respectively.

• Backtranslation fidelity favored BenchPress. An LLM regenerated SQL from the natural language descriptions alone, and the outputs were graded on a 5-level rubric ranging from Level 1 (Invalid) to Level 5 (Fully Correct). BenchPress yielded the highest proportion of Level 5 outputs, while the Manual and Vanilla LLM groups more often landed at Level 4, typically from row-order inconsistencies, omitted nuances, or subtle misinterpretations of intent.

• Enterprise queries are structurally more demanding. Beaver's Datawarehouse averages 15.6 keywords, 99.8 tokens, 4.2 tables, 11.9 columns, 5.5 aggregations, and 2.05 nestings per query. Relative to that baseline, Spider has 83.6% fewer aggregations and 75.6% fewer columns, and Bird has 87.3% fewer aggregations, while Fiben has 62.2% more tokens.

• Schema and data characteristics differ too. Beaver's DW has 15.6 columns per table on average, 128K rows per table, 99 tables per database, 45.9% uniqueness, 15.0% sparsity, and 4 data types. Bird shows 328.9% more rows per table relative to that baseline, while Fiben shows 84.0% fewer columns per table and 100% more data types; sparsity is 100% lower in Spider, Fiben, and Bird.

• The enterprise gap is large. The paper reports that state-of-the-art LLMs suffer an execution accuracy drop of up to 70–90% on enterprise SQL logs despite strong results on public datasets, and that the best-performing model differs across datasets (Spider: miniSeek; Bird: askData; Fiben: Athena++; Beaver: contextModel).

• Cold start shapes the workflow. At the start of the study no prior annotations existed, so the first annotation received no example-based RAG guidance; later annotations benefited from examples created earlier in the session. The authors note this mirrors real enterprise deployments and that future versions may offer initializing from public text-to-SQL datasets.

• Public versus enterprise difficulty diverged in the study. All three conditions performed reasonably well on Bird, but Beaver — with higher schema complexity, ambiguous column names, and enterprise-specific terminology — exposed substantial differences between conditions.

Methodology in Plain English

The authors started from an existing observation: their earlier Beaver benchmark showed LLMs doing far worse on private enterprise SQL logs than on public datasets, and building more such benchmarks by hand is expensive.

BenchPress works in two stages. In a one-time setup, a user configures a project (the API key is kept entirely in the browser's storage), uploads SQL logs and schema files or picks one of four supported benchmarks (Bird, Fiben, Spider, or Beaver), and chooses the annotation task — currently only SQL-to-NL — plus the language model (e.g., GPT-4o, GPT-3.5 Turbo, or DeepSeek). SQL logs and schemas are large, and browser storage limits such as the roughly 5–10 MB per origin for localStorage cannot reliably hold them, so they are stored on the server, which also lets RAG search the full document set.

Then each query enters an annotation loop. Nested queries are optionally rewritten into CTEs (decomposition). The system retrieves semantically similar prior annotated queries using dense vector embeddings (e.g., Sentence-BERT) and retrieves the relevant tables and columns from the schema, either by parsing the SQL or via the same embedding-based retrieval. A selected LLM then produces four candidate natural language descriptions per query or subquery using retrieval-augmented, few-shot prompting; the system always includes the relevant tables but suggests only the top-k retrieved examples to keep prompts efficient. For nested queries, subquery-level descriptions are merged back into one explanation (recomposition). Human annotators then rank, refine, discard, or assign priorities to the candidates, and can inject external domain knowledge or flag common failure patterns. Outputs can be scored with automatic metrics such as exact match or BLEU when ground truth exists, and final annotations are exported in JSON benchmark-ready format.

To test the design, the team ran a between-subjects user study to avoid learning or fatigue effects carrying across interfaces. Eighteen participants were first split into advanced and non-advanced SQL users based on a pre-study questionnaire, then randomly assigned within each stratum to one of three conditions. Group A used BenchPress with schema information, example tables, logs, and four LLM-generated suggestions per query. Group B had only schema files and logs. Group C could use a general-purpose LLM such as ChatGPT through its standard interface, with no RAG support or task integration. Everyone annotated the same 30 SQL queries drawn from Beaver and Bird. Accuracy was judged by manual inspection of whether key SQL components — column selections, aggregations, grouping, and ordering — were clearly described; latency was measured as total annotation time; and fidelity was tested by having an LLM regenerate SQL from the natural language alone and grading the results on the 5-level rubric.

Why This Matters

The paper reframes benchmark creation itself as the bottleneck in enterprise text-to-SQL adoption. Public benchmarks are plentiful, but companies cannot use them to answer the question that matters most: how well does a model perform on our schema, our terminology, and our query patterns? BenchPress targets that gap directly, and its user study gives concrete evidence that LLM suggestions plus light human oversight can cut annotation time dramatically while improving quality — total annotation latency dropped from 183.9 minutes under manual annotation to 28.1 minutes with BenchPress, with accuracy rising from 73.9% to 93.0% overall.

Real-world applications:

• Enterprise model selection before deployment. Organizations can benchmark candidate text-to-SQL models against their own private warehouse rather than relying on public leaderboards that may overstate readiness.
• Targeted fine-tuning. Identifying failure cases on in-domain data helps teams decide where to collect more data or adjust training.
• Interpretability and audit of model outputs. Systematically scoring how well generated queries match intent supports review processes in regulated or sensitive settings.
• Domain adaptation across industries. The paper points to finance, healthcare, IT, education, technology, and manufacturing as domains with specialized vocabulary and abbreviations absent from public datasets.

Industry relevance: The work came out of a collaboration spanning TU Munich, MIT, Intel Labs, and AWS AI Labs, and was supported by Intel as part of the MIT Data Systems and AI Lab (DSAIL). It addresses a practical constraint that pure research benchmarks sidestep: enterprise data often contains sensitive information such as employee salaries or internal performance metrics and cannot be publicly released, so evaluation must happen in a secure, private environment.

Future Directions

• Adding text-to-SQL generation for iterative validation. The current system focuses only on SQL-to-NL annotation; generating SQL back from annotations would create a tighter validation loop and, the authors argue, further increase accuracy and speed of benchmark curation.
• Stress-testing public benchmark models for overfitting. Because many models achieve near-perfect performance on Spider, Bird, and Fiben, the authors plan to systematically rephrase the natural language queries in existing benchmarks into more realistic, ambiguous, or underspecified variants and re-evaluate, testing whether current benchmarks reward genuine generalization or surface-level pattern matching.
• Better tooling for custom benchmarks on private data. Creating domain-specific benchmarks with private data remains a significant bottleneck for LLM adoption more broadly, and the authors frame BenchPress as one step toward solving it.
• Warm-start options. The paper notes BenchPress assumes a cold start by default, and that future enterprise use cases could offer both starting from an empty knowledge base or initializing the system with examples from public text-to-SQL datasets.

Target Audience

This paper is most useful to researchers and practitioners working on text-to-SQL evaluation, enterprise LLM deployment, and human-in-the-loop data annotation. Data platform and ML infrastructure teams in companies with large private warehouses will find the workflow and the reported time savings directly relevant, as will HCI researchers interested in how LLM suggestions change annotator behavior. Benchmark builders and dataset curators who need to produce domain-specific evaluation sets under privacy constraints are the core audience, and the tool is freely available via the authors' public GitHub repository and website.

Authors’ abstract

Large language models (LLMs) have been successfully applied to many tasks, including text-to-SQL generation. However, much of this work has focused on publicly available datasets, such as Fiben, Spider, and Bird. Our earlier work showed that LLMs are much less effective in querying large private enterprise data warehouses and released Beaver, the first private enterprise text-to-SQL benchmark. To create Beaver, we leveraged SQL logs, which are often readily available. However, manually annotating these logs to identify which natural language questions they answer is a daunting task. Asking database administrators, who are highly trained experts, to take on additional work to construct and validate corresponding natural language utterances is not only challenging but also quite costly. To address this challenge, we introduce BenchPress, a human-in-the-loop system designed to accelerate the creation of domain-specific text-to-SQL benchmarks. Given a SQL query, BenchPress uses retrieval-augmented generation (RAG) and LLMs to propose multiple natural language descriptions. Human experts then select, rank, or edit these drafts to ensure accuracy and domain alignment. We evaluated BenchPress on annotated enterprise SQL logs, demonstrating that LLM-assisted annotation drastically reduces the time and effort required to create high-quality benchmarks. Our results show that combining human verification with LLM-generated suggestions enhances annotation accuracy, benchmark reliability, and model evaluation robustness. By streamlining the creation of custom benchmarks, BenchPress offers researchers and practitioners a mechanism for assessing text-to-SQL models on a given domain-specific workload. BenchPress is freely available via our public GitHub repository at https://github.com/fabian-wenz/enterprise-txt2sql and is also accessible on our website at http://dsg-mcgraw.csail.mit.edu:5000.