Agent Bain vs. Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domain

AI summary

Overview

Research area: Natural Language Processing — text-to-SQL benchmarking, business intelligence (BI), and LLM-based evaluation.

Technical level: Intermediate. The paper assumes familiarity with text-to-SQL tasks, SQL execution metrics (SER, EA), and LLM-as-judge evaluation, but explains its benchmark construction and scoring design in accessible terms.

Scope: The paper introduces CORGI, a business-domain text-to-SQL benchmark of 10 synthetic enterprise databases and four escalating question types, paired with a six-dimension, multi-agent evaluation framework for open-ended business answers.

What This Paper Is About

Traditional text-to-SQL benchmarks treat the task as pure translation: turn a natural language question into a SQL query that retrieves past records. The authors argue this misses what real business users actually ask for — questions that require explanation, forecasting, and strategic recommendation, not just retrieval. CORGI is their attempt to redefine the benchmark around those richer queries, and to build an automatic way to grade the qualitative, free-text answers that such queries produce.

Key Contributions

1. The CORGI benchmark. A business-domain text-to-SQL benchmark built on 10 simulated enterprise databases across three broad categories (consumer platforms, retail and commerce, digital services), with questions split into four types: descriptive, explanatory, predictive, and recommendational.
2. A business-literature-inspired evaluation framework. Six scoring dimensions — Structure, Data Sense, Insightfulness, Operational Implementability, Purpose Alignment, and Compliance — developed collaboratively with business school faculty, professional consultants, and corporate practitioners.
3. An atomized multi-agent evaluation mechanism. One discriminator plus six scoring agents, where the discriminator routes each input to the dimensions relevant to its question type; the final score is the average of applicable dimension scores.
4. Preliminary experiments across five generator models, showing performance degradation on higher-level business questions and releasing the dataset, code, evaluation questions, evaluation framework, and a submission website on GitHub.

Main Findings

• Large drop against existing benchmarks: LLMs show an average 33.12% lower success execution rate (SER) on CORGI compared to existing benchmarks such as BIRD.
• Monotonic SER decline by question type: Average SER falls from 79.47% (type 1) to 59.16% (type 2), 58.14% (type 3), and 47.33% (type 4). Type 1 uses a single SQL query; types 2–4 allow up to five generated queries.
• Closed-source models dominate simple retrieval: On type 1, EA scores are 88.89% (GPT-5), 81.48% (Gemini-2.5-Pro), and 80.62% (Gemini-2.5-Flash-Lite); SER scores are 92.59% (GPT-5), 88.89% (Gemini-2.5-Pro), and 92.59% (Gemini-2.5-Flash-Lite).
• Open-source models lag on type 1: EA of 66.67% (Llama 4) and 29.63% (Qwen 3); SER of 85.19% (Llama 4) and 40.74% (Qwen 3).
• Open-source models struggle badly on types 2–4: Average scores under the proposed framework reach 1.82 on type 2, 2.45 on type 3, and 2.35 on type 4 questions. Closed-source overall averages are 3.75 (type 2), 3.77 (type 3), and 3.72 (type 4).
• Greater structural complexity than prior benchmarks: Compared with BIRD, CORGI has 7.48 JOINs per question versus 0.93, 2.31 queries per question versus 1.0, and answers of 1002 versus 161 characters. Type 2–4 SQL queries average 1159.33 characters versus 255 for type 1, with 3.80 versus 0.55 JOIN operations.
• Executable-query difficulty gap: While average SQL SER on the BIRD dev set reaches 88.0%, average SQL SER for CORGI type 2–4 questions drops to 54.88%.
• Moderate and declining human–LLM agreement: Kendall's tau consistency ranges from 0.52–0.71 (type 2) to 0.43–0.64 (type 3) and 0.31–0.57 (type 4), based on human expert annotation of 3 of the 10 databases (AppStore, Clothing, and Shopify).
• No clear self-preference bias among judges: On type 2 questions, GPT-5 assigns an average score of 3.87 to its own outputs, while Gemini-2.5-Pro and Llama 4 assign 4.16 when evaluating GPT-5's responses.
• Qualitative verdict: LLMs are good at generating structured, MBA-trained responses, but fail to give sound implementable action plans.

Methodology in Plain English

The authors first build database schemas from publicly available sources, technical documentation, and application interfaces to emulate real business operations. They then populate those schemas with synthetic data using three independent rule types: business operational constraints (for example, follow-up assessments conducted every six months), latent feature distributions (for example, a customer base of 35% trial users, 44% regular subscribers, 17% loyal customers, and 4% VIP customers), and seasonal trends (for example, a +45% New Year's resolution effect and a -31% summer vacation dip). Data was synthesized with Gemini-2.5-flash-lite (version date Sept 2025) and manually reviewed and randomly tested.

Questions are generated across four types: past/descriptive (direct factual retrieval), past/explanatory (explaining historical patterns), future/predictive (forecasting), and future/recommendational (actionable strategy). For each question, a model generates up to five SQL queries, executes them, discards or empties invalid results, and composes a final answer from the question, execution results, and a knowledge set combining database descriptions with externally retrieved information.

Evaluation happens two ways. Quantitative SQL metrics (SER across all types, plus EA for type 1) check whether the generated queries run. For the open-ended type 2–4 answers, a discriminator agent reads the question type and content, selects the applicable dimensions from the six-dimension framework, routes the answer to specialized scoring agents, and averages the resulting scores on a 0–3–5 scale, where 5 meets professional consulting standards and 0 is a critical failure.

Generators tested were Gemini-2.5-flash-lite, Gemini-2.5-Pro, GPT-5, Llama 4 (meta-llama/Llama-4-Scout-17B-16E-Instruct), and Qwen 3 (Qwen/Qwen3-235B-A22B-Instruct-2507). Automatic evaluators were Gemini-2.5-Pro, GPT-5, and Llama 4. Temperature was set to 0 for all models.

Why This Matters

Impact on research. CORGI reframes text-to-SQL from a translation problem into a decision-support problem, and argues that the community needs new automatic methods for evaluating open-ended, qualitative responses rather than gold-label matching alone. It provides a released dataset, evaluation framework, gold labels for descriptive queries, and human-annotated results for higher-level questions to support replication and extension.

Real-world applications:

• Business analytics and BI tooling: Users asking for trend analysis, forecasting, and strategy rather than past records, spanning pricing strategies and capacity planning.
• Management consulting workflows: The evaluation dimensions are explicitly borrowed from how professional strategic consulting assesses analysis.
• Healthcare decision support: The related work notes clinicians querying patient records to predict risk and support treatment decisions.
• Industry verticals simulated in the benchmark: food delivery (DoorDash), home rental (Airbnb), clothing e-commerce (Lululemon), freelancing (Upwork), app stores, car rental (Turo), luxury consignment (The RealReal), user analytics (Amplitude), personalized products (Persona Nutrition), and e-commerce enablement (Shopify).

Industry relevance. The benchmark's own framing is that end users need advisory answers, not just retrieved rows. The gap between closed-source and open-source models on SQL generation, and between all models on retrieval versus recommendation, identifies where deployed systems are most likely to fail today.

Future Directions

• Reduce the synthetic-data gap. All database contents and SQL queries are synthesized rather than drawn from production systems, omitting natural artifacts such as noise, redundancy, and schema drift.
• Support multi-turn interaction. CORGI covers only single-turn queries, leaving out the conversational refinement common in real systems.
• Test robustness to paraphrase. Each query has only one natural language formulation, so robustness to paraphrasing or informal inputs is not evaluated.
• Expand coverage. The study covers only 10 business verticals, with additional domains left for future work. The authors also note that the three simulation rule types are applied independently, and that modeling interactions across rules (for example, linking payment method distributions to seasonal trends during Black Friday) is future work.

Target Audience

Researchers and practitioners working on text-to-SQL, LLM benchmarking, LLM-as-judge evaluation, and question answering over databases; industry teams building BI or analytics assistants that must move beyond retrieval into explanation, forecasting, and recommendation; and business or management researchers interested in how LLM outputs compare against professional consulting standards.

Authors’ abstract

Text-to-SQL benchmarks have traditionally only tested simple data access as a translation task of natural language to SQL queries. But in reality, users tend to ask diverse questions that require more complex responses including data-driven predictions or recommendations. Using the business domain as a motivating example, we introduce CORGI, a new benchmark that expands text-to-SQL to reflect practical database queries encountered by end users. CORGI is composed of synthetic databases inspired by enterprises such as DoorDash, Airbnb, and Lululemon. It provides questions across four increasingly complicated categories of business queries: descriptive, explanatory, predictive, and recommendational. This challenge calls for causal reasoning, temporal forecasting, and strategic recommendation, reflecting multi-level and multi-step agentic intelligence. We find that LLM performance degrades on higher-level questions as question complexity increases. CORGI also introduces and encourages the text-to-SQL community to consider new automatic methods for evaluating open-ended, qualitative responses in data access tasks. Our experiments show that LLMs exhibit an average 33.12% lower success execution rate (SER) on CORGI compared to existing benchmarks such as BIRD, highlighting the substantially higher complexity of real-world business needs. We release the CORGI dataset, an evaluation framework, and a submission website to support future research.