BIRDTurk: Adaptation of the BIRD Text-to-SQL Dataset to Turkish

AI summary

Overview

• Research area: Natural Language Processing — cross-lingual Text-to-SQL and multilingual semantic parsing.
• Technical level: Intermediate. Readers need familiarity with Text-to-SQL benchmarks, execution-accuracy metrics, prompting versus fine-tuning paradigms, and basic statistics (confidence intervals, sample size formulas).
• Scope: The paper builds BIRDTurk, the first Turkish adaptation of the BIRD Text-to-SQL benchmark, validates its translation quality statistically, and benchmarks inference-only prompting, agentic reasoning, and supervised fine-tuning on it.

What This Paper Is About

Text-to-SQL benchmarks are overwhelmingly English-centric, and Turkish differs sharply from English in ways that matter for the task: it is agglutinative and uses Subject-Object-Verb word order, whereas English and SQL both follow Subject-Verb-Object, giving English a near-linear mapping from question to query. The paper's goal is to create a Turkish version of BIRD — a benchmark built on large, noisy, real-world databases — that preserves the original databases and SQL execution semantics while localizing the questions, evidence text, and schema identifiers into Turkish, then to measure how much performance Turkish costs and which modeling strategies cope best.

Key Contributions

1. BIRDTurk, described as the first Turkish Text-to-SQL dataset adapted from the BIRD benchmark, released with its training and development splits (test split not released).
2. A statistically grounded validation framework based on the Central Limit Theorem, which sets a required human-evaluation sample size (974 examples) and yields a dataset-level accuracy estimate with a stated confidence bound, instead of requiring exhaustive manual annotation.
3. A controlled three-stage translation and localization pipeline that uses a fixed schema mapping, joint question-evidence translation, and AST-based SQL rewriting so that English and Turkish queries are execution-equivalent.
4. Baseline results across three paradigms — zero-shot in-context learning, the agentic DIN-SQL pipeline, and supervised fine-tuning of mT5 and Qwen2.5-Coder families — measured with Execution Accuracy (EA), Valid Efficiency Score (VES), and Exact Match (EM).

Main Findings

• Translation quality is high and statistically bounded: Of 974 randomly sampled examples manually evaluated, 956 were judged correct, giving 98.15% observed translation accuracy, which the authors state places dataset-level accuracy within a ±3% margin at 95% confidence.
• Turkish compresses words but not content: Average words per question drop from 14.05 to 10.21 in training (-27.3%) and from 14.55 to 10.64 in development (-26.9%), while character counts fall only 5.1% and 4.3% respectively — the authors read this as preserved content at higher information density.
• Lexical diversity more than doubles: Type-Token Ratio rises from 6.07% to 13.49% in training (+122.2%) and from 9.84% to 19.76% in development (+100.8%); unique vocabulary grows from 9,002 to 15,142 (+68.2%) and from 2,450 to 3,704 (+51.2%).
• SQL structure is preserved: Average SQL tokens change by only -0.5% (training, 31.03 to 30.89) and -0.6% (development, 31.46 to 31.26), which the authors use as evidence that structural complexity carried over.
• Evidence text gets longer in token count: Average evidence tokens rise 9.3% in training (21.32 to 23.31) and 2.6% in development (21.49 to 22.04), attributed to a "multilingual tokenization tax."
• Inference beats fine-tuning by a wide margin: With gemini-2.5-flash-lite, in-context learning reaches 58.21 EA / 63.15 VES / 12.65 EM on English BIRD versus 43.16 EA / 45.34 VES / 10.82 EM on BIRDTurk. DIN-SQL reaches 60.89 EA / 64.66 VES / 17.60 EM in English versus 49.02 EA / 51.10 VES / 11.41 EM in Turkish.
• Agentic reasoning helps more in Turkish: DIN-SQL improves over direct prompting in both languages, but the relative gains are larger for Turkish, suggesting explicit decomposition, schema linking, and self-correction compensate for weaker surface-level alignment.
• Turkish degrades performance consistently but the method ranking is stable: English outperforms Turkish in absolute terms, yet DIN-SQL remains better than direct prompting in both languages, which the authors take as evidence the adaptation preserved BIRD's reasoning characteristics.
• Supervised fine-tuning on BIRDTurk is hard for multilingual baselines: mT5-small reaches 0.26 EA / 0.28 VES / 0.00 EM; mT5-base 0.98 / 1.04 / 0.06; mT5-large 1.83 / 2.05 / 0.13 — all under 2% execution accuracy.
• Instruction-tuned code models scale better: Qwen2.5-Coder-0.5B-Instruct reaches 1.24 EA / 1.38 VES / 0.33 EM; the 1.5B variant reaches 7.24 / 8.01 / 2.61; and the 3B variant reaches 15.38 / 17.12 / 4.82.
• Untuned base models are near-useless on the task: Base model results were excluded from the table because all performed near 0%, with the best (Qwen2.5-Coder-3B-Instruct) achieving only 2.22% execution accuracy.
• Motivation for the Qwen architecture is tokenization, not just scale: The authors cite a prior finding that the Qwen tokenizer compresses Turkish better than Llama-3 or Gemma, reducing over-segmentation of agglutinative suffixes; Qwen2.5-Coder was pretrained on 5.5 trillion tokens.
• Two sources of the English-Turkish gap are named: intrinsic Turkish properties (SOV word order, agglutinative morphology causing lexical sparsity, suffix-heavy forms causing tokenization fragmentation) and underrepresentation of Turkish in LLM pretraining corpora.
• Context from the original BIRD benchmark: BIRD contains 12,751 text-to-SQL pairs across train/dev/test and 95 databases totaling 33.4 GB over 37 domains, with 7.3 tables and roughly 549,000 rows per database on average; the largest database ("Donor") is 4.5 GB, and human performance on BIRD is 92.96%.
• Not reported: The paper does not report test-split results, per-domain breakdowns, statistical significance tests between model pairs, or the number of natively authored Turkish questions (all BIRDTurk questions are translations).

Methodology in Plain English

The authors started from the publicly available training and development splits of BIRD and rebuilt them in Turkish while keeping the databases and SQL untouched. The core constraint is that an English question and its Turkish counterpart must produce identical query results.

They first froze a one-to-one mapping between every English table and column name and a Turkish ASCII snake_case name, using gemini-2.5-flash under strict rules (convert Turkish characters to ASCII, translate recurring sub-terms consistently, leave unknown acronyms alone). Only identifiers were translated — actual data values inside the database, such as quoted strings in WHERE clauses, were left in English, deliberately avoiding a separate value-linking problem. Two columns required collision resolution where distinct English names mapped to the same Turkish form.

They then standardized the "evidence" text (BIRD's hints about external knowledge) so that every backticked identifier matched the new Turkish schema, and translated the question and evidence together in one context window, with backticked spans frozen verbatim and numbers, dates, and comparative logic preserved.

For SQL, they avoided letting a model write new queries. Instead they parsed each query into an abstract syntax tree and rewrote only the identifier nodes according to the mapping, leaving keywords, operators, functions, and literals untouched — this prevents the alias collisions and partial-match bugs that string substitution causes. Every instance was then executed in both languages against the underlying SQLite database; if the result sets differed, the example was flagged, checked with a rubric-based LLM judge, corrected, and re-verified.

For quality measurement, they treated translation correctness as a Bernoulli variable and used the Central Limit Theorem to size the human review. With a 95% confidence level (Z = 1.96), ±3% margin of error, and maximum variance assumed (p = 0.5), the infinite-population sample size came out to about 1,068. Applying the finite population correction for N = 10,962 total samples gave approximately 974 examples to review.

Evaluation used the dev split of each dataset, with metrics averaged over three independent runs. Inference experiments used a single model, gemini-2.5-flash-lite, in both languages so that differences would come from strategy rather than model capacity, including a structure-preserving translation of the DIN-SQL prompts. Supervised fine-tuning covered mT5-small, mT5-base, and mT5-large, then Qwen2.5-Coder 0.5B, 1.5B, and 3B instruction-tuned variants, with base-model inference run under the same in-context-learning prompt for a fair comparison.

Why This Matters

This work gives researchers a controlled testbed for asking a precise question: when a model fails on a non-English Text-to-SQL task, how much of that is the language itself and how much is benchmark construction? Because the databases, SQL semantics, and evaluation protocol are held fixed from BIRD, the English-Turkish gap can be attributed to language rather than to a different task. It also shows that translation-based benchmark adaptation can be validated with a bounded statistical claim rather than open-ended manual review, a template that extends to other morphologically rich, low-resource languages. For the Turkish NLP community, it is the first dataset in this line to inherit BIRD's "dirty data" and external-knowledge reasoning demands rather than Spider's cleaner schema-linking paradigm.

Real-world applications:

• Enterprise database assistants: Organizations with Turkish-speaking analysts and Turkish-named schemas can use these baselines to judge whether an LLM-based SQL assistant is viable before deploying it.
• Model and tokenizer selection: The comparison of mT5 against the Qwen2.5-Coder family gives practitioners concrete evidence that tokenizer compression for agglutinative languages matters as much as model size.
• Architecture choice under latency or cost constraints: The finding that agentic multi-stage reasoning narrows the English-Turkish gap helps teams decide when the extra inference cost of decomposition and self-correction pays off.
• Benchmark auditing: Teams building or purchasing multilingual evaluation suites can reuse the CLT-based sampling approach to put a defensible confidence interval on translation quality instead of claiming "reviewed by native speakers."

Industry relevance: the work comes from Roketsan's Artificial Intelligence Technologies Unit together with Middle East Technical University, and the target scenario — 33.4 GB of noisy enterprise databases, external domain knowledge, and non-expert users querying in their own language — mirrors real internal analytics deployments. The gap between BIRD's 92.96% human performance and the best BIRDTurk number reported here (49.02 EA for DIN-SQL) makes clear how far current systems are from replacing analysts in a Turkish-language environment.

Future Directions

• Build a native Turkish benchmark: The authors propose constructing a fully native Turkish Text-to-SQL dataset to complement translation-based evaluation, since translation may underrepresent phenomena such as ellipsis and pragmatic inference.
• Create hybrid datasets: Mixing translated and natively authored Turkish questions is suggested to strengthen realism while retaining comparability with BIRD.
• Move toward agentic and multi-turn evaluation: Extending BIRDTurk toward enterprise-oriented, multi-turn settings inspired by Spider 2.0, where models debug queries and navigate workflows rather than producing a single query.
• Reduce reliance on a single translation model: The limitations section notes that all translations came from one LLM (Gemini), which may introduce systematic stylistic bias despite post-processing and CLT validation — comparing multiple translators or models is an open question.
• Raise fine-tuning performance: Supervised fine-tuning remains far below inference-based accuracy (15.38 EA at best versus 43.16 for plain in-context learning in Turkish), leaving unclear how much of that gap is data scale, tokenizer quality, or task difficulty.

Target Audience

Researchers and engineers working on cross-lingual semantic parsing, multilingual LLM evaluation, and Text-to-SQL systems will get the most from this paper, particularly those targeting morphologically rich or low-resource languages. It is also useful for practitioners deciding between prompting, agentic pipelines, and fine-tuning for a non-English database assistant, and for dataset builders looking for a defensible, statistically framed method of validating large-scale translation. Readers without background in SQL benchmarking or basic sampling statistics will need to consult the cited BIRD and DIN-SQL papers first.

Authors’ abstract

Text-to-SQL systems have achieved strong performance on English benchmarks, yet their behavior in morphologically rich, low-resource languages remains largely unexplored. We introduce BIRDTurk, the first Turkish adaptation of the BIRD benchmark, constructed through a controlled translation pipeline that adapts schema identifiers to Turkish while strictly preserving the logical structure and execution semantics of SQL queries and databases. Translation quality is validated on a sample size determined by the Central Limit Theorem to ensure 95% confidence, achieving 98.15% accuracy on human-evaluated samples. Using BIRDTurk, we evaluate inference-based prompting, agentic multi-stage reasoning, and supervised fine-tuning. Our results reveal that Turkish introduces consistent performance degradation, driven by both structural linguistic divergence and underrepresentation in LLM pretraining, while agentic reasoning demonstrates stronger cross-lingual robustness. Supervised fine-tuning remains challenging for standard multilingual baselines but scales effectively with modern instruction-tuned models. BIRDTurk provides a controlled testbed for cross-lingual Text-to-SQL evaluation under realistic database conditions. We release the training and development splits to support future research.