MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training

AI summary

Overview

• Research area: Natural Language Processing, specifically conversational semantic parsing (multi-turn Text-to-SQL) and agentic reinforcement learning for LLMs.
• Technical level: Advanced. The paper formulates the task as a Markov Decision Process, designs multi-level outcome and process rewards, and trains with GRPO; comfort with RL terminology and SQL is assumed.
• Scope: The paper introduces MTSQL-R1, an agentic training framework that turns multi-turn Text-to-SQL from single-pass translation into a long-horizon propose-execute-verify-refine loop grounded in database execution feedback and a persistent dialogue memory, evaluated on CoSQL and SParC.

What This Paper Is About

Multi-turn Text-to-SQL asks a model to translate each user utterance in a conversation into an executable SQL query while staying coherent with earlier turns and grounded to the target database schema. The authors argue that most existing systems treat this as a short-horizon translation task: they emit one query per turn without executing it, without explicit verification, and without refinement, which produces non-executable or incoherent outputs. Their goal is to train an agent that verifies its intermediate SQL against both a database and a dialogue memory, and self-corrects until all checks pass.

Key Contributions

1. A long-horizon multi-turn Text-to-SQL framework with explicit verification and self-correction. The authors state that MTSQL-R1 is the first multi-turn Text-to-SQL framework that combines execution-based verification, memory-based coherence verification, and self-correction (Table 1 compares it against Reasoning-SQL, SQL-R1, CoE-SQL, and ACT-SQL, none of which have both tool integration and coherence verification).
2. A long-horizon training pipeline combining self-taught Warm-Start SFT with end-to-end RL. Warm-Start SFT uses autonomously collected trajectories with rejection sampling; RL then optimizes the policy with multi-level rewards derived from execution success and memory coherence.
3. A formalization of multi-turn Text-to-SQL as an MDP with an environment consisting of a relational database D=(S,T) (schema S, tables T) and a maintained long-term dialogue memory M_{t-1}, plus a six-action space (PROPOSE, EXECUTE, E-VERIFY, M-VERIFY, SELF-CORRECT, FINALIZE).
4. Empirical gains in coherence, executability, and generalization using 1.7B- and 4B-parameter open-source backbones, with the authors reporting state-of-the-art results on CoSQL and SParC and noting that full recipes will be released upon acceptance.

Main Findings

• Best-in-class results at small model sizes: On Qwen-3-1.7B, Warm-Start SFT + RL (Outcome + Process) reaches CoSQL EX 77.3 / EM 63.5 and SParC EX 76.2 / EM 66.1, for an average EX of 74.6 (+4.1) and average EM of 64.4. On Qwen-3-4B, the same configuration reaches CoSQL EX 79.9 / EM 65.2 and SParC EX 79.0 / EM 68.7, for an average EX of 77.6 (+3.5) and EM of 66.5. For comparison, the paper lists GPT-4.1 at an average EX of 61.4 / EM 32.7, OpenAI-O3 at 58.4 / 29.7, DeepSeek-R1 (671B) at 58.1 / 36.6, Qwen-3-32B at 70.4 / 53.9, CoE-SQL at 64.1 / 51.6, and RASAT+PICARD at 64.5 / 57.7.
• Both training stages contribute: Self-taught Warm-Start SFT increases the coverage of high-quality long-horizon trajectories and improves downstream performance across rounds (Round 1, Round 2, Round 3 reported), and end-to-end RL improves both EX and EM in in-domain and out-of-domain settings.
• Long-horizon training lifts logical correctness more than exact string matching: The authors report that conventional short-horizon SFT attains comparable EM but weaker logical consistency, while their agent substantially improves EX while maintaining or improving EM.
• Process rewards give modest but consistent gains: Moving from outcome-only to outcome-plus-process rewards raises the Qwen-3-1.7B average EX from 74.0 to 74.6 and the Qwen-3-4B average EX from 76.9 to 77.6.
• Both long-horizon actions are essential: Ablations in Table 3 show that removing the execution tool or the memory verification tool degrades performance. For Qwen3-4B + Warm-Start + RL (EX 79.9 / EM 65.2), removing the execution tool drops to 74.6 / 64.6 and removing memory verification drops to 77.8 / 64.1.
• Small models struggle with long-horizon tool calling without training: The untrained long-horizon base agent scores only 23.3 average EX / 17.1 EM at 1.7B and 59.0 / 44.9 at 4B.
• Larger gains on harder and deeper-turn questions: On CoSQL, the base model degrades sharply as turns increase, while the method improves across all turn levels with the largest gains for Turn ≥ 4; similarly, gains concentrate on Hard and Extra Hard difficulty buckets. More difficult or more multi-turn questions also require longer responses and more tool interactions.
• Generalization across settings: The method improves out-of-domain performance (e.g., Qwen3-4B + Warm-Start + RL averages 75.7 EX / 66.0 EM out-of-domain versus 72.7 / 65.0 for short-horizon SFT), transfers to LLaMA3.2-3B-Instruct (74.8 CoSQL EX and 75.2 SParC EX versus a base of 22.9 and 24.4 and a best short-horizon RL baseline of 70.4 and 70.9), and holds up in single-turn out-of-domain evaluation.
• Robustness under realistic history: When the model must condition on its own previously generated SQL rather than gold SQL on CoSQL, the method scores 76.5 EX (+5.3) versus the previous best baseline at 71.2, widening the gap from +4.7 under gold history.
• Long-horizon abilities evolve with training: Tracking five capabilities (Execute function calling, Memory-Verify function calling, execution verification, memory-based verification, generation/self-correction) across Base, Warm-Start, and Warm-Start+RL stages, all five improve, with RL especially boosting memory-related abilities.
• Error profile shifts: Using an LLM-as-judge setup with GPT-5 over an error taxonomy of Execution Error plus Constraint Coherence, Schema Linking, Aggregation Drift, and Join Path, execution errors drop sharply and context-coherence errors decrease substantially, while Aggregation Drift changes little; six of the remaining execution failures stem from an 8,000-token cap.
• Efficiency tradeoff: The paper reports a maximum latency of about 28 seconds and places the method on the Pareto frontier of execution accuracy versus latency, citing a survey (Pan et al., 2026) that around 70% of applications tolerate latencies on the order of minutes.

Methodology in Plain English

The authors reframe multi-turn Text-to-SQL as a decision-making problem rather than a one-shot translation. They define a Markov Decision Process whose state contains the dialogue history, the schema, the current utterance, the dialogue memory, the current candidate SQL, and accumulated observations. The agent can take six actions: propose a SQL query, execute it on the database, verify correctness from the execution result, verify coherence against memory, correct the query, or finalize it. A fixed transition rule chains the actions: propose, then execute, then verify, then either finalize or loop back into self-correction.

Training happens in two phases. First, a Warm-Start SFT phase collects trajectories autonomously. For each training instruction, the base model generates 20 rollouts at temperature 0.7, and only rollouts whose final SQL matches the gold query are kept. A difficulty-aware rejection sampling step keeps up to two short trajectories (≤2 interactions) for easy or perfectly solved items, and retains longer trajectories (≥2 interactions) for hard items, clustering them with Qwen3-Embedding and sampling three representatives. The model is then fine-tuned with loss masking that supervises only action and SQL tokens, not instructions, execution outputs, or memory prompts. The self-taught loop repeats, and training examples that already produced high-quality trajectories are removed from the working set for the next round.

The second phase is end-to-end RL using GRPO. Training follows an easy-to-hard curriculum: each example gets 20 sampled trajectories, the success count s_i out of 20 (measured by EX and EM) is computed, examples with s_i = 20 are discarded as too easy, and the rest are sorted by s_i and partitioned into contiguous bins of size 2000, with Curriculum Data = 1 being the easiest. Rewards are multi-level. Outcome rewards are Execution Match (whether the executed prediction matches the executed gold query) and Exact Match (strict string equality). Process rewards score each action: for PROPOSE and SELF-CORRECT, an average F1 over the SQL clauses SELECT, WHERE, JOIN, GROUP, and ORDER; for E-VERIFY, a table mapping execution results (ok, null, error) and the model's binary pass/fail verdict to a reward; and for M-VERIFY, an F1-based reward conditioned on the verdict. These are combined in a weighted sum, with weights chosen by grid search on a small held-out subset of training data. GRPO then normalizes rewards within each group of trajectories to compute advantages, applies clipping and a KL penalty to the reference policy, and again masks out execution outputs and human instruction tokens so the model learns the reasoning process.

Why This Matters

Research impact. The paper moves multi-turn Text-to-SQL away from prompting-based, short-horizon pipelines (ACT-SQL, CoE-SQL) and single-turn RL approaches (Reasoning-SQL, SQL-R1) toward a training paradigm where the environment itself supplies supervision. It also argues that execution accuracy, not just exact string match, is the more meaningful target for SQL generation, and shows that small open-source models (1.7B and 4B) can beat larger baselines under this framework.

Real-world applications.

• Enterprise data access in high-risk domains such as finance, healthcare, and security logs, where an incorrect query can expose sensitive information or cause business impact.
• Batch offline processing, where the reported maximum latency of about 28 seconds is acceptable.
• Business intelligence and interactive dashboard analytics, where practitioners can pick configurations balancing performance, latency, and cost.
• Conversational analytics assistants that must remember constraints from earlier turns and avoid contradicting them.

Industry relevance. The method is built on open-source LLMs (Qwen-3-1.7B/4B, LLaMA3.2-3B-Instruct), which matters for organizations that cannot rely on closed-source GPT models. The authors also describe paths to industrial deployment, including a proposed Schema_Retrieval tool that queries a vector database to retrieve only the top-K relevant tables instead of loading a full schema into context, and a backend-agnostic memory module accessed purely through calls like memory_retrieve.

Future Directions

• Aggregation Drift remains unsolved. The error analysis shows this category changes little after training, and the authors note aggregation-heavy SQL tends to be extra hard, flagging it as an open problem.
• Extra-hard cases and residual errors persist. The limitations section states that some extra-hard cases are still unresolved, and six of the remaining execution failures come from truncation at an 8,000-token cap.
• Efficiency versus accuracy. The method incurs higher latency and token usage than baselines, which may limit real-time use; the authors leave this to future work.
• Scaling to industrial schemas. The proposed Schema_Retrieval tool and the upgrade of the memory backend from parsed text to a vector database are described as natural extensions but are not part of the core experiments, so their effectiveness is not evaluated in this paper.

Target Audience

Researchers and engineers working on Text-to-SQL, conversational semantic parsing, and LLM agents trained with reinforcement learning will gain the most from this paper. It is also relevant to practitioners building enterprise data-access or BI assistants who need to weigh execution accuracy against latency, and to readers interested in how multi-level process rewards and curriculum design affect long-horizon agentic training. Because the paper assumes familiarity with MDPs, GRPO, and SQL, readers without that background will find the methodology sections dense.

Authors’ abstract

Multi-turn Text-to-SQL aims to translate a user's conversational utterances into executable SQL while preserving dialogue coherence and grounding to the target schema. However, most existing systems only regard this task as a simple text translation task and follow a short-horizon paradigm, generating a query per turn without execution, explicit verification, and refinement, which leads to non-executable or incoherent outputs. We present MTSQL-R1, an agentic training framework for long-horizon multi-turn Text-to-SQL. We cast the task as a Markov Decision Process (MDP) in which an agent interacts with (i) a database for execution feedback and (ii) a persistent dialogue memory for coherence verification, performing an iterative propose to execute -> verify -> refine cycle until all checks pass. Experiments on COSQL and SPARC demonstrate that MTSQL-R1 consistently outperforms strong baselines, highlighting the importance of environment-driven verification and memory-guided refinement for conversational semantic parsing. Full recipes (including code, trained models, logs, reasoning trajectories, etc.) will be released after the internal review to contribute to community research.