From Test-Time Scaling to Reusable Memory: Measuring Crystallization in Text-to-SQL

AI summary

Overview

• Research area: Natural language processing — text-to-SQL, test-time scaling, and agent memory / experience reuse; also an evaluation-methodology contribution.
• Technical level: Intermediate. The concepts (SQL generation, in-context retrieval, execution accuracy) are approachable, but the statistical protocol (paired seeds, two-stage hierarchical bootstrap, TOST equivalence tests) assumes some experimental-design background.
• Scope: A controlled measurement study on BIRD and Spider that separates three kinds of future value from stored test-time repair episodes — exact-query replay, cross-question retention, and held-out same-database transfer — and then isolates which single memory design choice produces each.

What This Paper Is About

Text-to-SQL systems can spend extra inference-time computation to repair a wrong query, but that computation is usually thrown away after the answer is delivered. The authors keep the verified repair episodes in a per-database memory bank and ask how much future value they actually create, and which design choice is responsible. They name this the "crystallization problem" and build a protocol that holds the solver fixed while changing one memory decision at a time.

Key Contributions

1. Formalizing the crystallization problem as an operational measurement problem. The paper defines a crystallization ratio (CR) that compares the held-out gain from memory against the on-demand repair headroom measured under the same solver, data split, and repair budget.
2. Separated measurement of future value. Exact-query replay, cross-question retention, and held-out same-database transfer are evaluated as three distinct settings rather than collapsed into one end-to-end score, so a reported gain maps to a concrete future-use case.
3. Controlled attribution of transfer. With a fixed solver and paired comparisons that change one choice at a time, the paper tests three competing explanations for why memory helps: generic well-formed examples, correctly matched question–SQL pairs, or exposure to the target database.
4. Open artifacts. Code, evaluation artifacts, and reproduction instructions are released at github.com/ai-jiaqian/text-to-sql-memory-crystallization.

Main Findings

• Memory delivers a substantial share of repair headroom. On BIRD, storing verified corrected queries verbatim raises held-out first-attempt accuracy from 62.04% to 66.38%, a gain of +4.34 pp (CI [+1.50, +7.49], p = .0034), which yields CR = 44.4% (95% CI [24, 65]%) of the on-demand repair headroom on the same held-out questions. 10 of 11 databases improve. Memory fixes 127 held-out questions and breaks 67, a net gain of 60 among 1,383 question–seed cases. The gain survives a stricter subset metric in every seed (+3.9 to +5.2 pp); 25 questions are stably fixed and five stably broken, and no question changes from a fix to a break.
• Replay and retention are very different quantities. On collection questions that produced cards, exact-query replay reaches 96.1% when the question's own card is eligible, dropping to 56.0% when that card is removed (cross-question retention). Both are reported against a 1.8% Reference floor.
• Database-specific content, not matched examples, carries transfer. Local cards beat equally valid foreign cards by 6.73 pp. Reducing local cards to natural-language lessons costs 2.68 pp. Similarity-targeted cards beat same-database random retrieval by 2.03 pp. In the correspondence intervention, an aligned bank improves accuracy by 5.36 pp and a bank with SQL permuted within each database by 3.91 pp — mispairing preserves 73% of the aligned lift. Restoring correct pairing adds 1.45 pp with a confidence interval that includes zero (CI [−0.45, +3.23], p = .14).
• Reliable verification is the clearest single lever. Verified self-vote cards beat an otherwise matched unverified bank by 4.85 pp (CI [+1.30, +8.06], p = .0064). The ungated bank sits 2.03 pp below no memory and is negative in all three seeds; only 3.9% of raw vote selections pass the oracle and the audited ungated bank is 5.6% correct. The gate need not be perfect: synthetically flipping 5–20% of verification decisions leaves transfer above the no-memory floor at every level.
• Wider retrieval helps; richer card formats do not show an advantage. Going from k = 1 to k = 10 adds 3.18 pp (CI [+1.09, +5.35], p = .0028); at k = 10, verbatim reaches CR = 57.6%. Intermediate steps are not individually significant. Retrieval width interacts with bank size: half the bank matches the full bank, a quarter causes accuracy to fall. Verbatim (W0) has the highest transfer point estimate; every richer writer is 1.3–2.0 pp lower, but all intervals cross zero, and no format passes hierarchical equivalence even at a ±2.5 pp margin. Format matters more for exact replay (89.8% to 99.7% across seeds), while W0 and W5 give similar retention (56.0/56.8%).
• Repair episodes are not uniquely useful for transfer. At matched per-database counts, repaired cards are higher than first-try-success cards by 1.30/2.32 pp in two independent draws, but both confidence intervals include zero. The source still matters in other ways: zero-feedback resampling and one-bit retry underperform interactive probe-grounded repair at matched bank sizes, and pre-answer exploration alone reaches only about 37% of the on-demand repair headroom on collection questions.
• The retrieval algorithm matters less than the bank. Once cards exist and retrieval stays within the target database, four targeted methods differ by less than 0.7 pp; BM25 matches dense embeddings, and a payload-aware key does not change the ordering of card formats.
• Memory mainly substitutes for repair rather than raising the ceiling. In a separate internally paired serving run, memory lifts the first attempt by +2.7 pp, and one repair round captures 80% of what three memory-less rounds achieve at 59% of the calls. Memory does not raise the repaired endpoint (70.0 vs. 70.4). Memory-less repair resolves 59–67% (mean 62%) of the questions memory fixes, versus 26% of comparable base failures.

Methodology in Plain English

The authors fix one solver and vary only the memory design around it. The solver is a single greedy call (temperature 0, 2,048-token budget, reasoning disabled) that sees the full schema with no lossy schema-linking step, so the retrieved memory block is the only thing that differs between arms.

Data is split 70/30 within each database: 1,073 collection questions on which repair runs and memory is built, and 461 held-out questions for transfer. Because the split is per database, every database appears on both sides, so "transfer" means same-database transfer to a new question. The second benchmark, Spider, runs 1,034 questions (723/311) through the identical pipeline. Three paired seeds (42/7/13) are used.

The memory lifecycle has four choices, each varied one at a time against a default:

• Acquisition (source). For questions the solver gets wrong, one of four sources spends an attempt budget K = 3. T1 resamples with zero feedback; T2 retries on a single correctness bit; T3 (the main source) interleaves read-only database probes with retries; T4 takes a self-vote over five sampled candidates clustered by result set.
• Admission (verification). A verified arm stores only candidates confirmed correct; a vote-admit arm removes that check. Critically, the oracle reveals only whether a candidate is correct — never a gold SQL string.
• Writing (card format). Seven formats are compared, from Verbatim (W0, question and corrected query) through Trace (W1), Diff (W2), Diff+Mode (W3), Anchor (W3+A), Diff+Guard (W4), up to Capsule (W5, an LLM-written natural-language summary). Diffs and guards are deterministic; only W5 is LLM-written. A sentinel test confirms that a marker placed in the gold-SQL field never reaches a card or the solver prompt — the writing is "gold-SQL-free."
• Consumption (retrieval). Cards are banked per database; the top-k (default k = 5) are retrieved by question-to-question cosine similarity using Qwen3-Embedding-0.6B (1,024-d, L2-normalized). The stored SQL does not affect the match, so no format gets a retrieval advantage.

Measurement uses execution accuracy (EX), where unevaluable counts as wrong. Transfer conditions the solver on retrieved cards for held-out questions; CR divides that gain by the repair headroom on the same questions. Statistically, the main analysis is a two-stage hierarchical bootstrap that first resamples the 11 databases and then unique questions within each, keeping all seed results for a question together. Four comparisons support the main claims (W0 vs. no memory, verified vs. unverified, k = 10 vs. k = 1, and the correspondence test); the remaining sweeps are labeled exploratory.

Why This Matters

• Impact on research. Self-evolving text-to-SQL systems have already established that memory helps; this paper asks how it helps and where the gain lands. It shows that a single end-to-end number conflates recurring-question replay with genuine generalization, and it demonstrates that attribution is possible without bundling the memory design with the surrounding pipeline. It also brings controlled-comparison methodology to a setting where outcomes are deterministic and execution-based, rather than resting on LLM-based judgments.
• Real-world applications:
  • Analytics assistants over a fixed warehouse. Where a team repeatedly queries the same schemas, verified corrected queries can raise first-attempt accuracy on new questions over those databases by 4.34 pp on BIRD without extra repair rounds.
  • Cost control in production query agents. Since one repair round captures 80% of the three-round gain at 59% of the calls, memory lets operators trade stored artifacts for inference calls.
  • Deployment-scoping decisions. The locality result (local cards beat foreign cards by 6.73 pp) tells operators that a bank should be scoped per database rather than pooled across schemas.
  • Human-in-the-loop verification design. The finding that ungated storage is worse than no memory (2.03 pp below) argues for routing admissions through user confirmations, executable assertions, or downstream business checks.
• Industry relevance. The paper provides a measurable recipe — verified, database-scoped, verbatim cards with broad retrieval — plus a caveat that richer formats, elaborate retrievers, and repair-specific content show no statistically supported advantage. That is directly actionable for teams deciding how much infrastructure to build around a text-to-SQL memory layer.

Future Directions

• Cross-database generalization. The paper's transfer definition is same-database transfer; generalization to unseen schemas is described as a different deployment setting probed directly in Section 5.3, but the outcome of that probe is not reported in the content available here.
• Parametric consolidation. The authors explicitly place reinforcement learning and other parametric updates that turn execution feedback into model weights outside the scope of the study, leaving open whether the same episodes would yield larger or more durable gains if consolidated rather than retrieved.
• Stronger verification sources in deployment. The experiments measure how reliable the verification signal must be, not where it comes from; how user confirmations, executable assertions, or business checks compare to the oracle bit as admission gates is an open question.
• Format and source equivalence. No card format or episode-source comparison passed the hierarchical TOST equivalence test, so the paper cannot establish that these choices are interchangeable — only that no improvement was demonstrated. A larger study could settle this.
• Formal cost models. The paper reports per-question call counts and an amortized lifecycle token account but states that neither is a complete dollar or latency model.

Target Audience

Researchers and engineers working on text-to-SQL, LLM agents, and agent memory will benefit most, particularly those designing or evaluating experience-reuse systems and those skeptical of single end-to-end benchmark scores. Practitioners deploying database query assistants will find the design-choice results (verification, database scoping, retrieval width) directly operational. Evaluation-methodology researchers interested in controlled attribution under a fixed solver will also find the protocol relevant.

Authors’ abstract

Test-time scaling can correct difficult text-to-SQL queries, but the extra computation is normally discarded after each answer. Systems increasingly retain verified repair episodes, yet evaluations still report one end-to-end score. It cannot distinguish replay on recurring questions from help on unseen questions, or identify the responsible memory choice. We call measuring this future value the crystallization problem. Our controlled evaluation holds the single-shot solver fixed and varies one memory choice at a time. We separately measure replay, cross-question retention, and held-out same-database transfer. On BIRD, storing verified corrected queries improves held-out first-attempt accuracy by 4.34 percentage points. This gain captures 44.4% of the accuracy headroom provided by on-demand repair on the same questions. Controlled interventions identify database-specific content as the main operating ingredient. Reliable verification and broader retrieval coverage yield supported gains; richer formats and elaborate retrievers do not. Open-source code, evaluation artifacts, and reproduction instructions are available at https://github.com/ai-jiaqian/text-to-sql-memory-crystallization.