SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables

AI summary

Overview

Research area: Natural Language Processing / Table–Text Question Answering (QA) benchmarks and cross-modal multi-hop reasoning.

Technical level: Intermediate. Readers benefit from familiarity with SQL semantics, QA evaluation metrics (EM/F1), and LLM-driven data generation, but the paper's core argument is accessible without deep database theory.

Scope: The paper introduces SPARTA, an automatic pipeline for building large-scale, low-error Table–Text QA benchmarks with tree-structured, multi-hop, analytically rich questions, and demonstrates that current state-of-the-art models fail badly on it.

What This Paper Is About

Existing Table–Text QA benchmarks (HybridQA, OTT-QA, TAT-QA, FinQA, MultiHiertt) are built by hand, which makes them expensive, error-prone, and shallow: questions rarely exceed two reasoning hops and almost never require aggregation, grouping, or ranking. They also rely on toy-scale tables of roughly 15 rows, far from real-world databases with thousands of records.

SPARTA's goal is to replace this manual bottleneck with an automated, domain-agnostic construction framework that generates deep, logically complex, executable questions over both text and tables, while keeping annotation cost low by requiring only lightweight human verification.

Key Contributions

1. A scalable end-to-end benchmark construction framework: SPARTA unifies tables and passages into a single SQL-queryable reference fact database by extracting atomic facts from text into "grounding tables," so text and tables can be reasoned over with the same query language. It is domain-agnostic, demonstrated on NBA, Movie (IMDB), and Medical data.

2. Provenance-based refinement: A repair loop that detects queries returning empty results and uses database "why-not provenance" to pinpoint the blocking predicate, feeding that diagnosis back to the LLM so it rewrites only the offending clause instead of regenerating the whole query.

3. Realistic-structure enforcement: A constraint that forces the LLM to build nested queries in post-order traversal of the query graph, matching how human analysts actually write nested SQL and ensuring every intermediate subquery is executable before it is composed upward.

4. A high-difficulty, low-noise benchmark with quantified evidence: Thousands of validated question–answer pairs covering aggregation, grouping, and deep multi-hop reasoning, with a 0% annotation error rate in a 100-sample audit versus 17–30% for prior benchmarks, and a >30 F1 point drop for state-of-the-art systems.

Main Findings

• Massive performance collapse on SPARTA: ODYSSEY with GPT-5 scores 35.6 average F1 on SPARTA (oracle setting) versus 69.5 on HybridQA — a 33.9-point drop. HProPro with GPT-5 scores 40.4 versus 70.5 on HybridQA — a 30.1-point drop.

• Retrieval compounds the failure: In the end-to-end setting with no gold evidence, the best configuration (HELIOS + HProPro with GPT-5) reaches only 22.6 F1. Fine-tuned readers do not transfer: FiE Reader beats HELIOS + HProPro by 9.2 points on OTT-QA but trails by 11.0 points on SPARTA.

• Depth and breadth both hurt: At fixed depth, expanding breadth from 1 to 3 branches reduces HProPro and ODYSSEY by 25.2% and 27.5%. At fixed breadth, increasing depth from 1 to 3 reduces them by 47.2% and 49.9%. Models cannot handle tree-structured reasoning regardless of where the complexity comes from.

• Analytical operations remain a weakness: Queries with GROUP BY/HAVING yield F1 of 35.4 (ODYSSEY) and 27.1 (HProPro); ORDER BY/LIMIT yields 31.2 and 21.4; aggregation yields 28.4 and 37.2. All are below each model's average.

• Cross-modal reasoning is the sharpest failure point: HProPro drops 63.9% (45.2 → 16.3 F1) when text must be combined with tables rather than tables alone; ODYSSEY drops 23.0% (39.2 → 28.6). Models handle tabular data moderately well but fail to integrate textual evidence.

• Generation efficiency validates the design: Execution-Guided non-nested generation needs 1,134 LLM calls versus an ideal 1,058 (+7.2%) and 2,466 seconds, while One-Shot inflated to 1,830 calls and produced 60 empty and 1,265 duplicate queries. For nested generation, Post-Order+Prov used 4,722 calls — 42.8% fewer than plain post-order and 66.2% fewer than one-shot — and the advantage held across 20B, 70B, and 120B parameter models.

• Naturalness improves over templates: Execution-Guided generation beat template-based generation by 37.5% (auto-eval) and 36.7% (human eval) on overall naturalness; post-order nested generation beat templates by 123.2% and 117.8% respectively.

• Annotation cost drops roughly 4x: Validating 3,300 SPARTA queries took about 1,493 minutes of worker time versus roughly 6,600 minutes to author the same number from scratch for HybridQA.

• Prior benchmarks are measurably noisy: Auditing 100 sampled dev examples per dataset found at least one error in 21% (HybridQA), 26% (MultiHiertt), 30% (TAT-QA), and 17% (FinQA) of instances, versus 0% for SPARTA.

Methodology in Plain English

The pipeline has three stages and treats SQL as the connective tissue between text, tables, and natural language.

Stage 1 — Build a reference fact database. Start with ordinary relational tables (source tables). Extract atomic facts from accompanying passages — either from a pre-validated corpus like ROTOWIRE or by generating text from structured rows with rule-based templates — and store each fact as a tuple in a grounding table. Because both tables are now rows in one database, a question can freely hop between text-derived facts and structured records using a single SQL query.

Stage 2 — Generate queries. An LLM is given the schema and sample rows and asked to produce SQL whose number of nested predicates matches a target hop count, covering four standard nesting patterns (set membership, aggregate comparison, correlated filtering, and correlated aggregate comparison). Two safety mechanisms apply. First, post-order construction: the model builds leaf subqueries first and validates each one by executing it before wrapping it in a higher-level block, ensuring no invalid intermediate state. Second, provenance-based refinement: if a query returns zero rows, the system peels back predicates until rows appear, samples a tuple that should have been returned, runs a why-not provenance tool to identify which predicate blocked it, and tells the LLM to rewrite only that clause. This keeps queries semantically meaningful rather than technically valid but vacuous.

Stage 3 — Turn SQL into questions. A state-of-the-art SQL-to-text LLM (AST-ICL) rewrites each executable query into a fluent natural-language question, using the SQL abstract syntax tree as a guide. The query is executed to obtain the gold answer, completing the pair. Three CS graduate students then verify the questions for fluency and factual soundness — a lightweight check that does not require re-deriving the multi-hop answer, unlike full manual annotation.

Evaluation then compares prior Table–Text QA systems (ODYSSEY, HProPro, FiE Reader, with GPT-3.5, GPT-5, and retrieval via HELIOS) on SPARTA under oracle and retrieval settings, plus ablations over query generation methods and query shape configurations.

Why This Matters

Impact on research: SPARTA reframes Table–Text QA evaluation around what the paper calls a principled, executable ground truth. Because every question is backed by a SQL query that provably returns its gold answer over a known database, the benchmark eliminates the ambiguity that has plagued hand-built datasets — multiple valid answers, redundant modality tagging, and unanswerable questions. It also isolates reasoning failure modes that prior benchmarks could not measure: depth versus breadth in query trees, analytical operator handling, and cross-modal versus table-only reasoning. This gives the field a diagnostic instrument rather than just a scoreboard.

Real-world applications:

• Business intelligence and analytics assistants: answering questions like "which regional managers exceeded their sales quota in quarters where their team's average deal size grew" requires joins across a CRM table and textual notes or reports.
• Clinical decision support: correlating structured patient records with unstructured physician notes — dropping the requirement that models reason over only one modality at a time.
• Financial analysis and regulatory reporting: querying ledgers alongside narrative filings, with grouping and aggregation over large tables rather than the few-row tables in FinQA-style benchmarks.
• Enterprise search over heterogeneous data: open-domain QA where the relevant evidence spans multiple documents and databases, mirroring the retrieval setting where current systems score only 22.6 F1.

Industry relevance: The paper's generation framework is domain-agnostic and portable to any relational database with an accompanying text corpus, which means organizations can synthesize their own internal benchmarks from proprietary data. The failure analysis also directly guides product decisions: systems that only handle tables will underperform, systems that ignore aggregation will fail a large fraction of real queries, and retrieval-plus-reasoning remains unsolved.

Future Directions

1. Extend to multimodal inputs: The authors propose summarizing images and video into atomic statements, normalizing them into grounding tables, and merging them into the same fact database — since the tuples would share the existing schema, the query-generation pipeline would apply unchanged.

2. Improve cross-modal alignment in models: The 63.9% F1 drop when unstructured text is required is the paper's most severe finding, yet it offers no architectural remedy. Future work must find ways to ground textual spans into structured reasoning without losing retrieval accuracy.

3. Handle tree-structured reasoning at scale: Performance degrades markedly with both depth and breadth, and current methods have no mechanism for parallel branch traversal or merging of results across branches. New architectures or prompting strategies are needed for genuine multi-branch inference.

4. Close the aggregation and ranking gap: Grouping, HAVING, ORDER BY, and LIMIT clauses consistently produce below-average scores. Whether this reflects weak numerical reasoning in LLMs or the difficulty of mapping natural language to analytical SQL remains an open question.

Target Audience

Researchers and practitioners building Table–Text QA systems, benchmark designers working on dataset construction or synthetic data generation, and engineers deploying LLM-based analytical assistants over hybrid structured–unstructured corpora. The paper is also useful for industrial teams who want a replicable recipe for generating domain-specific evaluation sets from their own databases, and for those studying LLM reliability in SQL grounding and cross-modal reasoning.

Authors’ abstract

Real-world Table-Text question answering (QA) tasks require models that can reason across long text and source tables, traversing multiple hops and executing complex operations such as aggregation. Yet existing benchmarks are small, manually curated - and therefore error-prone - and contain shallow questions that seldom demand more than two hops or invoke aggregations, grouping, or other advanced analytical operations expressible in natural-language queries. We present SPARTA, an end-to-end construction framework that automatically generates large-scale Table-Text QA benchmarks with lightweight human validation, requiring only one quarter of the annotation time of HybridQA. The framework first constructs a reference fact database by enriching each source table with grounding tables whose tuples are atomic facts automatically extracted from the accompanying unstructured passages, then synthesizes nested queries whose number of nested predicates matches the desired hop count. To ensure that every SQL statement is executable and that its verbalization yields a fluent, human-sounding question, we propose two novel techniques: provenance-based refinement, which rewrites any syntactically valid query that returns a non-empty result, and realistic-structure enforcement, which confines generation to post-order traversals of the query graph. The resulting pipeline produces thousands of high-fidelity question-answer pairs covering aggregations, grouping, and deep multi-hop reasoning across text and tables. On SPARTA, state-of-the-art models that reach over 70 F1 on HybridQA or over 50 F1 on OTT-QA drop by more than 30 F1 points, exposing fundamental weaknesses in current cross-modal reasoning. Our benchmark, construction code, and baseline models are available at https://github.com/pshlego/SPARTA/tree/main.