MATA: Multi-Agent Framework for Reliable and Flexible Table Question Answering

AI summary

Overview

• Research area: Natural Language Processing, specifically Table Question Answering (TableQA) with Large Language Model (LLM) multi-agent systems.
• Technical level: Intermediate. Readers should know what LLM inference, Chain-of-Thought reasoning, and exact-match evaluation mean, but the paper's architecture is described in accessible terms.
• Scope: This paper proposes MATA, a multi-agent TableQA framework that combines three reasoning styles with lightweight tools to improve accuracy and control inference cost across ten different LLMs.

What This Paper Is About

Most TableQA systems are built around large proprietary LLMs and rely on repeated inferences to improve reliability, which is expensive and hard to use when privacy, cost, or data-ownership constraints require open-source models. The authors address whether a system can be model-agnostic, keep accuracy high on both small and large models, and still limit how many costly LLM calls it makes. Their answer is MATA, a framework that generates candidate answers through Chain-of-Thought (CoT), Program-of-Thought (PoT), and text-to-SQL reasoning, then refines or selects among them using compact tool models and a Judge Agent.

Key Contributions

1. The authors propose MATA, a model-agnostic multi-agent framework that integrates Chain-of-Thought, Program-of-Thought, and text-to-SQL for answer generation and verification.
2. MATA is evaluated with ten LLMs spanning small models (up to 10B parameters) and large models (more than 10B parameters or closed-source), reporting strong results on three metrics: exact match, fuzzy matching, and token-level F1.
3. The framework introduces an inference-optimization design: a Scheduler that decides whether to run PoT or text2SQL first, and a Confidence Checker that lets the system skip the Judge Agent when a candidate score exceeds a threshold, reducing LLM calls.
4. The paper releases a training dataset of 173,664 samples derived from 57,888 unique table-question pairs, plus public code at a GitHub repository for reproducible research.

Main Findings

• Best baseline comparison on TableBench: MATA outperformed the strongest baseline (SynTQA) by 40.1% in exact match, 21.9% in fuzzy matching, and 33.1% in F1 on the challenging benchmark.
• Averaged benchmark scores on Penguins in a Table: MATA averaged 0.881 EM, 0.890 fuzzy, and 0.881 F1, compared with SynTQA at 0.810/0.839/0.811, MixSC at 0.626/0.674/0.637, and TabLaP at 0.524/0.593/0.544.
• Averaged benchmark scores on TableBench: MATA averaged 0.451 EM, 0.619 fuzzy, and 0.482 F1, compared with SynTQA at 0.322/0.471/0.362, MixSC at 0.286/0.508/0.331, and TabLaP at 0.260/0.480/0.307.
• Model-agnostic behavior: On Penguins in a Table, MATA maintained consistently high performance across all ten models, while TabLaP and MixSC showed significant declines with smaller LLMs. On TableBench, MATA achieved the best performance across all models.
• SynTQA's limited generalizability: SynTQA led on the easier benchmark for some small LLMs but did not keep that lead on harder tasks, and was sometimes outperformed by TabLaP and MixSC, especially with closed-source models.
• Confidence Checker is the most critical component: Removing it caused the largest performance drop in the ablation study, because the system then had to rely solely on the Judge Agent to evaluate all candidates.
• Reduced Judge Agent calls: Using the Confidence Checker reduced Judge Agent invocation frequency by 95.8% on Penguins in a Table and by 60.6% on TableBench, while maintaining or improving accuracy.
• Scheduler efficiency gain: Using the Scheduler led to a 14.6% reduction in LLM agent calls on Penguins in a Table and 7.6% on TableBench, aggregated across all ten models.
• Latency comparison: Average end-to-end latency per query for locally hosted open-source backbones was 27.55 seconds for MATA, versus 48.89 for TabLaP, 44.48 for MixSC, and 6.86 for SynTQA. Latency ranged from 17.77 seconds (qwen2.5-3b) to 47.31 seconds (mistral-small-24b) for MATA.
• Tool overhead is small: The combined latency of the lightweight Scheduler, Confidence Checker, and Format Matcher accounts for about 0.6% of MATA's total latency.
• Reasoning-path strengths vary by model: In the appendix analysis of 20 models (18 open-source from 3B to 32B and two closed-source) on TableBench, CoT performed well on fact-checking and data interpretation, while PoT and text2SQL were better for complex arithmetic and aggregation, with these trends most evident in models with 10B+ parameters or closed-source configurations.

Methodology in Plain English

MATA splits its work between two kinds of components. "Tools" are very small models (under 500M parameters) that make quick judgments, while "agents" are LLM-based modules (at least 3B parameters) that do the actual reasoning, all sharing the same backbone model and differing only in their role-specific prompts.

The pipeline works in three stages. First, the CoT Agent reads the table and question once and produces a text-based answer. At the same time, the Scheduler, a 24.65M-parameter model built from MobileBERT plus a two-layer MLP, looks at table features such as size, schema, and data types along with the question's meaning, and decides whether to run the PoT agent or the text2SQL agent first. If that chosen agent's answer matches the CoT answer, the third agent is skipped.

Second, the code-generating agents run with a debugging loop: the agent writes code, the code executes, and a dedicated debug agent (Python Debug Agent or SQL Debug Agent) fixes errors and re-executes, for up to N cycles. The authors found N = 3 sufficient. The loop stops early if the newly generated code is substantially similar to the previous version and produces identical execution results.

Third, all candidate answers go to the Confidence Checker, a fine-tuned DeBERTaV3-large with about 435M parameters. It scores each candidate. If any score exceeds a threshold θ (set to 0.1 through hyperparameter tuning), the highest-scoring answer wins and the Judge Agent is not called at all. Otherwise the Judge Agent adjudicates. Finally, if the chosen answer exceeds 100 characters, the Format Matcher (a 500M-parameter qwen2.5-instruct used without fine-tuning) extracts a concise entity.

The tools were trained on a dataset built from WikiTQ, TabMWP, and TabFact, with inferences run through phi4-14B, Qwen2.5-Coder-14B, and CodeLLaMA-13B across all three reasoning modes. Evaluation used two benchmarks deliberately chosen to differ from that training distribution: Penguins in a Table (easy) and TableBench (hard, with 18 subcategories). The framework was implemented with LangChain and Ollama, and local runs used a single A100 GPU.

Why This Matters

• Impact on research: The paper shows that careful orchestration of multiple reasoning paths, rather than simply scaling up model size or the number of inference calls, can produce state-of-the-art TableQA accuracy. It also challenges the assumption that high-performing TableQA requires proprietary models, and it argues that exact match alone is insufficient, adding fuzzy matching and token-level F1 alongside it.
• Real-world applications:
  • Business analytics and reporting tools that let non-technical users query spreadsheets or databases in natural language, the scenario the introduction highlights.
  • Privacy-sensitive deployments where a company runs open-source LLMs locally because data cannot be sent to paid APIs.
  • Financial, scientific, and sports data exploration, mirroring the domains TableBench draws its tables from.
  • Low-resource or cost-constrained settings where repeated calls to large proprietary models are impractical.
• Industry relevance: The latency results matter for deployment. MATA's 27.55-second average is slower than SynTQA's fixed 3-call approach at 6.86 seconds, but much faster than the call-heavy TabLaP (48.89) and MixSC (44.48) that made 12 and 10 LLM calls respectively. MATA sits between these extremes: it spends more computation when the question demands it, and exits early when confidence is high. That trade-off is directly relevant to teams choosing between accuracy and serving cost.

Future Directions

• Reduce backbone LLM cost: The authors state in the limitations that MATA does not address the fundamental expense of the LLMs themselves, since every reasoning path still requires full LLM inference. They suggest LLM compression, distillation, or hybrid architectures that offload part of the reasoning to smaller or non-LLM components.
• Extend with more techniques and agents: The conclusion notes future work may integrate additional techniques or agents to broaden applicability across diverse scenarios and data complexities.
• Resolve the diversity-efficiency trade-off: The ablation results show that removing the Scheduler sometimes improved accuracy on TableBench but slightly reduced answer diversity on easier tasks, so a better policy for choosing reasoning paths remains open.
• Improve answer extraction robustness: The Format Matcher was introduced because backbone LLMs sometimes produce overly verbose answers due to limited instruction-following ability; handling this more fundamentally is an unresolved issue.

Target Audience

This paper is most useful to NLP researchers working on table understanding, question answering, and multi-agent LLM systems, as well as practitioners building natural-language interfaces to structured data who need to work with open-source models. Engineers evaluating accuracy-versus-latency trade-offs in production TableQA pipelines will find the efficiency measurements and the ablation of module contributions directly applicable. Readers interested in benchmarking methodology will also benefit from the paper's case for using exact match, fuzzy matching, and token-level F1 together rather than exact match alone.

Authors’ abstract

Recent advances in Large Language Models (LLMs) have significantly improved table understanding tasks such as Table Question Answering (TableQA), yet challenges remain in ensuring reliability, scalability, and efficiency, especially in resource-constrained or privacy-sensitive environments. In this paper, we introduce MATA, a multi-agent TableQA framework that leverages multiple complementary reasoning paths and a set of tools built with small language models. MATA generates candidate answers through diverse reasoning styles for a given table and question, then refines or selects the optimal answer with the help of these tools. Furthermore, it incorporates an algorithm designed to minimize expensive LLM agent calls, enhancing overall efficiency. MATA maintains strong performance with small, open-source models and adapts easily across various LLM types. Extensive experiments on two benchmarks of varying difficulty with ten different LLMs demonstrate that MATA achieves state-of-the-art accuracy and highly efficient reasoning while avoiding excessive LLM inference. Our results highlight that careful orchestration of multiple reasoning pathways yields scalable and reliable TableQA. The code is available at https://github.com/AIDASLab/MATA.