FACTS: Table Summarization via Offline Template Generation with Agentic Workflows

AI summary

Overview

Research area: Natural Language Processing — query-focused table summarization, agentic LLM workflows, and privacy-preserving text generation.

Technical level: Advanced. The paper assumes familiarity with LLM prompting, agentic pipelines, SQL, and templating engines (Jinja2), along with standard summarization metrics (BLEU, ROUGE-L, METEOR).

Scope: The paper introduces FACTS, an agentic workflow that generates reusable "offline templates" — SQL queries plus a Jinja2 rendering template — so that a single LLM-generated artifact can summarize many tables sharing the same schema, without exposing raw table values to external models.

What This Paper Is About

Query-focused table summarization means producing a natural-language narrative about a table, conditioned on what the user actually asked. Existing methods sit at two unsatisfying extremes: prompt-based approaches paste the whole table into an LLM (slow, token-limited, and privacy-risky), while table-to-text models need costly fine-tuning and still struggle with numerical and logical reasoning. FACTS instead generates the summarization program — SQL for retrieving values and a Jinja2 template for verbalizing them — once per query and schema, so it can be reused and executed locally on any number of tables.

Key Contributions

1. Offline template generation. The paper proposes a new artifact, the offline template, bundling schema-aware SQL queries with a Jinja2 template. It is bound to a table schema and query semantics, and is reusable across any tables that differ only in their values (for example, multiple years of financial records or multiple patients' health records, as the paper puts it).

2. The FACTS agentic workflow. A three-stage pipeline: (1) schema-guided specification and filtering, which produces guided questions and abstract filtering rules without the LLM ever seeing raw table contents; (2) SQL query generation with local execution feedback and iterative repair; (3) Jinja2 template generation and alignment, which checks that the template's field references match the SQL output.

3. The LLM Council. Instead of a single critic model, an ensemble of heterogeneous LLMs (GPT-4o-mini, Claude-4 Sonnet, and DeepSeek v3) each independently returns a YES/NO judgment plus brief feedback; majority voting decides acceptance and aggregated feedback drives refinement. The Council validates at four points: guided questions/filtering rules, SQL queries, SQL–template alignment, and the final summary.

4. Empirical validation and a four-property comparison. Evaluations on FeTaQA, QTSumm, and QFMTS, plus human evaluation, human preference study, and reusability/scalability experiments. The paper's comparison table (Table 2, Appendix A) argues FACTS is the only approach satisfying all four desired properties — reusable, scalable, accurate, and privacy-compliant — while prior agentic methods such as TaPERA and SPaGe only produce partially reusable plans.

Main Findings

• Automatic metrics (RQ1). FACTS achieves the best or second-best result across all datasets and metrics. On FeTaQA it records BLEU 32.6, ROUGE-L 58.9, METEOR 67.7. On QTSumm it records BLEU 21.9, ROUGE-L 45.8, METEOR 51.3. On QFMTS it records BLEU 46.0, ROUGE-L 70.8, METEOR 73.2. SPaGe remains the strongest competitor (FeTaQA BLEU 33.8 and ROUGE-L 55.7; QFMTS BLEU 45.7 and METEOR 73.4), and the paper states FACTS outperforms SPaGe on every dataset in at least two of the reported metrics.

• Council diversity helps, but is not required. The FACTS (GPT-Only) variant, where all three Council members are GPT-4o-mini, still beats baselines on most datasets and metrics — for example FeTaQA ROUGE-L 55.7 and QFMTS ROUGE-L 70.5 — showing the core workflow is effective without cross-model diversity.

• Computation cost. Per sample, FACTS uses on average 2.47 accepted guiding questions or filtering rules (2.25 initially accepted and 0.22 accepted after one round of revision), 1.36 SQL refinement rounds, and 1.84 template refinement rounds, with maximum patience set to three rounds. The full workflow consumes 9,922 input tokens and 1,045 output tokens per sample on average, including all stages and Council outputs.

• Human evaluation (RQ2). Across 100 randomly sampled QTSumm examples and 100 QFMTS examples, FACTS reaches 97% intent match, 94% SQL execution accuracy, and 98% template rendering accuracy, with a Council consensus error rate of about 3%. No errors occurred during the specification stage, while about 6% of Council-approved SQL queries led to incorrect results during generation, yielding the ~3% average. The paper estimates overall factual correctness at 94% × 98% ≈ 92%.

• Human preference study. Against SPaGe on QFMTS, in a blinded side-by-side comparison, FACTS is preferred in 55% of cases for completeness, 59% for correctness, and 60% for hallucination reduction.

• Reusability (RQ3). With a single table, FACTS is slightly slower than Reason-then-Summ because it must generate the offline template for the first time, and comparable to SPaGe; that reported latency already includes initial template generation. With 100 tables under the same schema, FACTS wins, since new summaries need only SQL execution and Jinja2 rendering while baselines reprocess every table.

• Robustness to paraphrased queries. Applying the original templates without regeneration to GPT-5-paraphrased queries and reference summaries on 100 sampled QTSumm examples, FACTS maintains BLEU 21.8, ROUGE-L 43.5, and METEOR 50.8.

• Scalability. As table size grows from 100 to 1000 rows, FACTS runtime stays flat because templates depend only on the schema, while Reason-then-Summ and SPaGe grow steadily as larger tables must be serialized into the LLM.

• Acknowledged limitations. Reusability assumes an identical schema; schema drift (renamed columns or changed data types) may break pre-generated SQL and templates, and FACTS does not currently detect or adapt automatically. Privacy compliance assumes a threat model where raw values stay local but schemas and user queries are exposed — the paper notes this fails under zero-trust policies regarding schema exposure, and that no obfuscation of such metadata is employed.

Methodology in Plain English

The researchers reframe the task: rather than asking an LLM to write a summary of a specific table, ask it to write a small program that will produce that summary for any table with the same shape.

The workflow runs in three stages. First, given only the user query and the table schema (column names and data types, not values), an agent writes clarification questions and filtering rules in abstract form — for example, "exclude rows where category='expense'" — which later become SQL WHERE clauses. Second, the agent writes candidate SQL queries, runs them locally, and feeds back errors or empty results for repair. Third, the agent writes a Jinja2 template that turns the SQL rows into fluent prose, handling empty results gracefully; the system then checks that every field the template references is actually returned by the SQL, refining both together until they are consistent.

At every stage an LLM Council of three different models votes YES/NO on the candidate artifact and returns short feedback. Majority vote decides acceptance, and the aggregated feedback is what the agent uses to revise. This loop is what the authors credit for correctness, since the models never see the data itself — only structure and code.

Baselines are all training-free and fine-tuning-free, split into prompt-based methods (CoT, DirectSumm, ReFactor, Reason-then-Summ) and agentic frameworks (Binder, Dater, TaPERA, SPaGe). BLEU, ROUGE-L, and METEOR are reported following Zhang et al. (2025). The main agent backbone is GPT-4o-mini.

Why This Matters

The paper targets a practical gap: organizations in finance, healthcare, and law need custom summaries of their own tables, but regulations such as HIPAA and GDPR prohibit sending individual-level records to external LLM services. FACTS' claim is that by shipping only schemas and queries outward, and executing SQL locally, it can get LLM-quality summaries without that exposure — and at a cost that amortizes across many tables.

Research impact: It introduces offline template generation as a task formulation and supplies a comparison framework (reusable / scalable / accurate / privacy-compliant) for evaluating table summarization methods, plus a Council-based validation mechanism that other agentic pipelines can adopt.

Real-world applications:

• Financial reporting: the paper's running example is an analyst requesting gross income summaries for each of the past ten years — ten tables with the same schema, handled by one template rather than ten LLM generations.
• Healthcare records: the paper explicitly mentions summarizing multiple patients' health records under a shared schema without moving raw values off-premises.
• Legal and compliance review: narrative summaries over repeated document-structured tables where confidentiality of records matters.
• Enterprise analytics at scale: recurring dashboard-style queries over tables that grow over time, where runtime staying flat as rows scale from 100 to 1000 rows is the operational selling point.

Industry relevance: Deployment-time economics are the headline — token costs drop to per-sample figures of 9,922 input and 1,045 output tokens, and reuse means no repeated inference. The tradeoff is front-loaded: the first summary costs more than a direct prompt.

Future Directions

• Handling schema drift. The paper states FACTS cannot currently detect or adapt to renamed columns or modified data types without regenerating the template, and flags automatic detection or adaptation as an open gap.
• Strengthening the privacy model. Since schemas and user queries are still exposed, the authors point to obfuscation of metadata as a needed extension for environments with zero-trust policies around schema exposure.
• Improving SQL generation reliability. Roughly 6% of Council-approved SQL queries still produced incorrect results, and the specification stage produced no errors — making SQL generation the clear weak link that further validation could target.
• Broadening evaluation and Council analysis. The paper's benchmark set is three public datasets with the preference study limited to QFMTS against a single baseline; how Council composition and template reusability behave under semantically adjacent but not paraphrased queries, or across multi-table schemas, remains open.

Target Audience

Researchers and practitioners working on table understanding, agentic LLM pipelines, and privacy-preserving NLP — particularly those evaluating or deploying summarization systems inside regulated industries. It will also interest engineers who need reproducible, training-free baselines: the code is released at https://github.com/BorealisAI/FACTS, and the appendices include full prompts and pseudocode. Readers without background in SQL, Jinja2 templating, or agentic evaluation loops will find the methodology sections demanding, though the task framing and results tables are accessible.

Authors’ abstract

Query-focused table summarization requires generating natural language summaries of tabular data conditioned on a user query, enabling users to access insights beyond fact retrieval. Existing approaches face key limitations: table-to-text models require costly fine-tuning and struggle with complex reasoning, prompt-based LLM methods suffer from token-limit and efficiency issues while exposing sensitive data, and prior agentic pipelines often rely on decomposition, planning, or manual templates that lack robustness and scalability. To mitigate these issues, we introduce an agentic workflow, FACTS, a Fast, Accurate, and Privacy-Compliant Table Summarization approach via Offline Template Generation. FACTS produces offline templates, consisting of SQL queries and Jinja2 templates, which can be rendered into natural language summaries and are reusable across multiple tables sharing the same schema. It enables fast summarization through reusable offline templates, accurate outputs with executable SQL queries, and privacy compliance by sending only table schemas to LLMs. Evaluations on widely-used benchmarks show that FACTS consistently outperforms baseline methods, establishing it as a practical solution for real-world query-focused table summarization. Our code is available at https://github.com/BorealisAI/FACTS.