Table as a Modality for Large Language Models

AI summary

Overview

• Research area: Natural Language Processing / table reasoning with Large Language Models; multimodal architecture design for structured data.
• Technical level: Intermediate (assumes familiarity with LLM prompting, LoRA fine-tuning, and graph/hypergraph encoders).
• Scope: The paper argues that tabular data should be treated as its own modality rather than serialized into text, and proposes TaMo, a hypergraph-based encoder coupled to a decoder-only LLM, plus a new diagnostic benchmark (StructQA).

What This Paper Is About

Most LLMs handle tables by flattening them into markdown-like text sequences, which discards row/column and hierarchical structure. The authors show that this loss of structure causes leading models to fail on a basic human-friendly property — permutation invariance — and propose a framework, TaMo, that encodes the table as a separate modality and injects that representation into the LLM alongside the serialized text.

Key Contributions

1. StructQA benchmark: The first open-source benchmark (7500 QA pairs from 500 tables across 5 structural reasoning tasks) for robust tabular structure understanding, evaluating direct performance, permutation accuracy under row/column shuffling, and answer robustness. It also addresses potential data contamination in existing datasets.
2. Positioning tables as a modality: The paper frames the table as an independent modality, analogous to images or audio, that a multimodal-style encoder can represent and align with text tokens in the LLM's embedding space.
3. Hypergraph-enhanced table encoder: A hypergraph construction where leaf cells are nodes and branch cells are hyperedges, processed by a hypergraph-structure-aware transformer (HyperTrans; the paper states it uses HyTrel for its effectiveness) using permutation-invariant multiset functions based on the Set Transformer.
4. Modality interface with empirical validation: A single-layer MLP projector with mean pooling aligns table-structure embeddings into the LLM's hidden dimension as a soft prompt, described as plug-and-play across decoder-only LLMs without intrusive architectural modification.

Main Findings

• Serialization loses structure: On StructQA, leading LLMs — Llama2-7B, GPT-3.5, GPT-4, and the table-specific TableLlama — degrade significantly on permuted versions of the same table. Excluding GPT-4, answer robustness was below 40%.
• Large gains over text-only input (frozen LLM): TaMo scores 59.07 / 48.86 / 37.06 / 76.45 / 36.52 on StructQA / HiTab / WikiTQ / WikiSQL / FetaQA versus prompt tuning at 37.80 / 26.26 / 29.86 / 61.24 / 29.94, with an average relative improvement of +42.65% and a maximum of +86.06% on HiTab. Zero-shot inference-only baselines were 8.60 / 7.77 / 14.50 / 21.44 / 20.08.
• Gains persist under tuning: TaMo+LoRA (70.80 / 59.22 / 43.53 / 84.43 / 37.43) improves over plain LoRA (45.67 / 50.76 / 37.13 / 57.10 / 35.80) by an average of +28.27%. TaMo+SFT (71.60 / 63.89 / 45.81 / 85.90 / 39.01) improves over plain SFT (62.73 / 54.80 / 43.28 / 79.86 / 37.37) by an average of +9.71%, and TaMo+SFT is reported as best across all tasks under the paper's settings, with TaMo+LoRA a close second on 3 of 5 datasets.
• Competitive with larger and specialist models: The 7B TaMo+SFT beats GPT-4.1 (71.60 vs 60.33) and DeepSeek-R1 (71.60 vs 57.47) on StructQA, and beats them on WikiSQL (85.90 vs 71.21 and 71.91). Larger models lead on knowledge-intensive WikiTQ (DeepSeek-R1 at 75.76). TaMo+SFT averages over +19.83 score above GPT-3.5. Specialist SOTA per dataset was 64.71 (HiTab), 69.10 (WikiTQ), 92.07 (WikiSQL), 40.50 (FetaQA).
• Better permutation robustness: On a permuted StructQA test set, TaMo outperforms pure-text methods in the frozen setting and maintains the most consistent answers after permutation across frozen, LoRA, and SFT settings.
• Structure embeddings generalize across datasets: On a binary cell row/column membership probe evaluated on WikiTQ (50 epochs, learning rate 3e-4, F1), an MLP head alone scored 5.39, a randomly initialized encoder with MLP head reached 49.73, and pretrained encoders reached 71.32 (StructQA), 68.00 (WikiSQL), 66.39 (HiTab), 64.99 (FetaQA), and 62.63 (WikiTQ).
• Interpretable attention shifts: In a WikiSQL case where the base method answered incorrectly and TaMo correctly, attention visualization (using the PromptBench gradient-based method) showed TaMo attending to "Canada" (the correct answer), "US HL" (relevant context), and the "[table_structure_token]", whereas the base method largely ignored these tokens. TaMo could also locate answers far from the question in serialized order.

Methodology in Plain English

The authors start from two human observations about tables: real tables are usually hierarchical, and swapping rows or columns does not change meaning (permutation invariance). They build a hypergraph from a table, turning each leaf cell into a node and each branching cell — headers and other container cells — into a hyperedge, so reordering rows or columns leaves the graph structure unchanged. A hypergraph transformer with permutation-invariant set functions passes information back and forth between nodes and hyperedges, producing learned embeddings for cells and groupings. A single-layer MLP with mean pooling projects that output into the LLM's hidden dimension, and the result is prepended like a soft prompt so the model "sees" global structure before reading the serialized text. Text tokens for the table and the question still go through the LLM's usual embedding layer, so structure ("where") and content ("what") remain complementary streams. Training can freeze the LLM and learn only the encoder and alignment layers, or use LoRA, or fully fine-tune everything jointly. An ablation reported in the appendix shows the graph-only variant fails on generative tasks and the full model beats the text-only counterpart.

Why This Matters

• Impact on research: It reframes table handling as a modality-alignment problem rather than a prompting or serialization problem, and introduces a diagnostic benchmark targeting a property (permutation invariance) that prior datasets largely ignored.
• Real-world applications:
  • Enterprise analytics over spreadsheets and financial statements, including hierarchical reports like HiTab.
  • Text-to-SQL and database question answering over pre-structured corporate data.
  • Fact verification and question answering over government or scientific tables.
  • Free-form table summarization and explanation (FeTaQA-style) for business reporting.
• Industry relevance: The design is described as plug-and-play with decoder-only LLMs (the experiments use Llama2-7B), needs no LLM pretraining, and has reportedly low computational overhead (appendix). That makes it attractive for organizations that want better table reasoning without training a model from scratch or switching to a table-specific architecture. The work is supported by Ant Group.

Future Directions

• Beyond pre-structured tables: TaMo assumes tables are already extracted; tables embedded in unstructured text need text-to-table preprocessing first.
• Dynamic, multi-turn table interaction: The current scope is static, single-turn table understanding; multi-step reasoning, table editing, and multi-turn dialogue over tables are left open.
• Serialization template study: The experiments used one consistent serialization format; how markdown versus SQL-based templates interact with the structural modality is untested.
• Broader multimodal instruction data and comparison to layout-aware models: Building out-of-the-box multimodal capability is left to future work, and the paper distinguishes its goal from layout-aware document models such as DocLLM, which reason over table images.

Target Audience

Researchers and practitioners working on table question answering, text-to-SQL, or tabular foundation models; LLM architects interested in adding non-text modalities to decoder-only models; and applied teams at companies with large volumes of structured data who want structural fidelity without training a specialist model per dataset. Readers need working familiarity with LLM fine-tuning (LoRA, SFT) and graph neural network concepts.

Authors’ abstract

To migrate the remarkable successes of Large Language Models (LLMs), the community has made numerous efforts to generalize them to the table reasoning tasks for the widely deployed tabular data. Despite that, in this work, by showing a probing experiment on our proposed StructQA benchmark, we postulate that even the most advanced LLMs (such as GPTs) may still fall short of coping with tabular data. More specifically, the current scheme often simply relies on serializing the tabular data, together with the meta information, then inputting them through the LLMs. We argue that the loss of structural information is the root of this shortcoming. In this work, we further propose TAMO, which bears an ideology to treat the tables as an independent modality integrated with the text tokens. The resulting model in TAMO is a multimodal framework consisting of a hypergraph neural network as the global table encoder seamlessly integrated with the mainstream LLM. Empirical results on various benchmarking datasets, including HiTab, WikiTQ, WikiSQL, FeTaQA, and StructQA, have demonstrated significant improvements on generalization with an average relative gain of 42.65%.