Decoupling Skeleton and Flesh: Efficient Multimodal Table Reasoning with Disentangled Alignment and Structure-aware Guidance

AI summary

Overview

Research area: Multimodal table understanding and reasoning with Large Vision-Language Models (LVLMs); the paper is listed under Computer Vision (cs.CV) and keywords "Machine Learning, ICML."

Technical level: Intermediate. The paper assumes familiarity with LVLM alignment, supervised fine-tuning, LoRA, and standard table benchmarks, but its two core ideas (disentangled alignment and global-to-local structure-guided inference) are explained conceptually.

Scope in one sentence: The paper proposes and evaluates a two-part framework — DiSCo for disentangled structure–content alignment and Table-GLS for training-free, tool-free global-to-local table reasoning — to adapt LVLMs to table images with minimal annotation.

Authors are affiliated with Harbin Institute of Technology (Shenzhen, China) and Peng Cheng Laboratory (Shenzhen, China). Code and data are released at https://github.com/AAAndy-Zhu/TableVLM.

What This Paper Is About

Tables combine structural organization (rows, columns, cell layout) with semantic content (the text inside cells), and LVLMs struggle when layouts are complex, data is dense, or structural dependencies are intricate. Existing fixes either require expensive expert annotations and reinforcement learning or add external tools such as visual editors, which increase system complexity and latency without improving the model's intrinsic structural understanding. The paper asks whether LVLMs can be adapted to table reasoning with minimal annotation cost and no external tools, and answers with a disentangled alignment stage (DiSCo) plus a structure-guided inference procedure (Table-GLS).

Key Contributions

1. DiSCo, a disentangled structure–content alignment framework. It separates structure abstraction (learning table layouts independently of cell content) from semantic grounding (linking content to that structure via global and local natural-language descriptions), improving understanding of complex and unseen table layouts.

2. Table-GLS, a global-to-local structure-aware inference framework. It performs table reasoning at inference time without external tools or additional reasoning-oriented fine-tuning, by exploring global structure, extracting a self-refined sub-table as evidence, and reasoning over that evidence.

3. Efficiency claim anchored in experiments. Using only 10K table images for alignment, the combined framework reports improvements across 21 table understanding and reasoning tasks and benchmarks, with particular gains on unseen table structures.

4. A disentanglement-by-design argument. The paper argues that alignment quality (explicitly separating structure from semantics) matters more than alignment data quantity, supported by comparisons against models aligned on 97K table images (150K image–text pairs).

Main Findings

• DiSCo improves understanding across models and tasks. On MMTab understanding tasks — table size detection (TSD), table cell extraction (TCE), table cell location (TCL), merged cell detection (MCD), and row & column extraction (RCE) — DiSCo consistently improved results across all evaluated LVLMs and tasks, with particularly evident gains on structure-sensitive tasks such as TSD, TCL, RCE, and TCE.

• Data efficiency beats data volume. DiSCo with only 10K table images achieved comparable or superior performance to Textual-All, which uses the full 150K alignment samples from MMTab, across most understanding tasks. On Qwen3-VL-8B, DiSCo outperformed textual alignment at every data scale from 5K to 20K images; with only 5K images it already surpassed textual alignment with 10K, performance peaked at 10K, and remained stable at 15K and 20K.

• Both alignment components matter. In the ablation, DiSCo without local content alignment (w/o T_L) already improved over the Textual baseline, showing that structure alignment alone helps layout understanding; adding local content alignment further boosted structure-sensitive tasks (TSD, TCL, RCE) and out-of-domain robustness.

• Bigger models benefit more. The paper reports that larger models such as Qwen3-VL-32B benefit more substantially from alignment, attributing this to higher-capacity LVLMs better internalizing disentangled structure and semantic signals.

• Table-GLS improves reasoning over direct answering. On question answering (WTQ, HiTab, TAT-QA, AIT-QA, TabMCQ) and fact verification (TabFact, InfoTabs, PubHealthTab), Table-GLS outperformed direct answering on almost all benchmarks. Combined scores: for Qwen3-VL-8B the average rose from 50.31 (direct answering) to 59.48; for Gemma3n-E4B from 42.56 to 50.87.

• The combination gives the best averages and out-of-domain gains. DiSCo + Table-GLS achieved the best average score for both Gemma3n and Qwen3-VL, with notable gains on out-of-domain benchmarks such as AIT-QA and PubHealthTab for Qwen3-VL. On Qwen3-VL-32B the full combination reached an average accuracy of 71.84%, versus 69.16% with DiSCo alone and 66.12% for the base model.

• Works on a closed-source model too. Applying Table-GLS to GPT-5.4-mini raised average accuracy from 56.64% to 66.16%.

• Both reasoning stages are necessary. Removing Global Structure Exploration (-GSE) caused substantial drops on all benchmarks (e.g., AIT-QA 76.71 to 62.82; HiTab 27.35 to 24.30); removing Self-refined Sub-table Extraction (-SSE) also dropped most benchmarks (AIT-QA to 73.39; InfoTabs 72.67 to 70.20; PubHealthTab 77.14 to 73.94), though HiTab improved slightly (27.35 to 31.41), which the authors attribute to HiTab's complex nested structures where imperfect sub-table extraction may add noisy evidence.

• Table-GLS beats chain-of-thought and row-of-thought. Compared with CoT and Row-of-Thought (RoT), Table-GLS yielded more stable and superior performance on most benchmarks. DiSCo also improved both CoT and RoT on out-of-domain datasets such as AIT-QA and PubHealthTab.

• Prompt design is not the source of gains. Rephrased and prefixed prompt variants produced only minimal average accuracy changes (70.91 full, 69.89 rephrased, 70.27 prefixed), indicating robustness to reasonable prompt modification.

• Self-refinement is reliable and preferable to explicit verification. Removing the revision step dropped average accuracy from 70.91 to 68.56, while adding an explicit hallucination check produced 70.24 — no gain. In a human check on 50 randomly sampled instances, four undergraduates judged the model to have correctly identified and revised errors in 78%, 80%, 76%, and 72% of cases respectively.

• Favorable efficiency–accuracy trade-off. On three datasets (VWTQ, VWTQ_syn, VTabFact), DiSCo & Table-GLS reached 91.20% accuracy on VTabFact in 12.11s, compared with ReFocus at 88.40% in 20.70s and HIPPO at 86.00% in 0.86s. On VWTQ, ReFocus achieved 68.00% in 23.96s, DiSCo & Table-GLS 62.27% in 14.89s, and HIPPO 55.87% in 0.72s.

• General multimodal ability is preserved, not degraded. On non-tabular benchmarks, textual alignment degraded some tasks relative to the original model, while DiSCo improved all evaluated ones for Qwen3-VL-8B: ScienceQA 94.79 to 95.09, CRPE 77.68 to 77.92, HallusionBench 73.5 to 74.97, and TextVQA 80.34 to 80.83.

• Layout sensitivity. DiSCo's largest gains were on relatively small, compact tables (TABMWP and WikiBiO); it remained robust on high-density tables such as ToTTo (average 35 rows) and Rotowire (average 33 rows and 19 columns), and maintained a substantial margin on out-of-domain benchmarks such as LoTNLG and PubHealthTab.

• Understanding correlates with reasoning. The paper reports a clear positive correlation: benchmarks with higher understanding scores yielded stronger reasoning. Direct answering showed the weakest correlation, CoT was more consistent, and Table-GLS achieved higher reasoning scores at comparable understanding levels.

Methodology in Plain English

Step 1 — DiSCo: teach structure and content separately instead of together. Conventional alignment feeds the model an entire table serialized as HTML, Markdown, or LaTeX, which forces it to learn layout and meaning at the same time. DiSCo splits this into two objectives. For structure alignment, all cell text in the serialized table is replaced with a single placeholder token, leaving only layout cues such as row and column delimiters, hierarchical headers, and span markers; the model is trained to predict this anonymized structure from the table image alone, so it cannot lean on remembering cell values. For content alignment, the model is trained on two complementary tasks: a global task producing a lightweight semi-structured description of the table (total number of rows and columns, plus a concise summary of what each row and column contains), and a local task that, given a specific row index and column index, outputs the corresponding cell content. This separation makes the model ground meaning onto explicit structural coordinates.

Step 2 — Table-GLS: reason in three stages without training or tools. At inference time the model first performs Global Structure Exploration, analyzing headers, row labels, and their semantic roles and naming which rows and columns matter, outputting a short rationale plus target row and column lists — a "where to look" decision before any content is read. Second, Self-refined Sub-table Extraction asks the model to judge whether its chosen rows and columns are correct and sufficient, revise the plan if needed, and then extract a minimal sub-table containing only the needed information — a plan-before-extract discipline that limits error propagation. Third, Evidence-grounded Reasoning constrains the final answer derivation to that sub-table, with the original table image kept only as auxiliary visual context, separating evidence selection from answer derivation.

Step 3 — Training and evaluation setup. DiSCo was trained by sampling 10K table images from MMTab's pre-training corpus and building paired structure-alignment and content-alignment instances. Four LVLMs were fine-tuned with LoRA to preserve original performance and mitigate catastrophic forgetting: Gemma3-12B, Gemma3n-E4B-it, LLaVA-v1.6-7B, and Qwen3-VL-8B-Instruct. Reasoning evaluations used Gemma3n-E4B and Qwen3-VL-8B-Instruct with the vLLM framework, all in a zero-shot setting. Comparisons included Textual (10K), Textual (97K), Textual-All (97K), TableLlama variants, Table-LLaVA 7B/13B, Table-R1, HIPPO, ReFocus, GPT-4o-mini, and GPT-5.4-mini.

Why This Matters

Impact on research. The paper challenges the assumption that strong multimodal table reasoning requires heavy supervised fine-tuning, reinforcement learning, or external tools. Its central claim — that explicitly disentangling structure from content during alignment transfers an LVLM's existing textual-semantic reasoning to table layouts more data-efficiently — suggests alignment objective design, rather than data scale, as a lever for table-centric adaptation. The reported 10K-versus-150K-sample comparison is the concrete evidence for that claim, and the observation that larger models benefit more points to a scaling relationship worth investigating. The finding that textual alignment can degrade non-tabular abilities while DiSCo does not also has implications for how specialized adaptation is evaluated.

Real-world applications.

• Financial reporting and analysis, where tables in filings and statements must be read and queried accurately.
• Scientific and medical document processing, including extracting and reasoning over tables in articles and medical records.
• Government and administrative documents, where dense, variably formatted tables are common.
• General document QA and fact verification over scanned or photographed tables in settings where no bounding-box annotations or editing tools are available — a limitation the paper explicitly raises for tool-augmented approaches like ReFocus.

Industry relevance. The framework targets deployment constraints directly: 10K alignment images rather than tens of thousands of supervised reasoning examples, no external tools at inference, moderate latency (for example 12.11s per sample on VTabFact versus ReFocus's 20.70s while scoring higher), and demonstrated gains on a closed-source model (GPT-5.4-mini), suggesting the inference procedure is portable across model families rather than tied to one open checkpoint. The released code and data lower the barrier to reproducing and extending the approach.

Future Directions

• Handling nested and hierarchical table structures. The ablation showed that removing Self-refined Sub-table Extraction actually improved HiTab, which the authors link to complex nested structures where imperfect sub-table extraction introduces noisy evidence. Better extraction for nested headers is an open problem.

• Closing the reasoning gap with tool-augmented methods on simpler tables. On VWTQ_syn, ReFocus reached 70.80% versus 68.40% for DiSCo & Table-GLS, and on VWTQ ReFocus reached 68.00% versus 62.27%, so the training-free, tool-free setting is not yet uniformly ahead.

• Reducing inference latency further. HIPPO runs at under a second per sample across the three measured datasets (0.72s, 0.75s, 0.86s) versus 12.11–15.19s for DiSCo & Table-GLS; whether the multi-stage reasoning can be compressed without losing accuracy is not resolved.

• Scaling the disentangled alignment recipe. The paper reports that gains grow with model capacity and that performance is stable from 10K to 20K images, but the behavior beyond this range, and on models not tested here, is not reported. The token-efficiency analysis of the various reasoning strategies is placed in Appendix G and its figures are not given in the main content.

Target Audience

Researchers and engineers working on multimodal document and table understanding, LVLM alignment, and parameter-efficient adaptation (the work uses LoRA throughout). It is most useful to readers who already understand vision-language model training pipelines and table benchmarks such as MMTab, WTQ, HiTab, TAT-QA, AIT-QA, TabMCQ, TabFact, InfoTabs, and PubHealthTab, and who want a data-efficient alternative to supervised fine-tuning, reinforcement learning, or tool-augmented inference. Practitioners building document-QA or table-extraction systems with constrained annotation budgets and no access to bounding-box metadata are the secondary audience.

Authors’ abstract

Reasoning over table images remains challenging for Large Vision-Language Models (LVLMs) due to complex layouts and tightly coupled structure-content information. Existing solutions often depend on expensive supervised training, reinforcement learning, or external tools, limiting efficiency and scalability. This work addresses a key question: how to adapt LVLMs to table reasoning with minimal annotation and no external tools? Specifically, we first introduce DiSCo, a Disentangled Structure-Content alignment framework that explicitly separates structural abstraction from semantic grounding during multimodal alignment, efficiently adapting LVLMs to tables structures. Building on DiSCo, we further present Table-GLS, a Global-to-Local Structure-guided reasoning framework that performs table reasoning via structured exploration and evidence-grounded inference. Extensive experiments across diverse benchmarks demonstrate that our framework efficiently enhances LVLM's table understanding and reasoning capabilities, particularly generalizing to unseen table structures. Our data and code are available at https://github.com/AAAndy-Zhu/TableVLM.