TabReX : Tabular Referenceless eXplainable Evaluation

AI summary

Overview

Research area: Natural Language Processing — automatic evaluation metrics for LLM-generated structured data (tables), with connections to knowledge-graph extraction, LLM-as-judge systems, and benchmark design.

Technical level: Intermediate. The framework is conceptually approachable, but the scoring equations, rank-correlation statistics (Spearman's rho, Kendall's tau, RBO), and perturbation taxonomy assume some familiarity with NLP evaluation methodology.

One-sentence scope: The paper proposes TabReX, a reference-less, graph-based, property-driven metric for judging how faithfully a generated table reflects its source text, plus TabReX-Bench, a 9,120-instance perturbed benchmark used to test metric robustness across six domains and three difficulty tiers.

What This Paper Is About

Existing metrics for evaluating LLM-generated tables either flatten tables into plain text — losing row/column structure and unit semantics — or depend on fixed reference tables that limit generalization. TabReX addresses this by converting both the source text and the generated table into knowledge graphs, aligning them with an LLM, and producing interpretable penalties that separate structural errors from factual ones. The goal is a scoring system that is reference-less, tunable between coverage and hallucination control, and explainable down to the individual cell.

Key Contributions

1. TabReX, a reference-less, property-driven evaluation framework that aligns table graphs with text graphs and computes interpretable, rubric-aware scores spanning TablePenalty and CellPenalty components.
2. TabReX-Bench, a large systematically perturbed dataset spanning six datasets, 710 source tables, 12 perturbation types per table, and 9,120 perturbed instances across Easy/Medium/Hard difficulty tiers, generated by a planner-driven pipeline rather than per-edit LLM calls.
3. Empirical results showing strong correlation with human rankings and greater stability than baselines under harder perturbations, including a reference-less setting where the metric still performs competitively.
4. Rubric-wise analyses demonstrating cell-level and table-level diagnostics that compare models and prompting strategies, supporting model-versus-prompt alignment studies.

Main Findings

• Correlation with human rankings (synthetic perturbations): In Table 2, TabReX reaches Spearman's rho of 74.51, Kendall's tau of 64.24, weighted Kendall's tau of 62.28, RBO of 44.85, Spearman's footrule distance of 27.01, and a tie ratio of 13.59 percent. Among reference-less metrics it leads QuestEval, which scores 62.93, 52.29, 51.71, 42.70, 35.04, and 03.03 respectively.

• TabXEval scores higher on raw rank correlation but saturates: TabXEval attains the strongest overall correlation in Table 2 (rho 80.27, tau 72.37, weighted tau 66.87, RBO 47.54, footrule 20.94), yet its tie ratio of 45.33 percent and moderate rank dispersion indicate frequent scoring saturation. The paper positions TabReX as the most balanced performer across all dimensions and the best overall consistency with human judgment.

• Non-LLM metrics perform poorly: Exact Match, chrF, and ROUGE-L show rho below 0.45 and tau below 0.35. Embedding metrics such as BLEURT and BERTScore show RBO around 0.39 and footrule distances around 45 to 53, with tie ratios under 2 percent, indicating coarse differentiation.

• Ensembles do not close the gap: Table 3 compares Lex-Emb, LLM, and Hybrid ensembles using mean or harmonic aggregation. The best ensemble, LLM (Harmonic), reaches rho 56.00 and tau 46.93, versus TabReX at 74.51 and 64.24, with TabReX also showing lower rank dispersion.

• Stability under harder perturbations: In Figure 1, metrics such as TabXEval and TabReX show short arrow movements from easy to hard cases, while EM, H-Score, and the LLM-based P-Score experience sharp drops in sensitivity. TabReX stays closest to the green "ideal zone" across difficulty levels despite being reference-less.

• Real-world text-to-table evaluation: In Table 4, on generated tables from Gemma-3-(4/27B) and InternVL-3.5-thinking (on/off) with zero-shot, CoT, and Map&Make prompting, TabReX achieves the highest correlations: Spearman's rho 0.39, Kendall's tau-b 0.30, RBO 0.41. By comparison, QuestEval (reference-less) scores 0.28, 0.20, 0.39, and TabXEval scores 0.24, 0.17, 0.37.

• Rubric-wise model–prompt insights (Figure 4): Larger models such as Gemma 27B improve cell-level local fidelity, especially for numeric and structural rubrics, but show only modest gains in semantic consistency. Reasoning-oriented "Thinking" variants improve numeric and structural precision yet often underperform on partial or contextual agreement. Chain-of-Thought prompting enhances numeric correctness but sometimes amplifies inconsistency, while Map&Make maintains more balanced but slightly conservative performance. At table level, model-size gains narrow, and prompt design contributes as much as model scale to balanced rubric alignment.

• Benchmark validation: All perturbations and summaries were validated on 20 percent of the data, achieving inter-annotator agreement of 87 percent for summaries and 91 percent for perturbations. TabReX-Bench comprises 710 source tables with 12 perturbations each, totaling 9,120 perturbed instances, compared with TabxBench's 50 reference tables and 5 perturbations each.

Methodology in Plain English

The pipeline has three stages.

Stage 1 — Turn both inputs into graphs. Source text is converted into knowledge-graph triplets [subject, predicate, object] using an LLM guided by a strict entity-centric grammar: subjects are entities or time slices, predicates are normalized properties, and objects are canonical, unit-aware values. Tables are converted deterministically by rule-based unrolling: headers become predicates, each row becomes a subject, and every non-empty cell becomes a triplet. The authors implemented both RuleHTMLConverter and RuleMDConverter, using the latter in this work.

Stage 2 — Align the two graphs. Because there is no reference table, the generated table's graph is aligned against the source text's graph. A deterministic pass first matches triplets with identical or schema-normalized subject–predicate pairs; an LLM-assisted refinement then handles the remainder, resolving paraphrases, abbreviations, and compound attributes (for example, "GDP growth (YoY)" against "growth_rate_2021"). Each matched pair receives a difference vector recording unit-aware numeric gaps, categorical mismatches, and whether a fact is missing from the table or extra relative to the source.

Stage 3 — Score with interpretable penalties. Alignment statistics — counts of missing (MI) and extra (EI) entities and partial matches — feed two components. TablePenalty captures structural discrepancies at the row and column level. CellPenalty captures factual fidelity, including a term for the sum of normalized numeric deviations over partially aligned cells. The final score is simply TablePenalty plus CellPenalty. Weighting parameters α and β let users trade off sensitivity (rewarding coverage, via β_MI) against specificity (penalizing hallucinated entries, via β_EI). Default hyperparameters are β_MI = 1.0, β_EI = 0.9, β_partial = 0.8, α_r = 0.9, α_c = 1.0, α_cell = 0.8, and ω_p = 0.9.

A worked example in Appendix B uses N_r = 5, N_c = 4, N_cell = 20, with MI_r = 1, EI_c = 1, MI_cell = 2, EI_cell = 1, and two partially aligned cells with normalized deviations of 0.2 and 0.5. This yields TablePenalty = 0.405, CellPenalty = 0.1362, and a final score of 0.5412.

For the benchmark, an LLM-based planner generates executable code producing all 12 perturbations across both data-preserving and data-altering groups and all difficulty levels in a single pass. Each perturbed table is paired with a concise, fact-aligned table-level summary, enabling reference-less evaluation of factual consistency between tables and summaries.

Experiments used GPT-5-nano, Gemma-3 (4B/27B-Instruct), and InternVL3.5 (8B-Instruct/Thinking), with GPU-intensive work on NVIDIA-2x H100s.

Why This Matters

Impact on research. The paper reframes table evaluation as a property-driven alignment problem rather than a text-similarity problem, and it argues that metrics should satisfy permutation and format invariance, schema- and unit-consistent alignment, monotonic improvement as errors are fixed, and robustness to outliers. Its reference-less framing also challenges the assumption that a gold reference table is required for reliable evaluation. The accompanying 9,120-instance benchmark with tiered difficulty gives the community a reusable stress-test for metric robustness.

Real-world applications.

• Financial report-to-table conversion, where a misplaced column or unit mismatch can propagate costly errors and precision matters most.
• Clinical dashboards and patient data summarization, where recall of relevant facts is critical and missed discrepancies are dangerous.
• Scientific reporting and data journalism, where tables are derived from narrative sources and auditability of errors matters.
• Logistics and analytical data reformatting, where schema alignment and cell-level correctness drive downstream aggregation.

Industry relevance. Teams deploying LLM-based table generation need reward signals that can support model selection, prompt engineering, and verifiable reward modeling. TabReX's cell-level error traces and tunable sensitivity/specificity weights map onto domain-specific error tolerance — precision in finance versus recall in clinical data — and its rubric-aware outputs can feed into alignment training pipelines.

Future Directions

1. Richer structural formats. Extending TabReX to hierarchical or multi-modal tables, which the limitations section identifies as out of scope for the current implementation.
2. Lightweight, domain-adaptive evaluators. Distilling the LLM-based Text2Graph and Graph Alignment components so the metric can be deployed at scale without the computational cost and stochasticity of repeated LLM calls.
3. Beyond English and synthetic perturbations. The benchmark currently covers English only with synthetic perturbations; real-world noise, multilingual data, and complex layouts remain unexplored.
4. Tables in images and PDFs. The current implementation supports only structured digital tables (HTML, Markdown); documents requiring OCR or visual parsing are unaddressed.
5. Alternative judge back-ends. The authors explicitly invite contributions of new evaluation domains, perturbation types, and alternative matchers, including the embedding-based matcher described in Appendix C.

Target Audience

Researchers and practitioners working on LLM evaluation, table generation, and structured data-to-text systems will benefit most. The paper is also relevant to applied teams building evaluation or reward-modeling infrastructure for structured outputs, to benchmark designers interested in perturbation-based robustness testing, and to readers following the LLM-as-judge literature who want to understand where reference-less evaluation succeeds and where it still struggles. Readers should be comfortable with rank-correlation statistics and basic knowledge-graph notation.

Authors’ abstract

Evaluating the quality of tables generated by large language models (LLMs) remains an open challenge: existing metrics either flatten tables into text, ignoring structure, or rely on fixed references that limit generalization. We present TabReX, a reference-less, property-driven framework for evaluating tabular generation via graph-based reasoning. TabReX converts both source text and generated tables into canonical knowledge graphs, aligns them through an LLM-guided matching process, and computes interpretable, rubric-aware scores that quantify structural and factual fidelity. The resulting metric provides controllable trade-offs between sensitivity and specificity, yielding human-aligned judgments and cell-level error traces. To systematically asses metric robustness, we introduce TabReX-Bench, a large-scale benchmark spanning six domains and twelve planner-driven perturbation types across three difficulty tiers. Empirical results show that TabReX achieves the highest correlation with expert rankings, remains stable under harder perturbations, and enables fine-grained model-vs-prompt analysis establishing a new paradigm for trustworthy, explainable evaluation of structured generation systems.