Ensembling LLM-Induced Decision Trees for Explainable and Robust Error Detection

AI summary

Overview

Research area: Data quality and tabular data cleaning, specifically error detection (ED) using large language models, with graph neural networks and statistical ensembling as supporting machinery. The paper is filed under Natural Language Processing (arXiv:2512.07246v3, cs.CL; published 2025-12-08, revised 29 Jul 2026).

Technical level: Intermediate overall, with Advanced components. The high-level idea (use an LLM to write a decision tree, then ensemble many such trees) is graspable by a general reader, but the GNN training procedure and the Expectation-Maximization consensus derivation involve graduate-level machine learning notation.

Scope (one sentence): The paper proposes TreeED and ForestED, an "LLM-as-an-inducer" framework that replaces the black-box LLM-as-a-labeler pipeline with LLM-generated decision trees containing symbolic rule nodes and GNN nodes, ensembled via an EM-based consensus, and reports an average F1-score improvement of 16.1% over the strongest baseline across seven tabular datasets.

What This Paper Is About

Error detection means finding incorrect or inconsistent cell values in a table (missing values, typos, pattern violations, outliers, and cross-attribute rule violations). The current state of the art uses an LLM as a direct labeler, which produces predictions with no traceable rationale and depends on a single stochastic inference pass. This paper asks whether an LLM can instead be used as an inducer — generating an explicit, executable decision tree that can be inspected, and then ensembling several such trees to cancel out any single tree's unreliability.

Key Contributions

1. An LLM-as-an-inducer framework for tabular error detection, consisting of two components: TreeED (a single induced tree) and ForestED (an ensemble of induced trees).
2. TreeED, which prompts an LLM to induce a decision tree combining rule nodes for simple validation checks (format, range, domain) with GNN nodes for complex relational error patterns (such as functional dependencies), enabling accurate and explainable ED.
3. ForestED, which ensembles multiple TreeED-derived predictions using an EM-based consensus procedure that treats true cell labels and per-tree reliabilities as latent variables, yielding reliable and robust results without ground-truth labels.
4. Extensive experiments across seven datasets and multiple evaluation settings, demonstrating superiority over existing baselines in detecting multiple error types, with an average F1-score improvement of 16.1% over the strongest baseline, plus substantially improved explainability and robustness.

Main Findings

• Headline result: The framework achieves an average F1-score improvement of 16.1% over the best baseline. This figure is stated in the abstract and repeated in the introduction's contribution list.

• ForestED consistently beats ZeroED on F1 across the datasets whose rows are visible in Table 3: Rayyan 0.704 vs 0.686, Hospital 0.756 vs 0.531, Flights 0.720 vs 0.585, Beers 0.947 vs 0.678, Billionaire 0.566 vs 0.455, and Movies 0.735 vs 0.698.

• Precision near ceiling in several cases: ForestED reaches 0.998 precision on Rayyan, 1.000 on Flights, and 0.974 on Beers; TreeED alone reaches 0.995 on Rayyan and 1.000 on Flights and Tax. These come with lower recall in some cases (e.g., ForestED recall 0.544 on Rayyan, 0.562 on Flights).

• TreeED is not uniformly better than ForestED: on Rayyan, TreeED's average F1 is 0.709 versus ForestED's 0.704, and on Movies TreeED is 0.583 versus ForestED's 0.735 — showing the ensemble helps most where a single induced tree is weak.

• Where single trees underperform, ensembles recover: TreeED's F1 on Billionaire is 0.358, the lowest of its rows, while ForestED raises it to 0.566. TreeED's F1 on Hospital is 0.472 versus ForestED's 0.756.

• Dataset scale included: the evaluation includes Tax with 200,000 tuples and 15 attributes (4.05% errors), versus the smallest at 1,000 tuples (Rayyan and Hospital).

• Error types covered: the seven datasets span missing values (MV), pattern violations (PV), typos (T), outliers (O), and rule violations (RV). Movies lacks RV; Tax includes all five.

• Baselines set-up: GPT-5 is used as the default LLM. Following ZeroED, two labeled samples per dataset are provided for baselines that require supervision, while all other methods are evaluated zero-shot with default hyperparameters.

• Explainability positioning (Table 1): dBoost (statistical model) and Raha (classifier) and FM_ED and ZeroED are all marked as lacking explainability; only TreeED and ForestED are marked explainable, with the LLM role listed as "Inducer" rather than "Labeler." TreeED/ForestED are also marked label-free.

• Baseline weakness on the largest table: FM_ED records an F1 of 0.006 on Tax (precision 0.027, recall 0.115) and ZeroED records 0.505, illustrating the difficulty of scaling single-pass labeling.

• Theory provided for two design choices: Lemma 1 (informativeness of uncertainty-based sampling) states that for limited data, the expected information gain from uncertainty-sampled subsets is strictly greater than from random subsets; Lemma 2 (monotonicity of EM updates) states that each E-step or M-step monotonically increases the marginal log-likelihood of the ensemble predictions.

• Not reported in the available content: the source content is truncated inside Table 3 (the ForestED Tax row is cut off after a partial precision value), so the paper's later experiments — the Section 5.4 case study details, ablation studies, robustness tests, and the explainability and time-complexity analysis placed in Appendix D — are not visible here and cannot be summarized.

Methodology in Plain English

The core reframe. Instead of asking an LLM "is this cell wrong?", the authors ask the LLM to write a small decision tree that decides whether cells are wrong. The LLM never labels data directly; it produces a structure that can be executed repeatedly.

Step 1 — Prompt construction (TreeED). The LLM receives three things: (a) data context, meaning a profile of the table (per attribute: data type, distinctness, missing ratio, inclusion dependencies, inter-column similarity, correlation scores, summary statistics) plus a handful of representative sample rows, so the whole table need not fit in the context window; (b) a decision tree specification telling the LLM what kinds of nodes to emit; and (c) output requirements restricting the answer to two components, a tree_structure and a set of labeled sample records (labels).

Step 2 — Three node types. Rule nodes run simple symbolic checks (format, range, domain) — these are LLM-written code used directly. GNN nodes handle relational errors that need global context, such as cross-attribute correlations and functional dependencies. Leaf nodes emit the final binary decision (error or clean).

Step 3 — Training the GNN nodes. Because the LLM's labels component records which branch each sampled cell takes at each node, the paper derives supervision labels for every GNN node. The table is then modeled as a bipartite graph where tuple nodes and attribute nodes form two disjoint sets and each cell is an edge carrying a hash-based embedding. Following GRAPE, the authors extend GraphSAGE with explicit edge embeddings: tuple nodes start from a constant vector, attribute nodes from a one-hot vector, messages are formed by concatenating neighbor embeddings with edge embeddings, aggregation is a mean aggregator, and after two propagation layers the concatenated tuple and attribute embeddings go into a two-layer MLP that predicts the branch. Training uses binary cross-entropy. A GNN node's internal logic is not interpretable, but the paper argues its role is path-level explainable: each node carries an LLM-induced descriptive message about its intended function and names the attributes involved.

Step 4 — Prediction. Each cell traverses a root-to-leaf path; internal nodes apply rule or GNN checks, and the reached leaf gives the error label. The resulting path is the explanation.

Step 5 — ForestED sampling. Since labeling every cell is too expensive, the method selects informative rows using Gaussian Process uncertainty sampling: standardize numeric attributes, one-hot encode categorical ones, apply PCA for dimensionality reduction, fit a GP with an RBF kernel (length-scale χ) on a small dummy-target subset, and take the top-s tuples by predictive variance, where s = min(100, ⌈ρN⌉) and ρ is the sampling ratio. Those rows are split into R disjoint partitions, each inducing its own tree.

Step 6 — EM consensus. Each tree gets a 2×2 latent confusion matrix expressing how likely it is to output ŷ when the truth is y, plus a prior over error labels. The first E-step initializes posterior responsibilities from normalized TreeED vote counts. Then E-step and M-step alternate: the E-step computes the posterior over true labels via Bayes' rule, the M-step re-estimates each tree's confusion matrix and the error prior by maximum likelihood from those posteriors, and the final label is the MAP choice after convergence. The claimed benefit over majority voting is that unreliable trees get down-weighted automatically without any ground-truth labels.

Why This Matters

Impact on research. The paper reframes what an LLM contributes to a data-quality pipeline: not answers, but artifacts. That shift composes two streams that have mostly run in parallel — LLM-based data cleaning and neuro-symbolic learning (rule nodes plus learned GNN nodes inside one tree) — and it provides a label-free ensembling mechanism (EM over latent tree reliabilities) that could be lifted into other settings where several LLM-generated detectors disagree and no gold labels exist. Providing code at github.com/T-Lab/ForestED lowers the barrier to replication.

Real-world applications:

• Systematic review and study screening — the Rayyan dataset is drawn from reference-screening data, where mislabeled metadata can distort a review's evidence base.
• Healthcare records — the Hospital dataset covers MV, PV, T, O, and RV, where a wrong or inconsistent field can affect clinical or administrative decisions.
• Aviation and logistics operations — the Flights dataset (34.51% errors, the highest error rate evaluated) reflects the kind of inflated, inconsistent operational records that downstream analytics depend on.
• Tax and public-sector administration — the Tax dataset, at 200,000 tuples, tests whether the approach survives the scale that real government and enterprise records reach.

Industry relevance. Data cleaning is a recurring cost for any organization preparing tabular data for analytics or machine learning, and the explainability gap is a practical blocker: reviewers and domain experts need to see why a cell was flagged. A path through symbolic checks and named dependency checks is auditable in a way that a raw LLM verdict is not. The zero-shot plus two-labeled-samples setup also matters commercially, because it means the method does not assume a large annotated error corpus exists.

Future Directions

• From detection to repair. The paper frames ED as the first phase of cleaning (with data repair as the second) but does not implement or evaluate repair. The induced trees identify which checks a cell fails, which is arguably most of what a repair step needs.
• Beyond binary labels. The framework outputs only error or clean. It does not report which of the five error types (MV, PV, T, O, RV) a flagged cell exhibits, even though the datasets are annotated with those types and leaf nodes are described as outputting "final decision types."
• Closing the GNN interpretability gap. The authors acknowledge in Remark 1 that GNN node internals remain black-box and defend only their path-level role. Whether a fully faithful explanation requires interpretable relational nodes is left open.
• Cost, scale, and faithfulness questions not answered in the visible content. The truncated content does not report the token cost or latency of R multi-pass tree inductions, how performance varies with the number of trees R, the sampling ratio ρ, or the kernel length-scale χ, nor whether independent replications of the framework produce the same induced tree structures. These are the natural next experiments.
• Generalization of the theory. Lemma 1 covers uncertainty sampling under limited data and Lemma 2 guarantees only monotonic ELBO improvement for EM. Whether EM's consensus converges to the true labels rather than a self-consistent fixed point under adversarial or systematically biased trees remains an open question the reported lemmas do not settle.

Target Audience

Researchers and practitioners working on data quality, data cleaning, and tabular machine learning, particularly those already using LLMs for data-centric tasks and frustrated by their opacity or instability. It also suits engineers building production data-validation pipelines who need auditable decisions, and machine learning researchers interested in neuro-symbolic hybrids or in ensembling LLM-generated artifacts without gold labels. Readers should be comfortable with decision trees, graph neural networks, and the EM algorithm; those without that background can still follow the motivation, contributions, and headline results.

Authors’ abstract

Error detection (ED), which aims to identify incorrect or inconsistent cell values in tabular data, is important for ensuring data quality. Recent state-of-the-art ED methods leverage the pre-trained knowledge and semantic capability of large language models (LLMs) to directly label whether a cell is erroneous. However, this LLM-as-a-labeler pipeline produces predictions through an implicit black-box process with limited traceability and explicit justification, and relies on stochastic single-pass inference, resulting in inconsistent and insufficiently robust detections across contexts. To address these limitations, we propose an LLM-as-an-inducer framework that uses an LLM to induce a decision tree for ED, termed TreeED, and ensembles multiple such trees for consensus detection, termed ForestED. Based on prompts derived from data context, decision tree specifications, and output requirements, TreeED queries the LLM to induce a decision tree skeleton whose root-to-leaf paths specify the stepwise procedure for evaluating a sample. Each tree contains three types of nodes: (1) rule nodes that perform simple validation checks, such as format or range constraints; (2) Graph Neural Network (GNN) nodes that capture complex patterns, such as functional dependencies; and (3) leaf nodes that output the final decision as error or clean. ForestED employs uncertainty sampling to obtain multiple informative row subsets and constructs a decision tree for each subset using TreeED. It then applies an Expectation-Maximization-based algorithm to jointly estimate tree reliability and optimize the consensus ED prediction. Experiments demonstrate that our methods are accurate, explainable, and robust, achieving an average F1-score improvement of 16.1% over the best baseline.