CausalTAD: Injecting Causal Knowledge into Large Language Models for Tabular Anomaly Detection

AI summary

Overview

Research area: Tabular anomaly detection, combining large language models (LLMs) with causal discovery. Technical level: Advanced. Scope: The paper proposes CausalTAD, a method that discovers causal relationships among columns of tabular data, reorders and reweights columns accordingly, and uses that causal structure to improve LLM-based anomaly detection.

What This Paper Is About

Detecting anomalies in tables (for example, spotting fraudulent credit card transactions or fake job postings) has recently been handled well by converting tables into text and fine-tuning LLMs. The state-of-the-art method, AnoLLM, serializes columns in a random order, ignoring causal relationships between columns — which the authors argue is critical for deciding whether a record is actually anomalous. CausalTAD instead identifies which columns causally influence which others, orders the serialized columns to respect those relationships, and weights columns by how strongly they participate in the causal structure.

Key Contributions

1. A causal-driven column ordering method that reorders the columns of tabular data so that the serialized sequence aligns with causal relationships, formulated as a Linear Ordering Problem (LOP).
2. A causal-aware reweighting method that assigns different weights to different columns, so columns with stronger causal connections to the discovered factor graph have more influence on the anomaly score.
3. Empirical validation across more than 30 datasets — six mixed-type tabular datasets plus 30 numerical datasets from ODDS — showing consistent improvements over the state of the art.
4. A released implementation at https://github.com/350234/CausalTAD.

Main Findings

• Best overall performance: On the six mixed-type datasets, CausalTAD achieves average AUC-ROC of 0.834 (SmolLM-135M) and 0.833 (SmolLM-360M), compared with AnoLLM's 0.803 (SmolLM-135M) and 0.810 (SmolLM-360M), and surpasses all classical and deep-learning baselines.
• Largest gain on text-rich data: On the Fake job posts dataset with SmolLM-135M, AUC-ROC improves from 0.800 (AnoLLM) to 0.873, a relative improvement of 9.1%.
• Gains on numeric-heavy data too: On Seismic, CausalTAD (SmolLM-360M) improves from 0.746 to 0.791; with SmolLM-135M it reaches 0.783 versus AnoLLM's 0.712. On Vehicle insurance it improves from 0.555 to 0.589 (SmolLM-360M).
• AnoLLM can be unstable: On Seismic, AnoLLM (SmolLM-135M) scores 0.712, lower than the classical KNN baseline at 0.738; the authors attribute this to random column ordering disrupting causal relationships.
• Single-column special case: The 20 newsgroups dataset has only one column, so neither ordering nor reweighting takes effect; CausalTAD is identical to AnoLLM there (0.766 for both), described as a rare special case.
• Ordering matters more than the choice of causal algorithm: On Fake job posts, factor-count weighting yields 0.832 with random ordering, versus 0.870 (PC), 0.873 (LiNGAM), and 0.872 (FCI) with causality-based ordering. With the original uniform weighting, random ordering gives 0.800 versus 0.844 (PC), 0.842 (LiNGAM), and 0.843 (FCI).
• Model size matters little: On Seismic, AUC-ROC is 0.783 for SmolLM-135M, 0.791 for SmolLM-360M, and 0.785 for SmolLM-1.7B. The authors recommend a relatively smaller LLM for this task.
• Number of orderings K is not sensitive: Performance does not change significantly as the number of orderings increases; a moderate number (around 10) is suggested to limit testing time.
• Numerical benchmarks: On the 30 numerical datasets from ODDS, CausalTAD achieves the best average AUC-ROC; detailed per-dataset results and F1 scores are placed in the appendix.

Methodology in Plain English

1. Turn each table row into a sentence. Every sample is written as "column A is value, column B is value, ..." so that both structured values and free-text columns can be processed together.
2. Discover high-level factors, not raw columns. Causal relationships are not inferred directly between columns, because columns are often low-level (for example, a medicine ID) and free-text columns make column-level causality unreliable. Instead, an adaptation of the COAT framework is used: an LLM proposes high-level latent factors, each factor gets discrete values and annotation criteria, and the LLM annotates each sample's factor values to produce a factor value matrix. Each factor can involve multiple columns and each column can feed multiple factors.
3. Build a causal graph over factors. Standard causal discovery algorithms — PC, LiNGAM, and FCI — are applied to the factor value matrix to produce directed edges with weights.
4. Project factor-level causality onto columns. For each pair of columns, the absolute weights of all causal edges connecting factors that involve those columns are summed, giving a column-level preference strength. The intuition: if column i strongly influences column j, column i should appear earlier in the serialization.
5. Solve an ordering problem. Because the projected preference matrix is not guaranteed to be acyclic or transitive (a column-level cycle can appear even when the factor graph is acyclic), the method seeks the permutation that maximizes the total weight of satisfied precedence constraints — a Linear Ordering Problem.
6. Enumerate good orderings. LOP is NP-hard, but tabular datasets have few columns (typically tens). An integer programming solver (CP-SAT) finds the optimal objective value, then all permutations whose objective exceeds 0.9 × the optimum are enumerated, and the top-K are kept and averaged over to reduce noise.
7. Fine-tune the LLM. The same fine-tuning protocol as AnoLLM is used, with only the serialization changed: each training sample randomly uses one of the K causal-driven orderings, trained with the standard autoregressive cross-entropy loss.
8. Score anomalies with causal weights. For each column, the average per-token negative log-likelihood is computed under the chosen ordering, then multiplied by the column's causal contribution weight — defined as the number of factors that column maps to — and averaged across the K orderings. Columns tied to more factors influence the anomaly score more.
9. Setup details: Experiments use SmolLM-135M and SmolLM-360M as backbones, a learning rate of 5e-3, up to 18,000 training steps, and 4×NVIDIA RTX 4090 GPUs with 24GB each. GPT-4 is used for factor proposal and DeepSeek-R1-Distill-Qwen-32B for factor annotation. The training set is a random 50% of normal examples, and the test set contains the remaining normal examples plus all anomalies (uncontaminated, unsupervised setting). Missing column names get alphabetical placeholders, and missing values are mapped to "Unknown."

Why This Matters

The paper shows that how you serialize a table into text for an LLM matters, and that respecting causal structure is a concrete, reusable way to improve anomaly detection without changing the LLM architecture. It also demonstrates that the approach is largely agnostic to which causal discovery algorithm is used, making it easy to adopt.

Real-world applications mentioned or implied by the paper:

• Credit card fraud detection.
• Fake job posting detection (the dataset with the largest reported gain).
• Fraud in e-commerce transactions.
• Healthcare diagnostics and clinical-note monitoring, where free-text columns carry important signals.
• Reviewing user reviews or product descriptions for signs of fraudulent behavior or product defects.

Industry relevance: The finding that a small model (SmolLM-135M) performs comparably to or better than larger ones, combined with a recommended modest number of orderings (~10), suggests the method can be run at a practical cost. The authors position the work for "high-stakes domains such as fraud detection and healthcare monitoring," while noting that human oversight and careful deployment remain necessary.

Future Directions

• Efficiency: The authors state that their approach introduces extra computational overhead from causal discovery and multiple orderings, and list improving efficiency as future work.
• Robustness: Future work may focus on improving robustness while retaining the causal benefits.
• Reduce dependence on LLM capability: The pipeline relies on LLM quality for factor extraction and annotation, which is a stated limitation.
• Open questions raised by the results: Since the three causal discovery algorithms performed nearly identically, it is unclear which (if any) causal structure is "best" for detection; and the LOP formulation accepts solutions within 90% of optimal, leaving open how much causal fidelity is actually needed.

Target Audience

Machine learning researchers working on anomaly detection, tabular data, causal discovery, or LLM-based modeling; and practitioners in fraud detection, risk monitoring, and healthcare analytics who want a concrete way to improve LLM-based tabular anomaly detection pipelines. Familiarity with causal graphs, autoregressive language modeling, and anomaly detection evaluation (AUC-ROC) is helpful, making the paper most accessible to readers at an advanced level.

Authors’ abstract

Detecting anomalies in tabular data is critical for many real-world applications, such as credit card fraud detection. With the rapid advancements in large language models (LLMs), state-of-the-art performance in tabular anomaly detection has been achieved by converting tabular data into text and fine-tuning LLMs. However, these methods randomly order columns during conversion, without considering the causal relationships between them, which is crucial for accurately detecting anomalies. In this paper, we present CausalTaD, a method that injects causal knowledge into LLMs for tabular anomaly detection. We first identify the causal relationships between columns and reorder them to align with these causal relationships. This reordering can be modeled as a linear ordering problem. Since each column contributes differently to the causal relationships, we further propose a reweighting strategy to assign different weights to different columns to enhance this effect. Experiments across more than 30 datasets demonstrate that our method consistently outperforms the current state-of-the-art methods. The code for CausalTAD is available at https://github.com/350234/CausalTAD.