Towards Cross-Modal Error Detection with Tables and Images

AI summary

Overview

Research area: data-centric AI and data quality, specifically cross-modal error detection across tabular data and images.

Technical level: Intermediate. The paper assumes familiarity with data cleaning concepts, AutoML, and vision-language models, but its central problem statement is accessible to anyone who works with data pipelines.

Scope: The paper introduces the problem of detecting errors in tabular data by jointly reasoning over the table and its associated images, and presents a preliminary benchmark of five existing methods across four e-Commerce datasets.

What This Paper Is About

Organizations that store product, travel, or health records often have both a table and an image for the same item. An error can appear that is invisible in either modality alone, because the table looks internally consistent and the image looks normal, yet the two contradict each other. The authors formalize this as cross-modal error detection and test whether existing error detection, label error detection, and vision-language methods can actually find such errors.

Key Contributions

1. The paper motivates and formally introduces the problem of cross-modal error detection in tabular data, defining an erroneous tuple as one where at least one cell value differs from its unknown true value, and assuming no labeled erroneous examples are available for training (as in novelty detection).

2. It presents a preliminary benchmark of four e-Commerce datasets (Fashion, Baby, Sports, and Fashion 44K) and five baseline approaches (Raha, AutoGluon + Cleanlab, AutoGluon + DataScope, LLaVA, and LEMoN) evaluated under three modality settings: table only, image only, and table plus image.

3. It shows that label error detection methods paired with a strong AutoML framework, specifically AutoGluon + Cleanlab and AutoGluon + DataScope, achieve the highest F1 scores, and that in most cases using both table and image data is what uncovers cross-modal errors.

4. It releases the benchmark data and code at https://github.com/OlgaOvcharenko/find_errors.

Main Findings

• Tabular-only methods perform poorly. Raha, a state-of-the-art single-column error detector, scores F1 of 0.09 on Fashion, 0.20 on Baby, 0.10 on Sports, and 0.34 on Fashion 44K when given tabular data only.

• Images help substantially. The F1 scores of AutoGluon + Cleanlab and AutoGluon + DataScope in the image-only setup are significantly higher than in the table-only setup (for example, AutoGluon + Cleanlab goes from 0.49 to 0.81 on Fashion, and AutoGluon + DataScope goes from 0.50 to 0.89 on Fashion).

• Combining modalities is usually best. In three out of four datasets, joint access to both modalities produces the best performance, with an improvement of up to 5% in F1 score compared to the image-only setup. On Fashion, however, AutoGluon + DataScope degrades when images are combined with tabular data (F1 0.89 image-only versus 0.83 with both).

• Best overall performers. AutoGluon + DataScope and AutoGluon + Cleanlab with access to both image and tabular data achieve the highest F1 scores. AutoGluon + DataScope reaches F1 of 0.83 on Fashion, 0.89 on Baby, 0.73 on Sports, and 0.58 on Fashion 44K. AutoGluon + Cleanlab reaches 0.83, 0.70, 0.66, and 0.75 respectively.

• Remaining error gaps. AutoGluon + Cleanlab only reaches F1 scores of around 70% to 80%, missing 22% to 33% of errors according to its recall scores. AutoGluon + DataScope performs subpar on the larger Fashion 44K dataset, missing up to half of the errors as indicated by its recall.

• DataScope has practical weaknesses. Its performance relies heavily on the quality of AutoGluon's input embeddings, with F1 scores decreasing by 10% when using insufficiently trained representations. It also requires a large clean validation set, which may not exist in real-world settings.

• Vision-language models are unreliable here. LLaVA in zero-shot and few-shot modes produces low F1 scores and in several cases marks every input tuple as erroneous (the paper shows these cases in brackets). This is surprising because LLaVA was used to generate the tabular data for the Baby and Sports datasets.

• LEMoN underperforms. With random noise at level 0.4, LEMoN scores F1 of 0.52 on Fashion, 0.48 on Baby, 0.42 on Sports, and 0.44 on Fashion 44K when given both modalities.

• High-cardinality, skewed columns are hard. Errors are easiest to detect in columns with few distinct values and balanced frequencies, such as Fashion's Category (2 distinct values, F1 up to 1.00 with both modalities). Errors are hardest to detect in columns with many distinct values and a skewed distribution, such as Baby's ProductType (132 distinct values, F1 up to 0.88 for DataScope and 0.51 for Cleanlab) and Sports' SportType (65 distinct values).

• Repair is also possible but limited. Using AutoGluon + Cleanlab for error correction, joint table and image access improves accuracy, for example Fashion SubCategory from 0.07 (table) and 0.66 (image) to 0.94 (both), while difficult columns such as Baby's Color stay low (0.21 with both).

• Concrete cross-modal examples. The paper shows cases such as baby wipes labeled with the wrong product type and category, and a camping chair incorrectly marked as a volleyball net. A motivating real-world example is a LEGO toy whose tabular data states it is suitable for four-month-old babies while the image indicates it is meant for 16+ years old teenagers, a potential choking hazard.

Methodology in Plain English

The authors collected four e-Commerce datasets containing both tabular attributes (title, category, type, color, and dataset-specific columns) and one image per row. Fashion and Fashion 44K came from Kaggle as multi-modal datasets. Baby and Sports originally contained only images, so the authors used the vision-language model LLaVA 1.5-7b to generate tabular attributes, then manually post-processed and refined the results into ground truth.

Because real cross-modal errors are rare and unlabeled, they injected synthetic errors into the test splits only. They first cleaned the data manually and with Cleanlab, then randomly selected 50% of the rows in each test split, picked a random column per sampled row, and replaced that cell with a random different existing value from the same column. They also modified any other cell in the row that contained the original value (for example, changing the color name in the product title) so the correct answer could not be recovered from the table and had to be detected from the image. For correlated columns, they only substituted pairs that had already been observed.

They then ran five methods under three modality configurations and measured precision, recall, and F1. AutoGluon models were trained with each column as a target, and their outputs fed either into Cleanlab (confident learning for label error detection) or into DataScope (Data Shapley values computed exactly for a kNN proxy model with k = 1, scoring dirty test data against clean training data, with negative importance treated as erroneous). LLaVA was prompted zero-shot and few-shot, and LEMoN was given string-serialized rows as textual labels.

Why This Matters

Research impact: The paper opens a new problem area at the intersection of data management, data-centric AI, and multi-modal machine learning, and provides an initial open benchmark. It shows that existing single-modality error detection, label error detection, and vision-language approaches all fall short, providing a starting point for dedicated cross-modal methods.

Real-world applications:

• E-Commerce catalogs, where product metadata and product images must agree, as in the age rating, color, resolution, and content-rating errors the authors found on Amazon.
• Healthcare, where tabular records and medical images or documents co-exist and inconsistency can affect patient safety.
• Travel and real estate platforms, where listings combine structured attributes with photos.
• Autonomous vehicles and other systems that combine tabular, text, image, and video data.

Industry relevance: The authors interviewed the content quality team of a large e-Commerce platform whose catalog contains tens of millions of products with thousands of distinct attributes, sourced from several thousand external sellers and third-party data providers. That team builds custom LLM-based solutions per attribute, which are expensive and hard to scale. Undetected cross-modal errors also carry legal and regulatory risk, such as incorrect allergen, age restriction, or chemical content specifications, so better cross-modal validation supports consumer protection and compliance.

Future Directions

• Designing a dedicated multi-modal model for cross-modal error detection, possibly based on self-supervised contrastive learning, rather than adapting single-modality or label-error methods.
• Building a larger, more comprehensive benchmark that includes real-world data and cross-modal errors from domains beyond e-Commerce.
• Evaluating a broader set of baselines, including tabular foundation models such as TabPFN and CARTE, as well as novelty and anomaly detection methods.
• Resolving the open question of how best to combine modalities, since DataScope's multi-modal performance is inconsistent and it remains unclear when adding tabular data to images helps versus hurts.
• Reducing cost and scaling: label error detection still requires training a separate AutoML model for each column in each dataset, and vision-language approaches require multiple expensive calls per sample.

Target Audience

Data quality and data engineering practitioners who manage multi-modal datasets; researchers in data-centric AI, data cleaning, and data management; machine learning engineers working with AutoML, label error detection, or vision-language models; and product teams at e-Commerce, healthcare, travel, or real estate organizations who need to validate consistency between structured attributes and images at scale.

Authors’ abstract

Ensuring data quality at scale remains a persistent challenge for large organizations. Despite recent advances, maintaining accurate and consistent data is still complex, especially when dealing with multiple data modalities. Traditional error detection and correction methods tend to focus on a single modality, typically a table, and often miss cross-modal errors that are common in domains like e-Commerce and healthcare, where image, tabular, and text data co-exist. To address this gap, we take an initial step towards cross-modal error detection in tabular data, by benchmarking several methods. Our evaluation spans four datasets and five baseline approaches. Among them, Cleanlab, a label error detection framework, and DataScope, a data valuation method, perform the best when paired with a strong AutoML framework, achieving the highest F1 scores. Our findings indicate that current methods remain limited, particularly when applied to heavy-tailed real-world data, motivating further research in this area.