Train a Unified Multimodal Data Quality Classifier with Synthetic Data

AI summary

Overview

Research area: Multimodal machine learning — specifically data curation and filtering for pre-training Multimodal Large Language Models (MLLMs), published under arXiv category Computer Vision.

Technical level: Advanced. The paper assumes familiarity with vision-language model architectures (vision encoders, visual projectors, LLM backbones), pre-training/SFT pipelines, and multimodal benchmarks.

Scope (one sentence): The paper introduces UniFilter, a single efficient MLLM-based classifier that scores the quality of both image-text caption data and image-text interleaved document data, trained on semi-synthetic sample-score pairs generated with a proprietary MLLM.

What This Paper Is About

MLLMs are pre-trained on a mixture of short image-text captions and long interleaved image-text documents, but the current dominant filter, CLIPScore, can only judge a single image against a short caption and has never been shown to work on long interleaved documents containing many images and paragraphs. Because large-scale pre-training typically covers less than one epoch of the available web data, data quality is described as the major bottleneck, so the authors set out to build one unified quality classifier that handles both data formats. To train it, they generate their own labeled data by pairing real web images with synthetic text written at four controlled quality levels, avoiding the cost and inconsistency of human annotation.

Key Contributions

1. UniFilter, described as the first unified approach for filtering both image-text caption data and image-text interleaved document data, using an MLLM architecture instead of a CLIP-based one so it can assess complex multimodal structures rather than only single image-text pairs.
2. A semi-synthetic data generation method that combines real images sampled from DataComp and OBELICS with synthetic text generated at four quality levels by Claude-3-Sonnet, enabling scalable and cost-effective creation of labeled sample-score pairs for classifier training.
3. Demonstrated gains from filtered pre-training: MLLMs pre-trained on UniFilter-curated caption and interleaved data outperform models trained on baseline-filtered data, with the gains persisting after visual supervised fine-tuning.
4. Public release of the synthetic training data, UniFilter model checkpoints, and the high-quality interleaved document subset OBELICS-HQ, along with a project website and code repository.

Main Findings

• UniFilter beats caption-filtering baselines on zero-shot VQA. Pre-training MLLMs for 5B tokens on UniFilter-curated caption data from the DataComp-medium-128M pool yields an average score of 31.3 across GQA, VQA-v2, VizWiz, OKVQA and TextVQA, versus 28.7 for DFN, 30.2 for MLMFilter-Image-Text-Matching, 29.4 for MLMFilter-Object-Detail-Fulfillment, 30.4 for MLMFilter-Caption-Text-Quality, and 27.4 for MLMFilter-Semantic-Understanding. Per-dataset, UniFilter scores 29.6 on GQA, 43.2 on VQA-v2, 22.9 on VizWiz, 28.2 on OKVQA and 32.5 on TextVQA.
• One exception is TextVQA. The MLLM trained on UniFilter-curated data lags the MLMFilter-CTQ metric on TextVQA (32.5 versus 35.2), and the authors suggest CTQ may be the best metric for OCR- or text-rendering-related pre-training data.
• Interleaved-document filtering improves in-context learning. Training on 10B tokens (a fixed 5B of UniFilter-curated caption tokens plus 5B of interleaved tokens at a 1:1 ratio validated in MM1), UniFilter achieves +0.7 and +2.8 average accuracy improvements over the DFN baseline at 4-shot and 8-shot, and a +3.2 average VQA improvement at 0-shot. Its 8-shot average is 50.2 versus 47.4 for DFN and 47.4 for no filtering; 4-shot is 46.2 versus 45.5 and 43.9.
• No-filtering baselines trail badly. Without filtering, the 10B-token model scores 21.1 / 43.9 / 47.4 at 0/4/8 shots, compared with 28.4 / 46.2 / 50.2 for UniFilter. The paper also lists MM1-3B and MM1-7B (400B training tokens) and BLIP-3 (100B tokens) as references trained on 10 to 40 times more data than this work.
• Gains survive supervised fine-tuning. After visual SFT on a joint 575k multimodal instruction set from LLaVA-1.5 and ShareGPT4V, the UniFilter-pre-trained MLLM reaches 38.1 average VQA accuracy, 83.2 POPE, 42.0 MMMU (Val), 77.0 MMBench (Dev) and 38.5 MMStar, surpassing the best baseline by +3.1 average VQA accuracy, +1.5 MMMU accuracy and +1.6 MMBench accuracy. The SFT-only No-Pretrain baseline reaches 30.7 average VQA, lagging all pre-trained fine-tuned models.
• Architecture ablation selected a small backbone. Across the ablated MLLM configurations, SigLIP-ViT-SO400m-384px as vision encoder, Adaptive Average Pooling plus MLP (144 tokens per image) as projector, and Qwen-2.5-0.5B as LLM backbone gave the best trade-off between quality-classification performance and efficiency and became the final UniFilter. The Qwen2.5-1.5b variant scored highest on validation (95.2 accuracy, 94.3 F1) but added roughly 1B parameters over the 0.5B model (94.8 accuracy, 93.8 F1).
• Efficiency is on par with CLIP-based filtering. UniFilter reaches a throughput of 130 samples/s with the Qwen-2.5-0.5b backbone, slightly above the 128 samples/s measured for CLIPScore on the same hardware, and is described as achieving comparable inference speed to DFN-CLIP-Large.
• On the DataComp-Medium benchmark, results are comparable to DFN, and union helps. UniFilter-30% averages 34.3 over the 38-dataset suite versus 34.5 for DFN-15%, and the union of UniFilter-25% and DFN-15% (26.9M samples) achieves the best reported 38-dataset average of 35.0 and VTAB average of 36.3. UniFilter-30% achieves the best retrieval average of 27.9, and adding UniFilter data to DFN's improves DFN's retrieval performance by +1.8.
• UniFilter wins on 19 of 38 datasets but trails DFN on image-classification-style sets. It lags on ImageNet-related datasets because DFN is continually fine-tuned on ImageNet-1k, and notably on Oxford-Pet (37.99% versus 52.52%), where CLIPScore scores 35.25% and MLM-Filter 38.62%. The authors hypothesize DFN's advantage comes from coverage of fine-grained pet class names in Apple's internal HQ-357M dataset.
• UniFilter selects longer, richer interleaved documents. Curated documents average 3.15 images, 1627.8 text length and 2078.3 document length at a 15% filtering fraction, compared with 1.98 / 842.5 / 1125.8 for no filtering (100%) and 1.88 / 841.1 / 1110.4 for DFN (90.5%).

Methodology in Plain English

The researchers first define four quality levels for multimodal data — easy negative, medium negative, hard negative and positive — mapped to integer scores 0, 1, 2 and 3, mirroring the four-level scheme used in FineWeb-Edu-Quality-Classifier. They write separate prompt templates describing each level for captions and for interleaved documents.

Rather than generating images from scratch, they keep real web images and generate only the text. Fully synthetic images were rejected because image generators such as Midjourney and Dalle-3 were found to get stuck in particular styles such as cartoon. Instead, they sample source images: DataComp-small images are clustered into 10k clusters using CLIP ViT-L/14 image embeddings and 4 images are drawn from each cluster to produce 40k images; for OBELICS, image embeddings within each document are average-pooled into a document-level embedding, documents are clustered into 10k clusters, and 40k interleaved documents are sampled. Claude-3-Sonnet then writes text at the specified quality levels, producing 40k synthetic caption samples and 40k synthetic interleaved document samples (80k total), with 5% held out for validation. Llama-guard-3-8B scans the synthetic text for safety, and 4k non-synthetic high-quality caption samples from MSCOCO and Flickr are added with a positive label.

The classifier itself is an MLLM. Images go through a vision encoder and text through a word embedding layer; for interleaved documents the encoded pieces are reordered into the original interleaving sequence, while for captions the image encoding and caption embeddings are simply concatenated. A trainable one-dimensional classification head replaces the language modeling head and outputs a scalar logit, trained against the synthetic quality labels with Mean-Square-Error loss. The authors ablate six combinations of vision encoder, projector and LLM backbone, train each for 10 epochs, and pick the best based on validation accuracy, favoring the smallest configuration that preserves performance.

For pre-training experiments, the same MLLM architecture (SigLIP-so400m encoder, AvgPool projector at 144 tokens per image, Phi-3-mini-3.8b LLM) is used with a frozen vision encoder. On captions, every method retains a fixed 30% of the DataComp-medium-128M pool and trains for 5B tokens. On interleaved data, UniFilter selects the top-15% of documents by quality score, DFN is adapted by discarding images with less than 0.15 cosine similarity to any text paragraph, and a no-filtering baseline is included. Instruction tuning uses 575k multimodal instructions drawn from LLaVA-Conversations, LLaVA-Reasoning, ShareGPT4V-Caption, OCRVQA, A-OKVQA, TextCaps, RefCOCO, VG and ShareGPT.

Why This Matters

Impact on research. The paper argues that data quality, not data quantity, is the current bottleneck for MLLMs, and that existing filter families such as CLIPScore, DFN and MLMFilter cannot handle document-level interleaved data at all. By showing that one small MLLM can score both formats, it opens a path to treating data curation as a unified classification problem, and it releases the training data, checkpoints and the OBELICS-HQ subset so the result can be reproduced and extended.

Real-world applications:

• Curating large web-crawled corpora before expensive pre-training runs, so limited GPU budgets are spent on higher-quality data.
• Building and maintaining multimodal document datasets for retrieval-style products where long documents with many images must be scored as a whole.
• Providing a quality-scoring component for teams that assemble their own image-text training collections and need a cheap, high-throughput classifier.
• Selecting educational-quality interleaved content, since the quality prompts are explicitly framed around educational value for textbook-style material.

Industry relevance. The throughput result matters commercially: UniFilter runs at 130 samples/s versus 128 samples/s for CLIPScore on the same hardware, so a quality improvement over CLIP-based filtering does not come with a proportional inference cost increase at pre-training scale. The authors are affiliated with UC Santa Barbara, Amazon Stores Foundational AI and UC San Diego, and the work is positioned directly against production-grade filters such as DFN.

Future Directions

• Closing the image-classification gap. UniFilter trails DFN on ImageNet-related datasets and substantially on Oxford-Pet (37.99% versus 52.52%), which the authors attribute to DFN's exposure to Apple's internal HQ-357M dataset. How to recover fine-grained class coverage without proprietary data is left open.
• Beyond the caption/interleaved pair. MLMFilter is explicitly noted as unusable for interleaved data, and the paper's own framing suggests the unified design could extend to further multimodal formats, but no such extensions are reported.
• Filtering-fraction and mixing policy. The paper fixes a 30% retention fraction for caption comparisons, reports an ablation on the filtering fraction in its appendix, and uses a 1:1 caption-to-interleaved token ratio validated by MM1; whether other ratios or adaptive fractions help is not settled here.
• Better text-centric pre-training filters. The single dataset where UniFilter loses to a baseline is TextVQA, where MLMFilter-CTQ wins, hinting that a dedicated OCR/text-quality signal could complement the unified classifier.

Target Audience

Researchers and engineers working on multimodal pre-training data pipelines, MLLM training at scale, or model-based data filtering benchmarks such as DataComp. It also suits practitioners who need a practical, efficient quality-scoring tool for image-text corpora and want to reuse the released checkpoints and synthetic training data. Readers without background in vision-language architectures or pre-training evaluation will find the ablation and benchmark sections demanding.

Authors’ abstract

The Multimodal Large Language Models (MLLMs) are continually pre-trained on a mixture of image-text caption data and interleaved document data, while the high-quality data filtering towards image-text interleaved document data is under-explored. We propose to train an efficient MLLM as a Unified Mulitmodal Data Quality Classifier to Filter both high-quality image-text caption and interleaved data (UniFilter). To address the challenge of collecting diverse labeled multimodal data, we introduce a semi-synthetic approach that leverages readily available raw images and generates corresponding text across four quality levels. This method enables efficient creation of sample-score pairs for both caption and interleaved document data to train UniFilter. We apply UniFilter to curate high-quality caption data from DataComp caption dataset and interleaved data from the OBELICS image-text interleaved dataset. MLLMs pre-trained on the filtered data demonstrate significantly enhanced capabilities compared to those trained on baseline-filtered data, achieving stronger zero-shot reasoning and in-context learning capabilities. After visual supervised fine-tuning, these UniFilter-induced MLLMs achieve stronger performance on various benchmarks, highlighting the downstream benefits of high-quality multimodal pre-training. We release the synthetic training data used for training UniFilter, the UniFilter model checkpoints, and the high-quality interleaved document subset OBELICS-HQ, curated by UniFilter, to the community for reproduction and further development.