M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset

AI summary

Overview

Research area: Natural Language Processing, specifically multimodal automated fact-checking (AFC) and dataset construction for vision-language models.

Technical level: Intermediate. The paper is accessible to anyone familiar with machine learning evaluation, though the six-task pipeline and metric choices assume some background in NLP benchmarking.

Scope: The paper introduces M4FC, a real-world dataset of 4,982 fact-checked images and 6,980 claims spanning 10 languages and six multimodal fact-checking tasks, and benchmarks current multimodal LLMs against it.

What This Paper Is About

About 80% of online misinformation verified by fact-checkers since 2020 combines text with images, video, or audio, yet existing datasets for automated fact-checking are small, limited to one or two languages, or cover only a single task (usually the final true/false verdict). The authors build a dataset drawn from real fact-checking articles worldwide so that models can be evaluated on the full verification workflow, not just its endpoint. The goal is to expose where current multimodal models fail once misinformation is realistic, multilingual, and culturally varied.

Key Contributions

1. M4FC dataset. 4,982 images and 6,980 claims sourced from 22 fact-checking organizations across 17 countries, covering 10 languages and images from 159 countries on six continents. It is the first real-world multimodal AFC dataset to combine this breadth of languages, geography, and tasks.

2. Two new AFC tasks. Visual claim extraction turns a screenshot of a social media post into an explicit, verifiable claim, and location verification checks whether a candidate location matches an image using maps and satellite imagery as evidence.

3. A six-task pipeline. Alongside the two new tasks, the dataset labels claimant intent prediction, fake image detection, image contextualization, and verdict prediction, including evidence sets filtered to prevent ground-truth leakage.

4. Comprehensive baseline evaluation. Five multimodal LLMs and two state-of-the-art AFC systems are benchmarked on all tasks, including an analysis of how predicted versus ground-truth intermediate task outputs affect final verdict accuracy.

Main Findings

• Visual claim extraction is mostly within reach. Gemini-1.5-Flash leads across all metrics. In a manual review of 50 of its predictions, 38 claims were accurate, 9 were partially correct but missed external world knowledge, and 3 failed because the model did not recognize sarcasm.

• Claimant intent is far from solved. All models score low. GPT4o-mini underperforms mainly because it refuses the task outright, treating intent prediction as a policy violation.

• Fake image detection transfers well. A pre-trained deepfake detector reached 70.31% F1 while a general AI-image detector managed only 6.41%. Fine-tuning on M4FC's small training set produced large gains across all detectors, even though the train and test splits do not overlap in time.

• Image contextualization varies sharply by attribute. Qwen2.5-VL-7B led on date (30.3%), location (45.0%), motivation (16.8%), and event (16.2%), while Gemini-1.5-Flash led on people (45.0%) and things (22.7%). The dedicated COVE system was outperformed by nearly every general-purpose MLLM. Predicting dates remains harder than predicting locations.

• Models struggle to match images to aerial views. In location verification, F1 often dropped when models were given satellite or map views instead of a text location name, suggesting they can name a place but cannot confirm it visually. Gemini-1.5-Flash led consistently; Llama-3.2-11B performed worse than random when given the location as text.

• Cross-lingual generalization is model-dependent. Proprietary models held roughly even performance across languages, while open-weight models dropped sharply on multilingual claims, with Llama-3.2-11B declining the most. Qwen2.5-VL-7B was the exception, scoring slightly higher in multilingual settings. No model exceeded 80% F1 on verdict prediction.

• Evidence retrieval matters most. Reverse image search evidence gave the largest consistent boost to verdict prediction, and the DEFAME framework's additional tools (web search, geolocation) added little beyond it. Using predicted visual claim extraction instead of ground-truth claims measurably hurt final verdict accuracy.

• Intermediate tasks are complementary when accurate. Combining all ground-truth intermediate outputs with web evidence produced the best verdict F1 (78.26% for GPT4o-mini). Ground-truth image contextualization alone added 16.96 F1 points, more than any other single task. Notably, DEFAME, an end-to-end system, still outperformed the pipeline built on predicted intermediate outputs.

Methodology in Plain English

The authors gathered fact-checking articles published by members of the International Fact-Checking Network, spanning outlets in India, Jordan, Kenya, the United States, Portugal, Germany, Peru, and elsewhere. Rather than hand-labeling everything, they used GPT4o to extract structured fields from the articles—the claim, its language, the claimed date and location, the actual context of the image, and the final verdict—following a template of prompts.

Because fact-checkers mostly write about false claims, the dataset is naturally skewed. To balance it, the authors used GPT4o to generate plausible true captions by recombining the verified context of false claims (date, location, people, event) and swapped these in for half the false cases, producing a "balanced" evaluation setting.

Labels were validated in two stages. First, paid annotators on Prolific reviewed sampled articles, with 33 of 79 excluded for failing attention checks. Second, the authors manually corrected the entire test set and found only 0.7% of GPT4o labels were (partially) wrong. To reduce the risk that models simply memorized the answers, the dataset uses a temporal train/dev/test split so the test claims are the most recent.

For location verification, the authors semi-automatically identified articles that referenced maps, satellites, or landmarks, then had GPT4o propose likely-but-incorrect locations, giving each image one correct and two incorrect candidates. Evidence for other tasks was collected via Google reverse image search, filtered to exclude anything published by fact-checking organizations or dated after the claim.

Why This Matters

Impact on research. M4FC shifts multimodal fact-checking evaluation away from single-task verdict classification toward the multi-step workflow humans actually perform. Its multilingual and multicultural coverage makes it possible to measure whether a model's fact-checking ability holds outside English, and the finding that models cannot reliably match ground images to satellite views identifies a concrete capability gap for future work.

Real-world applications.

• Newsroom support: giving human fact-checkers a pipeline that flags likely out-of-context or manipulated images, extracts the underlying claim, and surfaces evidence for review.
• Platform content moderation: triaging user-uploaded images that misrepresent their origin, in languages like Tamil, Telugu, or Arabic that commercial systems often handle poorly.
• OSINT and human rights investigations: verifying the location and date of footage circulating from conflict or crisis zones.
• Search and provenance tooling: detecting whether an image has appeared before and under what circumstances.

Industry relevance. The results are directly relevant to companies building trust-and-safety systems, news verification products, and multilingual AI assistants. The finding that fine-tuning on a small real-world dataset transfers to newer, unseen cases is encouraging for organizations that cannot afford to train at synthetic scale. The result that an end-to-end system like DEFAME still beats a pipeline with predicted intermediate steps also suggests that chaining models naively is not yet the answer.

Future Directions

• Extending to video. Videos are now the most common misinformation format verified by fact-checkers, but the authors excluded them because the source videos are often deleted and fact-checking articles preserve only screenshots.
• Scaling location verification. The task has only 195 instances, limited by the rarity of images with precise, mappable coordinates and outdoor scenes. The authors suggest building larger synthetic datasets from outdoor news imagery with known coordinates.
• Improving evidence retrieval for context items. Predicted image contextualization performed worse than using retrieved web evidence directly, showing that current retrieval is insufficient for many context attributes and pointing to a need for better context-aware search.
• Designing better pipelines. Combining intermediate tasks with ground-truth labels delivered the largest accuracy gains, but the tradeoffs between accuracy, interpretability, and compute cost remain open.
• Continuous benchmark refreshing. Even with temporal splits and evidence filtering, pretraining leakage cannot be fully ruled out; the authors note follow-up work like VERITAS that continuously updates benchmarks with recent claims.

Target Audience

This paper is most useful to NLP and computer vision researchers working on fact-checking, misinformation detection, or multimodal LLM evaluation, especially those focused on multilingual or low-resource settings. It also serves data scientists and engineers at social media platforms, news organizations, and OSINT tooling companies who need realistic benchmarks for image verification systems, as well as researchers studying cross-view geolocalization or image provenance.

Authors’ abstract

Existing real-world datasets for multimodal fact-checking have multiple limitations: they contain few instances, cover only one or two languages, focus on a single task, or rely on external news article sets to source true claims. To address these shortcomings, we introduce M4FC, a new real-world dataset comprising 4,982 images paired with 6,980 claims. The images, verified by professional fact-checkers from 22 organizations, represent a diverse range of cultural and geographic contexts. Each claim is available in one or two out of ten languages. M4FC spans six multimodal fact-checking tasks: visual claim extraction, claimant intent prediction, fake image detection, image contextualization, location verification, and verdict prediction. We provide baseline results for all tasks and analyze how combining intermediate tasks affects verdict prediction performance. We make our dataset and code publicly available.