Media Intelligence Platform: Multimodal Detection of Synthetic and Manipulated Content

AI summary

Overview

Research area: Computer vision and multimedia forensics — specifically the detection of synthetic and manipulated images, video, audio, and text.

Technical level: Intermediate. The paper is a systems/integration paper: it assumes familiarity with model families (CNNs, ViTs, WavLM, XLM-R) and benchmark metrics (F1, AUC, IoU, EER), but the architecture and results are described in accessible terms.

Scope: The paper presents the Media Intelligence Platform (MIP), a single web-based framework that combines six specialized forensic modules — AI-generated image detection, image forgery localization, face morphing detection, deepfake video detection, audio deepfake detection, and text analysis — behind one standardized REST interface, and reports benchmark performance for each.

What This Paper Is About

Generative AI now produces synthetic faces, forged videos, AI-generated images, fake audio, and locally edited photos that are hard to distinguish from authentic media, but most existing detectors are built for one manipulation type or one modality, which makes them hard to use in real investigations that involve mixed media. The authors build a unified, container-based platform where independently developed detection models of very different kinds sit behind a common interface and return results in a common format. The goal is to reduce tool fragmentation and give forensic analysts, journalists, and trust-and-safety teams one place to analyze heterogeneous media.

Key Contributions

1. A unified multimodal forensic platform (MIP). A modular microservice architecture where each analysis task is a "Module" and each neural network behind it is a "Model," all exposed through a standardized REST API returning a shared ModuleResult schema. It runs on an on-premises Kubernetes cluster with Helm-based infrastructure-as-code, operates fully offline for privacy-sensitive workflows, and shares GPUs across modules using HAMi.

2. Six integrated analysis modules with heterogeneous models. AI-generated image detection (four detector variants plus 16 XAI methods), image forgery detection and localization (a ConvNeXtV2-Base encoder with a U-Net decoder and a global authenticity head), face morphing detection (ArcFace with a ResNet-100 backbone), deepfake video detection (19 integrated detectors), audio deepfake detection (a WavLM-based three-stage pipeline), and text analysis (XLM-RoBERTa classifiers aggregated into reliability levels).

3. Benchmark evidence across modalities. The paper reports F1 scores for AI-image detectors on eight evaluation sets, AUC/F1/IoU for forgery localization against five prior methods on four datasets, EER for morphing detection against six prior methods across six morph generation techniques, video AUC for 19 deepfake detectors, and cross-dataset accuracy/F1/AUC for audio deepfake detection on three unseen datasets.

4. A public demonstrator and open benchmark setup. A live instance at https://media-intelligence.ait.ac.at/ with a demo account (mip-demo / spot_the_fakes2026), plus a curated video benchmark of 1000 videos (50 real and 50 fake from each of 10 datasets) built in the style of DeepFakeBench to allow like-for-like comparison of detectors.

Main Findings

• AI-generated image detection generalizes to unseen generators. The four variants (ResNet-50 with different fine-tuning depths, and a fully fine-tuned ViT-B-16) were trained on 162k real MS-COCO images and 209k synthetic images from the AIText2Image dataset generated with Stable Diffusion XL, DALL-E 2, and Midjourney 5, yet still scored well on unseen generators such as FLUX.1, Firefly, and DALL-E 3. ViT-B-16 reached 94.9 F1 on FLUX.1, 86.9 on Firefly, 93.8 on Midjourney 5&6, and 97.0 on DALL-E 3.

• No single detector variant wins everywhere. ResNet50[fc+l4+l3] scored the highest on ImgN100 (99.2) and OImg-v7 (99.6) but dropped to 79.0 on Firefly and 74.8 on GLIDE, while ResNet50[fc+l4] reached 98.8 on FLUX.1 and 96.4 on Firefly. This variability is part of why the platform lets users compare multiple models side by side.

• The forgery localization model beats prior methods on average. The proposed IFDL model achieved an average mean score of 0.718 in default mode and 0.739 when averaging over augmented (mirrored, rotated) inputs, against 0.583 for both DF-Net and Wu22, 0.437 for FSim, 0.431 for DFCN, and 0.377 for MNet. Gains were largest on Columbia (AUC 0.991, F1 0.962, IoU 0.951 with augmentation).

• Performance on NIST imagery is markedly weaker. On the NIST dataset the IFDL model reached only 0.766 AUC, 0.428 F1, and 0.352 IoU with augmentation — the lowest of the four test datasets and an explicit soft spot in the results.

• The ArcFace-based morphing detector has the lowest average error of the compared methods. It achieved an average EER of 3.58% across DiffMorpher, Greedy-DiM, MipGAN-II, OpenCV, StyleGAN2, and UTW, versus 5.92% for Triplet-Loss, 10.02% for FatFormer, 21.95% for CrEffViT, 22.89% for SelfMAD++, 31.10% for SelfMAD, and 37.19% for SPL-MAD. Its best single result was 1.99% EER on UTW morphs.

• Large pretrained backbones dominate deepfake video detection. Across 19 detectors benchmarked on the 1000-video set, EFFORT (ViT, CLIP) led with a video AUC of 0.928, followed by Selimsef FbF (0.917) and GenConvViT (0.908); the weakest was LRNet (RNN, landmark, temporal) at 0.639 and MesoNet4Inception at 0.720.

• Preprocessing is a first-order factor, not a detail. Replacing per-detector preprocessing with a unified GPU-accelerated face detection and tracking pipeline improved runtime by up to 100x for some detectors while increasing robustness across all integrated methods.

• Identity leakage undermines cross-dataset claims. The authors report, citing prior work, that many video detectors implicitly learn facial identities rather than manipulation artifacts, producing strong in-dataset scores but poor transfer — making identity-disjoint train/test splits essential.

• Audio deepfake detection transfers well to unseen datasets. The WavLM module reached 0.99 accuracy, 0.99 F1, and 0.99 AUC on FakeAVCeleb; 0.91/0.91/0.92 on the ASVspoof5 evaluation subset; and 0.87 accuracy, 0.79 F1, 0.83 AUC on In-The-Wild, after training on ASVspoof 2019 (LA), ASVspoof 2021 (LA & DF), and ASVspoof5.

• Text modules are described qualitatively only. The paper lists the tasks (hate speech, toxicity, sexism, extremism, offensive language, sentiment, reporting style, writing style) and the aggregation into four reliability levels (Highly Questionable, Questionable, Slightly Questionable, Reliable), but reports no quantitative benchmark numbers for these classifiers.

Methodology in Plain English

The authors did not try to invent one model that does everything. Instead they treated each forensic problem separately, picked or built a strong model for it, and then wrapped every model in the same software envelope so the rest of the system does not need to know what is inside.

Each analysis task becomes a module packaged as an isolated Docker container. A module answers three questions over a standard web API: run an analysis, report status, and return results. Results come back in one shared format regardless of whether the module analyzed an image, a video, an audio clip, or a text — which is what makes a single front end and a single job history possible. Modules can be added, swapped, or scaled without touching the surrounding infrastructure, and each may ship its own configuration form and result visualization. The reference front end (MIP-UI) handles user management, role-based access control, and job tracking, and lists connected modules dynamically.

For the individual detectors, the approaches differ by modality. AI-image detection uses two architectural families (ResNet-50 and ViT-B-16) trained on a large mixed real/synthetic corpus with JPEG and filter augmentations, and is paired with 16 explainability methods whose heatmaps are combined with object detection so an analyst can see which objects carry the strongest generation artifacts. Forgery localization uses an encoder–decoder network that simultaneously classifies an image as authentic or manipulated and outputs a pixel-level mask of the tampered region. Morphing detection fine-tunes an ArcFace ResNet-100 model, which already encodes identity-discriminative features, for binary bona fide-versus-morphed classification using a morph dataset spanning both landmark-based and deep-learning-based morphing. Video detection does not propose a new model — it re-implements 19 published detectors under identical preprocessing and inference conditions. Audio detection uses WavLM on 4-second segments with a three-stage recipe: joint training with A-Softmax, contrastive, and center losses; then binary fine-tuning; then freezing the backbone and fitting a Mahalanobis distribution on bona fide embeddings so unseen attacks can be flagged as anomalies. Text analysis fine-tunes XLM-RoBERTa on German news and social media data, optionally fusing embeddings with handcrafted linguistic features, and aggregates classifier outputs into a single reliability rating through weighted rules.

Why This Matters

Impact on research. Most forensic papers optimize a single detector on a single dataset. This paper's contribution is infrastructural and comparative: it shows that heterogeneous state-of-the-art models can be operationalized together, and it produces apples-to-apples comparisons (19 video detectors under identical preprocessing, four image detectors across eight evaluation sets) that single-method papers rarely provide. The finding that identity leakage and preprocessing quality can matter as much as architecture is directly useful to anyone reporting cross-dataset deepfake numbers.

Real-world applications:

• Border control and identity verification, where morphed passport photos let multiple people match one document — the target scenario of the morphing module.
• Newsroom verification and fact-checking, where a journalist receives an unknown file and needs image, video, audio, and text signals in one place rather than four separate tools.
• Trust-and-safety and content moderation teams, who need API access to screen uploads at scale and to classify abusive or manipulative text alongside media.
• Fraud and impersonation investigations, where cloned voices or face-swapped video are used against individuals or organizations.

Industry relevance. The design choices read as production requirements rather than research conveniences: offline operation for privacy-sensitive evidence, on-premises Kubernetes deployment, GPU sharing via HAMi to keep multi-model inference affordable, standardized REST APIs so external partners can build their own front ends, and explainable outputs (heatmaps, localization masks, per-person confidence timelines) so that a human analyst can justify a decision. The authors explicitly contrast this with commercial systems such as TrueMedia, Sensity, and Reality Defender, which they describe as limited in transparency, deployment options, and evaluation openness.

Future Directions

• Diffusion-based morphing. The authors flag that diffusion-based morph generation increasingly removes visible blending artifacts and poses new challenges for the ArcFace-based detector.
• Harder post-processing for forgery localization. Named next steps include robustness against unseen editing tools, strong compression, and adversarial post-processing, plus integrating forensic evidence with metadata and cross-modal reasoning. The weaker NIST results point at the same gap.
• Diffusion-generated video. The video framework is to be extended toward multimodal diffusion-based video generation, where the task stays the same — real recording or synthetic generation — but the artifacts change.
• Broader robustness and multimodal reasoning. The conclusion calls for extending coverage to emerging manipulation techniques, improving robustness against unseen generative models, and strengthening reasoning across heterogeneous evidence types.

Target Audience

This paper is most useful to practitioners and researchers building or operating media-forensics systems: engineers who need a reference architecture for serving multiple heterogeneous detection models behind one API, forensic analysts and fact-checkers evaluating whether such a platform fits their workflow, and benchmark-oriented researchers who want cross-detector comparisons under controlled conditions. It is also relevant to policy and trust-and-safety audiences who need a concrete picture of what current detection can and cannot do — the paper is candid about weak spots such as NIST localization performance, identity leakage in video detectors, and the absence of quantitative text results. Readers looking for a novel detection algorithm will not find one here; readers looking for integration, deployment, and comparative evaluation will.

Authors’ abstract

The rapid progress of generative artificial intelligence has led to a significant increase in manipulated multimedia content, including synthetic faces, forged videos, AI-generated images, fake audio, and locally edited images. Existing detection approaches are typically designed for a single manipulation type, limiting their applicability in real-world scenarios involving heterogeneous media sources. In this work, we present the Media Intelligence Platform (MIP), a unified web-based framework for multimodal media forensics integrating specialized modules for fake video detection, face morphing detection, AI-generated image detection, audio deepfake detection, localized image manipulation detection, and text analysis. Its modular architecture enables flexible integration of heterogeneous models and supports future extensions for emerging manipulation techniques. Experimental results on representative benchmark datasets demonstrate reliable detection performance across multiple media modalities while providing a practical and user-friendly interface for forensic investigations. The platform highlights the potential of unified multimodal systems for comprehensive deepfake detection in real-world environments, and is accompanied by a public demonstrator released with this workshop submission to support reproducibility and community evaluation.