VideoASMR-Bench: Can AI-Generated ASMR Videos Fool VLMs and Humans?

AI summary

Overview

Research area: Computer vision — AI-generated video detection, video generation evaluation, and multimodal (vision-language) model benchmarking, with a specific focus on ASMR (Autonomous Sensory Meridian Response) content.

Technical level: Intermediate. The paper's core ideas are accessible (real vs. fake video detection, human vs. model comparisons), but the evaluation protocol involves formal accuracy/F1 metrics and an adversarial metric design that assumes some familiarity with VLM benchmarking.

Scope in one sentence: The paper introduces VideoASMR-Bench, a 1,500-video real ASMR dataset paired with 2,235 synthetic ASMR videos from nine video generation models, and uses it to test whether vision-language models and humans can tell AI-generated ASMR videos apart from real ones.

What This Paper Is About

Existing video benchmarks mostly measure broad semantic alignment (does the video match the caption?) and coarse physical plausibility (do objects disappear? does gravity work?). The authors argue these tests are too easy to discriminate among today's frontier models. They propose using ASMR videos — tapping, brushing, whispering, food cutting — as a harder testbed because ASMR content depends on fine-grained audio-visual perception, precise temporal-rhythmic alignment, and sensory immersion. The goal is to answer two questions: (i) can today's video understanding models (VLMs) detect AI-generated ASMR videos by spotting minor visual, physical, or auditory artifacts, and (ii) can today's video generation models (VGMs) produce convincing, immersive ASMR videos?

Key Contributions

1. First systematic study of the ASMR domain for generative AI, treating ASMR as a stress test for fine-grained physical fidelity and audio-visual synchronization rather than as a niche content category.
2. VideoASMR-Bench, a dataset of 1,500 high-quality real ASMR videos curated from social media, plus an open-source ASMR Image Suite and ASMR Prompt Suite that let the benchmark scale as new video generation models are released. Synthetic videos total 2,235 across nine VGMs and four generation settings.
3. An adversarial understanding–generation evaluation framework in which VGMs try to produce fakes that fool VLMs while VLMs try to detect them, spanning what the paper describes as eleven VLMs and nine VGMs (Table 2 reports results for 10 VLMs; Table 3 reports six named generation models).
4. A set of diagnostic findings about why VLMs fail — reliance on watermarks as shortcuts, a bias toward labeling videos "real," and the underuse of audio cues — plus mitigation observations (prompt suite design, preference-based evaluation, explicit reasoning traces).

Main Findings

• Even the strongest VLMs fail at detection. The paper reports that Gemini-3-Pro struggles to detect AI-generated ASMR videos, with a detection score of 45 versus a human score of 70. In Table 2, humans average 87.50 accuracy and 70.34 F1, while Gemini-3-Pro-Preview reaches 70.57 accuracy but only 43.88 F1. Most VLMs score below 70 accuracy and below 45 F1.

• Best average F1 among VLMs is an open-source model. GLM-4.5V ranks second overall with 59.20 average accuracy and 45.39 average F1 — better average F1 than any proprietary model in the table, including Gemini-3-Pro-Preview (43.88) and Gemini-2.5-Pro (41.74).

• Some VLMs perform near chance. Qwen3-VL-235B-A22B averages 52.35 accuracy and 18.19 F1, and GPT-4o-mini averages 52.42 accuracy and 18.87 F1, against a random baseline of 50.00/50.00.

• Generators largely fool VLMs. Veo-3.1-Fast performs best, with only 12.54% of its generated videos identified as fake. HunyuanVideo (Text2Vid) follows at 14.77% and Wan2.2-A14B (ImgText2Vid) at 16.10% — both better than Sora2 with its watermark removed (16.55%), which the authors read as the proprietary/open-source gap narrowing.

• Worst generator is far behind. StepVideo (Text2Vid) records 83.62, meaning most of its videos were correctly flagged as fake. Sora2 without audio removal settings also performs poorly: 55.72 for Text2Vid and 46.58 for ImgText2Vid.

• Watermarks are a shortcut, not a signal. State-of-the-art VLMs detect the default Sora2 watermark and label videos fake without any prompt cue. Removing it causes an average drop of roughly 30 points; GPT-5 and Gemini-2.5-Pro/Flash drop sharply (for example, 95.4 to 53.4). GPT-4o and GPT-4o-mini remain stable.

• VLMs have a "real" bias. Across all VLMs tested, the models consistently lean toward predicting "real," misclassifying up to approximately 71% of fake videos.

• Audio helps. Adding audio inputs improves real-versus-AI detection, with an average gain of 5 points on the two most realistic fake video sets (Veo3.1 and Sora2). In the generation tables, adding audio raises Sora2's Fakeness score by 2.89 and Veo-3.1-fast's by 2.51. The authors attribute this to poor audio-visual alignment in current generators.

• Sora2 produces speech, not ASMR sound. Qualitative analysis finds Sora2 often generates human speech instead of continuous ASMR sounds, while Veo3.1-fast generates more action-aligned audio without speech.

• The ASMR prompt suite improves realism. Comparing against a minimal "generate the video as realistically as possible" baseline, the prompt suite consistently raises realism scores across models, especially GPT-4-turbo and Gemini-2.5-pro.

• Preference beats direct judgment. Giving a model a real and a fake video side by side produces higher realism scores than asking it to classify a single video. Pair versus Pair Shuffle settings have little impact. Explicit reasoning traces also improve VLM detection accuracy.

• Humans still win. Human annotators identify generated videos relatively easily while VLMs do not. Table 2 shows human accuracy of 81.25–91.25 and F1 of 64.00–74.07 across the four generators tested.

Methodology in Plain English

The authors started by scraping YouTube for highly viewed ASMR videos, using view count above 900K as a proxy for human immersion, and collected over 100 raw videos totaling more than 2,000 minutes. Four cleaning steps — splitting compilations into clips, removing artificial backgrounds and watermarks, extracting first frames, and manual inspection — reduced this to 1,500 high-quality clips.

For each clip they wrote a text prompt describing environment, subjects, actions, temporal dynamics, and sounds. Easy-level prompts (735 videos, mostly kitchen food-cutting scenes) were generated using Sora2's storyboard module conditioned on the ASMR image; hard-level prompts (1,500 videos) were generated with Gemini-2.5-Pro via its API from eight uniformly sampled frames per video. All prompts were manually inspected. Videos were clustered into eight semantic classes using Qwen3-Embedding-4B and sampled to reduce class imbalance.

They then generated fake ASMR videos with six named models (Wan2.2, Opensora-V2, HunyuanVideo, Sora2, Veo3.1-fast, StepVideo) under four settings: text-image-to-video (default), text-to-image-to-video, text-only, and image-only with the minimal prompt "Generate the video as real as possible."

For evaluation, they built a two-player game. VLMs see a balanced mix of real and fake videos and must output a step-by-step reasoning process before a final real/fake answer parsed from an <answer> tag. VLM performance is measured by detection accuracy and macro-averaged F1 (F1 is preferred over accuracy because a model that always says "real" gets 50% accuracy by construction but has no discriminative power). Generator performance is measured by "Fakeness" — the share of its videos that fool the VLMs — where lower is better. Ten annotators with mixed backgrounds (video generation researchers, non-experts, PhD students, undergraduates) judged videos independently with no prior information about the real-to-fake ratio and with audio included when available.

Why This Matters

Impact on research. The paper argues that current benchmarks — VBench, VideoPhy, PhyGenBench, SEED-Bench, MV-Bench, TempCompass, GenVideo, LOKI, IPV-Bench — evaluate either generation or understanding, and mostly visual-only or coarse physical cues. VideoASMR-Bench is positioned as the first benchmark using strict audio-visual alignment and sensory immersion as the distinguishing criteria, and the first to combine real/fake pairs, VLM detection, VGM scoring, human baselines, and text/image/audio/video inputs in one protocol. It also documents a concrete failure mode: frontier VLMs leaning on watermarks rather than perceptual evidence.

Real-world applications (note: the paper's Broader Impacts section states only that there are many potential societal consequences and does not enumerate specific applications; the following follow from the benchmark's design):

• Content authenticity verification on video platforms, where synthetic ASMR and similar sensory content could be misrepresented as real footage.
• Media forensics and provenance tooling, since watermark removal collapses VLM detection by roughly 30 points on average and shows detection cannot rely on embedded marks.
• Evaluation and red-teaming of video generation models before release, using Fakeness scores as a difficulty measure for how convincing outputs are.
• Dataset and benchmark engineering for fine-grained audio-visual realism, since the prompt and image suites are released to keep the benchmark extensible.

Industry relevance. Proprietary generators Veo-3.1-fast and Sora2 sit at opposite ends of the Fakeness ranking (12.54 versus 55.72 in Text2Vid), while open-source HunyuanVideo and Wan2.2-A14B land at 14.77 and 16.10. That distribution is directly relevant to teams deciding which generators to build on, to platform trust-and-safety teams setting detection thresholds, and to VLM developers whose models currently sit near the random baseline on this task.

Future Directions

• Scaling the benchmark as generators improve. The authors designed the prompt and image suites to be extensible so newly released video generation models can be incorporated while preserving evaluation consistency; the paper does not report results for all nine VGMs it names in the abstract across every table.
• Fixing watermark dependence. The roughly 30-point average drop when the Sora2 watermark is removed shows detection must be trained on intrinsic perceptual cues rather than embedded marks; what those cues should be is left open.
• Improving audio-visual alignment in generators. Adding audio helped VLMs detect fakes and worsened Fakeness scores, which the authors link to inadequate audio-visual coherence — particularly harmful for ASMR, where consistency and immersion are essential.
• Reducing the "real" bias. VLMs misclassify up to approximately 71% of fakes as real. Whether preference-based evaluation and explicit reasoning traces can close this gap at scale is raised but not resolved.
• Rethinking evaluation protocol. The paper notes that human annotation may introduce subjective bias and proposes a mutual-supervision setup among diverse VLMs and VGMs as a more rigorous alternative; it is presented as a benchmark and evaluation protocol rather than a methodological advance.

Target Audience

Researchers working on AI-generated video detection, video generation evaluation, and vision-language model benchmarking will get the most from this paper, particularly those interested in fine-grained physical realism and audio-visual synchronization. It is also useful for practitioners building trust-and-safety or provenance systems for video platforms, and for VGM and VLM developers who need a harder test than existing semantic and physical-consistency benchmarks. Readers do not need deep prior knowledge of ASMR, though familiarity with accuracy/F1 evaluation and multimodal model prompting will make the experimental sections easier to follow.

Authors’ abstract

With AI-generated videos increasingly indistinguishable from reality, current benchmarks primarily focus on broad semantic alignment and basic physical consistency, offering limited discriminative power for evaluating them. To address this, we introduce VideoASMR-Bench, a benchmark based on Autonomous Sensory Meridian Response (ASMR) videos that emphasizes fine-grained audio-visual perception and sensory immersion. This benchmark aims to answer two key questions: (i) Are today's video understanding models (VLMs) sensitive enough to detect AI-generated ASMR videos by recognizing minor visual, physical, or auditory artifacts? (ii) Can today's video generation models (VGMs) produce convincing ASMR videos with immersive experiences? This benchmark comprises a diverse set of 1,500 high-quality real ASMR videos curated from social media, alongside 2,235 synthetic counterparts generated by nine VGMs. Additionally, we open-source an extensible suite of prompts and reference images, enabling the benchmark to scale dynamically with future video models. Moreover, we design an automatic understanding-generation evaluation framework between VGMs and VLMs, where VGMs aim to produce realistic fake videos to fool the VLMs, while the VLMs seek to detect them, forming an adversarial game between the two parties. Our evaluation on VideoASMR-Bench reveals that even state-of-the-art VLMs, such as Gemini-3-Pro, fail to reliably detect AI-generated ASMR videos. Meanwhile, current frontier video generation models can produce ASMR videos that are difficult for VLMs to distinguish from real ones, while humans can still identify them relatively easily.