# WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding

AI.info

> Understanding real videos means jointly interpreting what is seen and what is heard — objects, actions, speech, music, and ambient sound — and reasoning over how those streams relate in time.

**arXiv**: 2611.26409

**Published**: 2026-11-19

**Authors**: Changli Tang, Jie Yang, Tianyi Wang, Lei Li, Daoze Zhang, Zijie Xin, Zhan Liu, Li Li, Jinchang Xu, Fengyun Rao, Chao Zhang, Jing Lyu

## AI summary

## Overview

**Research area:** Multimodal computer vision and multimodal large language models — specifically omni-modal (text + image + audio + video) video understanding and reasoning.

**Technical level:** Intermediate. The paper is a technical report with architectural detail, a training recipe, and large benchmark tables; readers should be comfortable with terms like encoders, token budgets, distillation, preference optimization, and reinforcement learning.

**Scope:** A single report presenting the WeMM-Video family of Chinese–English bilingual omni-modal video models (2B, 4B, 9B, 35B-A3B), covering architecture, data mixture, training pipeline, and evaluation across more than 120 benchmarks.

## What This Paper Is About

Understanding real videos means jointly interpreting what is seen and what is heard — objects, actions, speech, music, and ambient sound — and reasoning over how those streams relate in time. Existing open-source models, the authors argue, are brittle here: they miss critical audio-visual details, fail to align a sound with its matching visual event, and hallucinate across modalities in ways the video does not support. The paper's goal is to build and openly release a family of models that handle text, images, audio, and video in one system, with both a fast "instruct" mode and a step-by-step "thinking" mode, and to show that they beat comparable open-source models and rival proprietary ones on omni-modal video.

## Key Contributions

1. **An open model family across four scales.** WeMM-Video spans 2B, 4B, 9B, and 35B-A3B variants, is Chinese–English bilingual, and unifies text, image, audio, and video within a hybrid-thinking paradigm offering a non-thinking instruct mode and an explicit chain-of-thought thinking mode.
2. **A progressive training recipe at scale.** Training uses approximately 1.2 trillion tokens through three pre-training stages (acoustic modality alignment, omni-modal knowledge injection, long-context capability scaling) followed by a three-stage post-training pipeline of multi-teacher on-policy distillation (OPD), anti-repetition multi-round direct preference optimization (MrDPO), and specialist reinforcement learning.
3. **A dedicated audio-visual front end.** A Whisper-based audio encoder operating on 16,000 Hz audio in 30-second chunks feeds a window-level Q-Former that compresses 0.1-second local windows into single audio tokens, yielding 10 audio tokens per second; audio and visual tokens are interleaved chronologically with explicit textual timestamps to support temporal alignment.
4. **Open evaluation and reproduction tooling.** Alongside model weights, the authors release their evaluation and reproduction pipeline to support the open-source community.

## Main Findings

- **Open-source state of the art in omni-modal video.** WeMM-Video-35B-A3B (Thinking) surpasses Seed2.0-Mini-0428 on 16 of 17 general audio-visual understanding benchmarks and Qwen3.5-Omni-Flash on 16 of 17, while narrowing the gap with Gemini-3.0-Flash. On Video-MME with audio it scores 86.4 in thinking mode and 84.2 in instruct mode, versus 88.6 for Gemini-3.0-Flash, 83.6 for Seed2.0-Mini-0428, and 79.3 for Qwen3.5-Omni-Flash.
- **Largest gaps on harder joint benchmarks.** On Video-MME-v2 with audio the model reaches 54.9 in thinking mode against Gemini-3.0-Flash's 63.2, Seed2.0-Mini-0428's 51.7, and Qwen3.5-Omni-Flash's 36.6. On OmniVideoBench it reaches 55.3 versus 43.7 for Qwen3.5-Omni-Flash; on SciVideoBench, 69.4 versus 51.3; on AV-SpeakerBench, 72.6 versus 65.2.
- **Thinking mode delivers measurable gains.** Comparing the 35B-A3B instruct and thinking columns shows explicit step-by-step reasoning lifts most audio-visual comprehension and complex reasoning scores — for example OmniVideoBench 48.6 → 55.3, SciVideoBench 57.8 → 69.4, SVHalluc 75.3 → 80.3, EvaOmni 48.8 → 57.8, and Video-MME with audio 84.2 → 86.4.
- **Small models outperform much larger open baselines.** WeMM-Video-9B (Thinking) beats Qwen3-Omni-30B-A3B-Instruct and MiniCPM-o-4.5-9B across most general audio-visual understanding and downstream tasks (e.g., Video-MME with audio 85.1 versus 77.3 for the 30B baseline and 69.2 for MiniCPM-o-4.5-9B). WeMM-Video-4B (Thinking) also surpasses the 30B-scale baseline on the vast majority of benchmarks, and even the 2B model matches or beats it on numerous perception, hallucination, and captioning benchmarks.
- **Silent-video ability is largely preserved.** The authors report little visual degradation from adding audio: WeMM-Video-35B-A3B (Thinking) reaches 80.4 on Video-MME, 85.3 on MLVU, 78.4 on MME-VideoOCR, and 84.1 on VidHalluc, leading Qwen3.5-Omni-Flash across nearly all silent-video benchmarks and performing on par with its Qwen3.6-35B-A3B backbone and Seed2.0-Mini-0428.
- **Audio, image, and text skills remain competitive.** On audio, the 35B-A3B model scores 70.6 on MMAU-Pro versus 64.8 for Qwen3.5-Omni-Flash, 96.5 on Speech-IFEval, and 62.0 on Audio MultiChallenge; LibriSpeech WER is 1.6 | 3.4. On images it reaches 81.1 on MMMU (backbone Qwen3.6-35B-A3B: 81.7) and 97.5 on DocVQA. On text it reaches 84.7 on MMLU-Pro (backbone: 85.2), 93.2 on IFEval and IFBench, and 90.0 on HMMT Nov 25 (backbone: 89.1).
- **Real deployment.** The models have been deployed across multiple core video services within WeChat, including live-stream commerce, short-form drama analysis, and interactive video scenarios.
- **Evaluation breadth.** Evaluation covers 33 omni-modal video benchmarks, 32 silent-video benchmarks, 18 audio benchmarks, 28 image benchmarks, and 12 text benchmarks — over 120 in total.

## Methodology in Plain English

The team starts from existing language models (Qwen3.5-2B/4B/9B and Qwen3.6-35B-A3B) and keeps their native visual encoders untouched, so pre-existing image and video skills are preserved. A new audio path is bolted on: a Whisper-based encoder turns sound into features, and a small compression module (the window-level Q-Former) squeezes every 0.1 second of audio into a single token, so one second of sound costs only ten tokens. For video, audio and visual tokens are interleaved in time order and accompanied by explicit textual timestamps so the model can link a sound to the moment it happened.

Training happens in stages that gradually widen what the model learns and how much context it can hold. First, only the audio connector is trained with everything else frozen, teaching basic hearing (speech recognition, audio captioning). Second, all parameters except the visual encoder are unfrozen and the model sees a broad mix of text, image, audio, and joint audio-visual data to build a shared cross-modal space. Third, the context window is stretched so the model can digest long videos, long recordings, and multiple videos at once. Data is dominated by joint audio-visual material (45.4% of tokens) and by captioning tasks (37.7%), with added audio-only corpora, cross-media comparison data, and knowledge/reasoning question-answer data enriched with thinking traces.

Post-training fixes three specific problems. Multi-teacher OPD distills from stronger external teachers for text and vision-language while same-family teachers preserve audio-visual skill, plus cross-modal distillation from text to audio. Anti-repetition MrDPO uses offline-mined hard negatives where the model rambles or loops, updating only LoRA adapters each round to stay stable. Specialist RL then targets hard tasks (audio-visual speaker identification, video-soundtrack matching, structured music understanding, complex audio-visual reasoning) using rule-based rewards for verifiable tasks and rubric-based or teacher-consistency rewards for open-ended ones, with a dynamic reward router and joint optimization of instruct and thinking modes.

## Why This Matters

The paper shows that a fully open model family can be competitive with proprietary omni-modal systems on joint audio-visual video understanding, and it releases not just weights but the evaluation pipeline — addressing a reproducibility gap that has made cross-paper comparison difficult in this area. It also demonstrates a concrete mechanism: explicit reasoning traces act as a practical mitigation for cross-modal hallucination, with judgment-task accuracy on SVHalluc rising from 75.3 in instruct mode to 80.3 in thinking mode for the 35B-A3B model.

Real-world applications, drawn from the paper's stated deployments and motivations:

- **Live-stream commerce** — checking that products shown on screen match what is said and heard, and verifying product-to-ad consistency.
- **Short-form drama analysis** — describing and indexing films and short dramas where dialogue, music, and on-screen action carry the plot.
- **Interactive video scenarios** — responding to users about video content in real time with either fast or deliberative answers.
- **Short-form video feeds and embodied intelligence** — the introduction cites these as the driving scenarios for accurate perception of dynamic audio-visual environments.

**Industry relevance:** the family spans 2B to 35B-A3B, so the same recipe covers latency- and compute-constrained deployment as well as flagship quality; the 2B model matching a 30B-scale open baseline on many benchmarks is the headline efficiency claim. The reported production use inside WeChat indicates the approach is already operating under real traffic rather than only in benchmark conditions.

## Future Directions

- **Closing the gap with Gemini-3.0-Flash on harder joint benchmarks.** The remaining deficit is concentrated in tasks like Video-MME-v2 with audio (54.9 versus 63.2) and Video-MME-Logical (26.0 in thinking mode), which suggests audio-visual reasoning under ambiguity and logical inference are still the frontier.
- **Scaling verifiable audio-visual reasoning data.** The authors explicitly note that verifiable audio-visual reasoning data is naturally scarce and had to be synthesized via a dual strategy; building better sources of it is the obvious bottleneck to extend specialist RL.
- **Long-context and multi-video robustness.** The paper extends RL to longer video frame coverage and sequence lengths, and includes streaming and multi-video benchmarks — extending these further, and improving weaker dimensions such as spatial understanding (e.g., MMSI-Video-Bench 35.6 in thinking mode for the 35B-A3B), remains open.
- **Eliminating residual cross-modal hallucination and decoding instability.** The need for a dedicated anti-repetition MrDPO stage, and imperfect scores on hallucination benchmarks such as AVHBench Judgement (80.0 in thinking mode), indicate generation reliability is not fully solved.

## Target Audience

Researchers and engineers working on multimodal LLMs, audio-visual learning, and video understanding who want a concrete, reproducible training recipe and a dense benchmark reference. It is also useful for practitioners choosing an open omni-modal video model under size constraints (the 2B/4B/9B comparison tables are directly actionable), and for teams building video products — streaming, commerce, media analysis — who need to know what current open models can and cannot do. Readers without a multimodal background will find the benchmark tables informative but the architecture and post-training sections dense.

## Authors’ abstract

Video understanding, particularly with accompanying audio, remains a central challenge in multimodal intelligence. In this report, we introduce \textbf{WeMM-Video}, a family of Chinese-English bilingual omni-modal foundation models spanning 2B, 4B, 9B, and 35B-A3B variants. Centering on joint audio-visual perception and reasoning, WeMM-Video unifies text, image, audio, and video processing within a hybrid-thinking paradigm, seamlessly supporting both a non-thinking instruct…

[The full abstract on arXiv](https://arxiv.org/abs/2611.26409)

[Read the original paper](https://arxiv.org/abs/2611.26409)

---

Canonical: https://ai.info/paper/wemm-video-pushing-the-frontiers-of-open-source-omni-modal-video-2611.26409
