Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception

AI summary

Overview

Research area: Multimodal perception — detailed captioning and evaluation for omni language models (OLMs) that process audio and video jointly.

Technical level: Advanced. The paper combines agentic data generation, large-scale fine-tuning of a 7B omni model, and construction of a human-verified cloze benchmark.

Scope: A single paper covering the full stack for fine-grained audio–visual captioning: an agentic data pipeline (Omni-Detective), two trained captioning models (Audio-Captioner, Omni-Captioner), and a new cloze-style benchmark (Omni-Cloze).

What This Paper Is About

Current omni language models can describe audio and video, but as their captions get longer the amount of hallucinated content rises along with the amount of correct detail — a trade-off the authors call "co-growth," demonstrated empirically on Gemini-2.5-Pro. The paper attacks this from three angles at once: generating training data through an iterative, tool-using "detective" agent that checks claims rather than one-shot prompting; training audio-only and audio–visual captioners on that data with a two-stage curriculum; and building a cloze-style benchmark that can score detailed captions cheaply and stably across audio, visual, and audio–visual settings.

Key Contributions

1. Omni-Detective, an agentic data generation pipeline in which a "Detective Agent" iteratively queries independent Observers through a Tool Box (MLLM, OCR, ASR, etc.) across multiple Query–Observation rounds to produce highly detailed yet minimally hallucinatory captions.
2. Audio-Captioner and Omni-Captioner, two 7B models built on the Qwen-2.5-Omni-7B backbone and trained with a two-stage curriculum (frozen visual encoder for audio alignment first, then full joint audio–visual training).
3. Omni-Cloze, described as the first cloze-style benchmark covering audio-only, visual-only, and audio–visual detailed perce

Authors’ abstract

Fine-grained perception of multimodal information is critical for advancing human-AI interaction. With recent progress in audio-visual technologies, Omni Language Models (OLMs), capable of processing audio and video signals in parallel, have emerged as a promising paradigm for achieving richer understanding and reasoning. However, their capacity to capture and describe fine-grained details remains limited explored. In this work, we present a systematic and comprehensive investigation of omni detailed perception from the perspectives of the data pipeline, models, and benchmark. We first identify an inherent "co-growth" between detail and hallucination in current OLMs. To address this, we propose Omni-Detective, an agentic data generation pipeline integrating tool-calling, to autonomously produce highly detailed yet minimally hallucinatory multimodal data. Based on the data generated with Omni-Detective, we train two captioning models: Audio-Captioner for audio-only detailed perception, and Omni-Captioner for audio-visual detailed perception. Under the cascade evaluation protocol, Audio-Captioner achieves the best performance on MMAU and MMAR among all open-source models, surpassing Gemini 2.5 Flash and delivering performance comparable to Gemini 2.5 Pro. On existing detailed captioning benchmarks, Omni-Captioner sets a new state-of-the-art on VDC and achieves the best trade-off between detail and hallucination on the video-SALMONN 2 testset. Given the absence of a dedicated benchmark for omni detailed perception, we design Omni-Cloze, a novel cloze-style evaluation for detailed audio, visual, and audio-visual captioning that ensures stable, efficient, and reliable assessment. Experimental results and analysis demonstrate the effectiveness of Omni-Detective in generating high-quality detailed captions, as well as the superiority of Omni-Cloze in evaluating such detailed captions.