OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

AI summary

Overview

Research area: Omni-modal large language models — systems that jointly perceive images, video, audio (ambient sound and human speech), and text.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, contrastive representation learning (CLIP-style losses), rotary positional embeddings (RoPE), and reinforcement learning from verifiable rewards (GRPO). The high-level argument is accessible, but the architectural contributions are described with formal notation.

Scope (one sentence): OmniVinci is NVIDIA's open-source omni-modal LLM built on Qwen2.5-7B-Instruct that introduces three alignment/embedding innovations plus a 24M-sample data curation pipeline, and reports state-of-the-art omni-modal, audio, and video benchmark scores while using roughly 6× fewer training tokens than Qwen2.5-Omni.

What This Paper Is About

Training a model that can simultaneously understand what it sees and what it hears is expensive and under-explored — most video LLMs either ignore the audio track or bolt it on without any mechanism for relating the two streams in time. The authors attack both halves of the problem: they redesign how vision and audio embeddings are aligned and timestamped inside the model, and they build a data engine that produces 24 million high-quality multimodal conversation samples. The resulting model, OmniVinci, is evaluated for whether joint audio-visual input actually helps perception and reasoning, not just whether it adds another modality.

Key Contributions

1. OmniAlignNet — a contrastive alignment module that projects visual and audio embeddings from the same video into a shared latent space and pulls matched pairs together while pushing unmatched pairs apart, using a symmetric CLIP-style loss (L_o-align = ½ of vision→audio plus audio→vision losses).

2. Temporal Embedding Grouping (TEG) — a method that partitions the time axis into chunks of duration T_G and interleaves vision and audio embedding groups according to their timestamps (e.g., [G_v1, G_a1, G_v2, G_a2]), so the LLM backbone sees relative temporal order between modalities.

3. Constrained Rotary Time Embedding (CRTE) — a RoPE-like rotation scheme with a defined maximum time horizon T_max and geometrically spaced base frequencies, encoding absolute timestamps across multiple frequency scales (high-frequency dimensions for fine temporal differences, low-frequency dimensions for long-range temporal relationships).

4. A 24M-sample curation and synthesis pipeline spanning 150+ sub-datasets, including an omni-modal data engine that fixes "modality-specific hallucination" by having an LLM cross-correct visual-only and audio-only captions, plus generated QA pairs with reasoning traces, and text-prompt-to-speech conversion via the Magpie TTS model.

Main Findings

• Omni-modal benchmarks (Table 3): OmniVinci reaches an average of 53.73 versus Qwen2.5-Omni's 49.66 (+4.07). It scores 48.23 on Worldsense (+2.83 over Qwen2.5-Omni), 66.50 on Dailyomni (+19.05 over Qwen2.5-Omni and +5.18 over Gemini-2.0-Flash-Lite), and 46.47 on Omnibench (above Gemini 1.5 Pro's 42.91).

• Ablation of alignment techniques (Table 1): Starting from a token-concatenation baseline at 45.51 average, TEG raises it to 47.72 (+2.21), CRTE to 50.25 (+4.74), and adding OmniAlignNet on top reaches 52.59 (+7.08 over baseline). A "Learned Time Embedding" alternative slightly degrades performance to 47.30, and RoTE reaches 47.80.

• Implicit vs. explicit omni-modal learning (Table 2): On Video-MME, visual-only training scores 66.37 (61.67 without subtitles); adding 270K video conversations with audio for implicit learning gives 66.96 (63.76 without subtitles); adding data-engine explicit learning gives 68.63 (67.37 without subtitles). The largest gains appear in long-video and no-subtitle settings.

• Audio understanding (Table 4): 58.40 on MMAR versus Qwen2.5-Omni's 56.70 (+1.7). The paper also reports +0.6 on MMAU.

• Speech recognition (Table 5): Word error rates of 1.7 on LibriSpeech-clean, 3.7 on LibriSpeech-other, 16.1 on AMI, 3.4 on Tedlium, and 6.8 on VoxPopuli, averaging 6.3. Two agentic-cascaded setups (ASR text history, retriever-based training) yield average WERs of 5.7 and 5.0.

• Video understanding (Table 7): 61.3 on LongVideoBench val (+3.6 over NVILA), 70.6 on MVBench, and 68.2 on Video-MME without subtitles (+3.1 over Qwen2.5-VL-7B and +3.9 over Qwen2.5-Omni's 64.3).

• Image understanding (Table 8): Scores across ten benchmarks (AI2D 91.5, ChartQA 84.6, DocVQA 91.5, InfoVQA 69.0, MathVista 63.5, MMMU 49.7, RealWorldQA 44.6, SEED 26.4, TextVQA 67.5, VQAv2 77.1, VQAv2 85.4), described as comparable to the NVILA baseline rather than a large leap.

• RL post-training (Table 9): GRPO on an 18K omni-modal MCQ dataset improves the average from 53.73 to 54.52 (+0.79), with Worldsense +0.47, Dailyomni +0.58, and Omnibench +1.32. OmniVinci overtakes Qwen2.5-Omni on the GRPO accuracy curve within 15 steps.

• Token efficiency: 0.2T training tokens — about a 6× reduction against Qwen2.5-Omni's 1.2T.

• Audio helps video, including for reasoning: The authors state (Key Insight 2) that audio understanding improves video benchmark metrics, and (Key Insight 3) that joint audio-visual input beats visual-alone input for GRPO, with audio present converging +0.1 higher in accuracy reward.

• Training data mix (Figure 5): Image 36%, non-speech sound 21%, speech 17%, omni-modal 15%, video 11%.

Methodology in Plain English

The starting point is a pretrained Qwen2.5-7B-Instruct LLM. Video is decomposed into a sequence of temporally correlated images plus audio, and a single unified audio encoder handles both ambient sound and speech. Each modality has its own encoder and projection layer producing embeddings.

Training happens in two stages. First, the model develops vision and audio abilities separately using modality-specific data. Second, in joint training, three kinds of data are mixed: randomly resampled modality-specific data, "implicit" omni-modal data (existing video QA datasets whose videos already carry audio — the audio is used even though the labels were not written with it in mind), and "explicit" omni-modal data generated by the data engine.

The data engine splits video into 20-second clips and generates visual and audio captions independently. Because each captioning model only sees one modality, both produce errors — the paper calls this "modality-specific hallucination," illustrating it with a deep-sea video that the vision model misread as being about human technology and the audio model misread as being about Earth's interior. A separate LLM then corrects and summarizes both captions into a joint description per 2-minute segment, and a reasoning LLM expands these into QA pairs with reasoning traces.

On the architecture side, embeddings pass through the three proposed mechanisms before entering the LLM backbone: TEG reorders vision and audio tokens into timestamp-based groups, CRTE rotates each embedding dimension pair by an angle proportional to the timestamp (rotating high-frequency pairs fast and low-frequency pairs slowly), and OmniAlignNet applies contrastive alignment between pooled visual and audio representations. Finally, GRPO reinforcement learning is applied on top with a rule-based reward for format and accuracy.

Why This Matters

Impact on research: The paper argues that audio and video are not merely additive but mutually reinforcing — for perception and for reasoning. It backs this with an ablation showing implicit learning from existing audio tracks improves video scores even when subtitles are available, and an RL ablation showing audio input accelerates and improves GRPO convergence. It also establishes that strong omni-modal performance is achievable without a correspondingly huge token budget.

Real-world applications demonstrated in the paper:

• Speech-prompted robot navigation
• Sports video understanding
• Cross-lingual speech translation
• Medical analysis that incorporates a physician's verbal explanation alongside imagery
• Semiconductor factory monitoring

Industry relevance: The 0.2T token budget versus Qwen2.5-Omni's 1.2T directly addresses the cost of training omni-modal systems, and the paper reports both training and inference cost reductions. The downstream demonstrations are industrial and clinical rather than purely academic, and the model, code, and webpage are released as open source.

Future Directions

• Scaling token budget: Since the paper deliberately caps training at 0.2T tokens, it is an open question how much further the reported gains extend if the token budget approaches or exceeds that of prior work.

• Text-to-speech output quality: The architecture is text-output based, and the paper notes that connecting off-the-shelf TTS modules involves tradeoffs discussed in Appendix D.4 — spoken-output quality is not a solved part of the design.

• Generalizing the data engine: The joint-captioning approach depends on a cross-modal correction LLM and a reasoning LLM; the paper does not report how sensitive results are to those components' quality or whether the pipeline transfers to long-form video beyond the 2-minute segment granularity described.

• Why audio helps reasoning: The finding that joint audio-visual input improves GRPO convergence and reward is reported as an observation with a modest +0.1 margin; the paper does not provide a mechanistic explanation for the synergy, which the abstract itself flags as an area of interest ("modalities reinforce one another").

Target Audience

Researchers and engineers working on multimodal or omni-modal foundation models, particularly those interested in cross-modal alignment, temporal encoding, and data curation pipelines. The paper is also useful to practitioners deciding whether adding audio to a video-language system is worth the engineering cost, and to teams building agentic or domain-specific applications (robotics, medical, industrial monitoring) who need a concrete comparison of open-source model performance and training-token efficiency.

Authors’ abstract

Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens - a 6 times reduction compared to Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.