JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation

AI summary

Overview

Research area: Multi-modal large language models (MLLMs), specifically joint audio-video (JAV) understanding and generation — informally, "sounding video" comprehension and synthesis.

Technical level: Advanced. The paper assumes familiarity with encoder–LLM–decoder architectures, diffusion transformers (DiTs), cross-attention fusion, LoRA fine-tuning, and next-token prediction versus diffusion objectives.

Scope: The paper introduces JavisGPT, a unified 7B-scale system that both understands synchronized audio-video content and generates it from instructions, built on an encoder–LLM–decoder stack with a custom synchronization module and a three-stage training recipe.

What This Paper Is About

Existing multimodal models typically handle audio and video as separate streams that get concatenated or interleaved, which loses the fine-grained timing and spatial relationships between a sound and the visual event producing it. JavisGPT's goal is to unify both sides of the task — understanding sounding videos and generating them — inside a single LLM, with explicit mechanisms for maintaining audio-video synchrony. The authors also build the instruction dataset needed to train such a model.

Key Contributions

1. First unified MLLM for joint audio-video comprehension and generation. JavisGPT accepts audio, video, text, or any combination, and can output text, synchronized sounding video, or interleaved sequences of both within multi-turn dialogue.

2. A synchronization-aware architecture. The design combines a SyncFusion module that fuses audio into video patches at matching spatio-temporal positions, and hierarchical learnable queries (JavisQueries for semantic conditions plus spatio-temporal prior conditions) that bridge the LLM's hidden states to a frozen JavisDiT generator.

3. A three-stage training pipeline. Multimodal pretraining to add the audio branch and align with the generator, audio-video fine-tuning for synchronization, then large-scale instruction tuning — progressively adapting Qwen2.5-VL rather than training end-to-end from scratch.

4. The JavisInst-Omni dataset. Roughly 200K GPT-4o-curated audio-video-text dialogue trajectories, manually verified, covering comprehension and generation across single- and multi-turn scenarios.

Main Findings

• State-of-the-art synchronized AV comprehension. JavisGPT reaches 93.8 on AVQA and 82.1 on MU-AVQA, beating Qwen2.5-Omni (91.5 / 79.9) and VideoLLaMA2.1 (both lower), while using fewer training samples (1.5M total).

• Minimal degradation of unimodal abilities. It retains Qwen2.5-VL's visual comprehension (70.2 vs. 70.5 on Perception) and slightly outperforms Qwen2.5-Omni on MVBench (video) and TUT2017 (audio: 82.1 vs. 78.3).

• Generation quality exceeds the base generator and other unified MLLMs. On JavisBench-mini, JavisGPT scores JavisScore 0.157 versus JavisDiT's 0.153, NExT-GPT's 0.038, and UnifiedIO-2's 0.053, with the best or near-best FVD (317.5), KVD (1.8), and AV synchrony metrics.

• SyncFusion is more efficient than alternatives. It achieves the best AV understanding scores with only 2.0K tokens and 224 ms latency, compared to concatenation (3.5K tokens, 246 ms), interleaving (3.5K tokens, 555 ms), and Q-Former (fast but much weaker accuracy).

• All three training stages matter. Removing AV-FineTune collapses generation (JavisScore 0.069); removing MM-Pretrain causes instability requiring a 10x lower learning rate and yields 0.135; the full pipeline gives 0.157.

• Comprehension and generation reinforce each other. Joint training beats training the two capabilities separately, with the larger gain on the generation side — better semantic understanding produces better conditioning embeddings.

• Human evaluation on interleaved dialogue. Across 100 multi-turn QA dialogues judged by five volunteers, JavisGPT substantially outperforms both UnifiedIO-2 and NExT-GPT, which frequently refuse, hallucinate, or fail to generate coherent synchronized output.

Methodology in Plain English

The system starts from an existing vision-language model (Qwen2.5-VL-7B) and bolts on an audio encoder (BEATs) plus a diffusion-based audio-video generator (JavisDiT) that stays frozen.

The key trick is how audio and video get merged. Rather than stacking the two streams side by side, the authors split the audio into segments matching each video frame and use cross-attention to inject the audio into the corresponding visual patches. Each resulting token then represents "a sound happening at this specific spot in this specific frame." This keeps the token count low and preserves timing.

For generation, the LLM emits special query tokens when it decides to produce a sounding video. One set of queries encodes the high-level semantic description of what should happen; a second set encodes fine-grained spatio-temporal priors — where events occur and when sounds start, stop, and fade. Both sets are projected into the conditioning space the frozen generator already understands, and trained with alignment losses against the generator's original text and prior encoders. This avoids fine-tuning the generator itself, cutting compute dramatically.

Training proceeds in three stages: first learn to hear (600K audio-text pairs) and roughly align with the generator (1.5M captions); then learn synchronization using 360K audio-video-text triplets with LoRA adapters; finally instruction-tune on the JavisInst-Omni dataset to handle dialogue, in-context reasoning, and multi-turn interaction.

Why This Matters

Research impact. This is the first system to treat sounding-video comprehension and generation as one problem inside one MLLM, and it establishes that the two capabilities mutually improve each other. The SyncFusion module and the frozen-generator-with-alignment-loss strategy are reusable design patterns for anyone working on multimodal generation. It also releases a substantial instruction dataset, which has been a bottleneck in this area.

Real-world applications:

• Avatar chatbots that both understand what a user shows them (a video with sound) and respond with generated synchronized video.
• Sounding movie curation and dubbing, where a model can describe, index, and generate matched audio-video clips.
• Music-driven video analysis, linking what is heard to what is seen for search, editing, or recommendation.
• Accessibility tooling that generates synchronized visual or audio descriptions for media, and educational content that responds to audiovisual prompts.

Industry relevance. The architecture is deliberately concise and built by adapting off-the-shelf components, which lowers the barrier to productization. Because the generator stays frozen, inference and training costs are manageable. The capabilities map onto short-form video creation, gaming content, and any platform that needs instruction-followed audiovisual synthesis rather than separate text-to-video and text-to-audio pipelines that desynchronize.

Future Directions

1. Fix the architectural asymmetry. Comprehension uses next-token prediction while generation uses diffusion loss, and generation queries can read comprehension embeddings but not vice versa. The authors suggest moving toward end-to-end autoregressive unified modeling, following recent image-domain work.

2. Scale up. The current system is 7B-scale on public data only. The authors see potential in scaling to 70B+ with trillions of multimodal tokens.

3. Add reinforcement learning post-training. RL could turn the model into a stronger reasoning system for comprehension and improve generation quality, textual consistency, and audio-video synchrony on the output side.

4. Extend to speech and finer control. Replacing the BEATs tokenizer with speech models like Whisper or WavTokenizer would add speech input/output, and additional conditioning signals could enable controllable editing and more complex instruction-driven generation.

Target Audience

Researchers and graduate students working on multimodal LLMs, audio-video generation, or unified understanding-generation models will get the most from this paper. It is also directly relevant to engineers building media generation products or interactive avatar systems, and to dataset builders interested in how instruction data for cross-modal tasks is curated and verified. Readers without a background in diffusion models, cross-attention fusion, or LLM fine-tuning will find the architecture sections difficult without supplementary reading.

Authors’ abstract

This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has a concise encoder-LLM-decoder architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. For instruction tuning, we construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.