LTX-2: Efficient Joint Audio-Visual Foundation Model

AI summary

Overview

Research area: Computer Vision / generative multimodal modeling — specifically text-to-audio+video (T2AV) diffusion transformers.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiTs), rectified flow / flow matching, latent VAEs, rotary positional embeddings (RoPE), and classifier-free guidance (CFG).

Scope: This paper introduces LTX-2, an open-source foundation model that jointly generates temporally synchronized video and audio from a single text prompt using an asymmetric dual-stream diffusion transformer.

What This Paper Is About

Existing text-to-video diffusion models produce visually strong but silent outputs, while most text-to-audio systems are specialized for speech, music, or foley alone. Attempts to combine the two typically use decoupled sequential pipelines (generate video, then add audio, or vice versa), which the authors argue cannot model the full joint distribution of the two modalities — lip sync is driven by audio, but reverberation and foley are dictated by visual context. LTX-2's goal is a unified, open, efficient model that learns both modalities together with bidirectional influence between them.

Key Contributions

1. Efficient asymmetric dual-stream architecture: A 14B-parameter video stream and a 5B-parameter audio stream (19B total, per Table 1), linked by bidirectional audio-video cross-attention with cross-modality AdaLN for shared timestep conditioning and 1D temporal RoPE for cross-modal alignment.
2. Text processing blocks with "thinking tokens": A text-conditioning module built on Gemma3-12B with a multi-layer feature extractor (features from all decoder layers, mean-centered, flattened, and projected via a learnable matrix), plus learnable thinking tokens processed through bidirectional transformer blocks for improved prompt understanding.
3. Compact neural audio representation: A causal audio VAE operating on two-channel (stereo) mel spectrograms computed from a 16 kHz waveform, producing latent tokens where each token covers roughly 1/25 seconds of audio as a 128-dimensional vector.
4. Modality-aware classifier-free guidance (bimodal CFG): A CFG extension with separate text guidance (s_t) and cross-modal guidance (s_m) scales, allowing independent control of textual conditioning and inter-modal alignment at inference.

Main Findings

• Joint architectural performance: The asymmetric design allocates more capacity to video than audio, and cross-modality AdaLN conditions one modality's scaling/shift parameters on the other modality's hidden states, which the authors say helps synchronize features when diffusion timesteps or temporal resolutions differ.
• AV cross-attention visualization: Attention maps averaged across heads and layers show the model spatially tracking a moving vehicle, shifting attention between speakers (and to both simultaneously), and focusing on the lip region during close-up speech.
• Audiovisual quality (human preference studies): LTX-2 "significantly outperforms" open-source alternatives such as Ovi on visual realism, audio fidelity, and temporal synchronization, and achieves human preference scores comparable to proprietary models including Veo 3 and Sora 2. These are described as internal benchmarks; no numeric preference scores are reported.
• Video-only rankings: In Artificial Analysis public rankings (as of November 6th, 2025), LTX-2 ranked 3rd in Image-to-Video and 4th in Text-to-Video, surpassing Sora 2 Pro and Wan 2.2-14B.
• Inference speed: On an NVIDIA H100 with 121 frames at 720p, a single-step Euler solver, and CFG=1, LTX-2 (19B, audio+video) ran at 1.22 seconds per diffusion step versus 22.30 seconds per step for Wan 2.2-14B (video only) — approximately 18× faster. The authors state the gap widens at higher resolutions and longer durations.
• Temporal scope: LTX-2 generates up to 20 seconds of continuous video with synchronized stereo audio, exceeding Veo 3 (12s), Sora 2 (16s), Ovi (10s), and Wan 2.5 (10s).
• CFG settings used for reported results: s_t=3 and s_m=3 for the video stream; s_t=7 and s_m=3 for the audio stream.
• Parameter-count inconsistency: The abstract and body describe a 14B video stream plus a 5B audio stream (19B total, matching Table 1), while the conclusion describes extending "a pretrained 13B video diffusion transformer with a lightweight 3B audio stream." These figures do not match each other.

Methodology in Plain English

The team built the model in three stages. First, raw signals are compressed: video goes through a spatiotemporal causal VAE, and audio (converted to stereo mel spectrograms at 16 kHz) goes through a separate causal audio VAE. The two modalities deliberately do not share a latent space, which lets each use its own positional encoding scheme (3D for video, 1D for audio) and its own compression level.

Second, text is processed more deeply than usual. Rather than using only the final layer output of a decoder-only LLM, the model pools features from every decoder layer of Gemma3-12B, projects them into a unified space, and passes them through bidirectional transformer blocks along with learnable "thinking" tokens. A separate text connector serves each modality stream.

Third, denoising happens in a dual-stream transformer where each block performs self-attention, text cross-attention, audio-visual cross-attention, and a feed-forward pass. In the audio-visual cross-attention step, only the temporal component of RoPE is applied to queries and keys, forcing cross-modal attention to focus on time synchronization rather than spatial alignment. At inference, a bimodal CFG formula adds separate text-guidance and cross-modal-guidance terms. For high resolution, the model generates a base latent at roughly 0.5 megapixels, upscales latents, then refines overlapping spatial and temporal tiles independently and blends them in latent space before VAE decoding — reaching 1080p without exceeding GPU memory. Audio latents are decoded to mel spectrograms and then to a 24 kHz waveform by a HiFi-GAN-based vocoder modified for joint stereo synthesis and upsampling, with double the generator channels of HiFi-GAN V1.

Training used a subset of the same dataset as LTX-Video, restricted to clips with significant and informative audio. The dataset size is not reported. A new captioning system was built that describes both visual and auditory tracks factually, including music, ambient sounds, dialogue transcriptions with speaker, language, and accent identification, camera motion, lighting, and subject behavior.

Why This Matters

Research impact: LTX-2 argues against the prevailing decoupled pipeline paradigm and demonstrates that a joint, asymmetric dual-stream model can achieve cross-modal alignment without duplicating a full visual backbone. The bimodal CFG formulation and cross-modality AdaLN conditioning are transferable ideas for other paired-modality generation problems.

Real-world applications:

• Content creation: generating expressive audiovisual material without specialized equipment or large production teams.
• Accessibility: producing inclusive media with speech and sound for visually impaired audiences.
• Localization: dubbing and localizing educational content across linguistic and cultural boundaries.
• Low-resource language content: producing audiovisual material for languages that lack extensive production infrastructure.

Industry relevance: The paper positions efficiency as the central competitive claim — an open 19B audiovisual model reported as roughly 18× faster per diffusion step than a 14B video-only open model on H100 hardware, and faster than Ovi, which uses two 5B streams fine-tuned from Wan 2.2-5B. All model weights and code are publicly released, making this a direct open-source alternative to closed systems like Veo 3 and Sora 2.

Future Directions

• Language coverage: Performance varies across languages and dialects; prompts in underrepresented languages may yield less accurate speech synthesis or weaker audio-visual alignment.
• Multi-speaker handling: The model may inconsistently assign spoken content to characters, sometimes confusing which character should speak specific lines.
• Long-form generation: Coherent audiovisual sequences longer than roughly 20 seconds can lead to temporal drift, degraded synchronization, or reduced scene diversity.
• Reasoning and grounding: LTX-2 is a generative diffusion model without explicit reasoning or world-modeling capabilities; deeper narrative coherence, factual grounding, and complex situational understanding depend on external systems such as large language models used to produce the conditioning text.
• Responsible deployment: The authors call for work on bias mitigation, authenticity verification, and improved traceability, plus clear disclosure of synthetic origin.

Target Audience

Researchers and engineers working on multimodal generative modeling, diffusion transformers, and cross-modal alignment will get the most from this paper, particularly those interested in joint audio-video synthesis architectures. It is also relevant to practitioners evaluating open-source alternatives to proprietary video generation systems, and to teams building video editing, dubbing, or content production pipelines. The paper is written for readers already comfortable with diffusion model internals; newcomers to DiTs, flow matching, and CFG will need background reading first.

Authors’ abstract

Recent text-to-video diffusion models can generate compelling video sequences, yet they remain silent -- missing the semantic, emotional, and atmospheric cues that audio provides. We introduce LTX-2, an open-source foundational model capable of generating high-quality, temporally synchronized audiovisual content in a unified manner. LTX-2 consists of an asymmetric dual-stream transformer with a 14B-parameter video stream and a 5B-parameter audio stream, coupled through bidirectional audio-video cross-attention layers with temporal positional embeddings and cross-modality AdaLN for shared timestep conditioning. This architecture enables efficient training and inference of a unified audiovisual model while allocating more capacity for video generation than audio generation. We employ a multilingual text encoder for broader prompt understanding and introduce a modality-aware classifier-free guidance (modality-CFG) mechanism for improved audiovisual alignment and controllability. Beyond generating speech, LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene -- complete with natural background and foley elements. In our evaluations, the model achieves state-of-the-art audiovisual quality and prompt adherence among open-source systems, while delivering results comparable to proprietary models at a fraction of their computational cost and inference time. All model weights and code are publicly released.