LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds

AI summary

Overview

Research area: Audio generation / multimodal machine learning — specifically streaming video-to-audio (V2A) synthesis for interactive and immersive applications.

Technical level: Advanced. The paper assumes familiarity with autoregressive language modeling, diffusion and flow-matching, causal VAE latents, and audio-visual representation learning (Synchformer, CLIP-style contrastive pretraining).

Scope: LiveFoley is a streaming, causal, autoregressive diffusion system that turns arbitrary live video into synchronized sound with latency below human perceptual thresholds, and it is evaluated against offline and streaming V2A baselines on VGGSound.

What This Paper Is About

Real-time interactive worlds (games, AR/VR, live camera feeds) need sound that is generated on the fly from video, but existing streaming video-to-audio systems are far behind offline models: they either shrink the visual encoder's receptive field to cut delay (ruining audio-visual alignment) or denoise each short audio patch in isolation (hurting audio quality). The authors first measure how much audio lag humans actually notice, then design a system that meets that latency budget while matching offline models on temporal alignment and leading streaming systems on audio quality.

Key Contributions

1. A perceptual latency study that sets the design budget. A six-participant, nine-clip adaptive-staircase user study finds that action sounds are the most sensitive category, becoming noticeable at about 0.34 s, versus about 0.49 s for speech and about 0.85 s for ambient sounds during free roaming.

2. Trailing Sliding Window for the visual frontend. Instead of shrinking the segment window, the window's tail is anchored to the current frame, with first-frame padding for the startup period. This converts off-the-shelf non-causal segment-level encoders (the frozen Synchformer visual encoder) into streaming modules without retraining, decoupling look-ahead from window length and tying it to a configurable hop size s (s ∈ {1, 2, 4} gives look-ahead τ ∈ {0.08, 0.16, 0.32} s).

3. Unified Varhop Framework. A two-layer transformer "Synchronizer" maps trailing tokens from different hop sizes into a shared latent space (with positional embeddings, intra-window aggregation, and a learnable latency-mode embedding), so one set of weights supports multiple latency modes and the hop size can be switched per scene at inference. Hop sizes are rotated in equal proportion across training batches.

4. Widened denoising windows for the local diffusion head (LocDiT). History LocDiT adds the two previous clean patches as context at no latency cost and is the default; Future-Warmup LocDiT breaks the autoregressive deadlock by conditioning the upcoming patch on the current hidden state plus the next visual token, giving cross-patch joint denoising for 0.08 s of extra backend look-ahead.

Main Findings

• Alignment matches offline models at 0.08 s look-ahead. On the full VGGSound test set, LiveFoley-48k reaches a DeSync of 0.48, essentially on par with offline MMAudio-L (0.47) and well ahead of streaming baselines V-AURA (0.69), WorldSonus (0.67), MMAudio (online) (0.71), and SoundReactor (1.03).

• Best audio quality among streaming systems. LiveFoley-16k gets the best FD_PANNs (9.56) and FD_PaSST (92.48) among streaming systems; LiveFoley-48k gets the best IS (11.42) among streaming systems and the highest PQ (6.26) of all systems. FD_PaSST of LiveFoley-16k is second only to offline MMAudio-L (74.16).

• A quality gap to offline models remains. LiveFoley-16k still trails MMAudio-L on FD_PaSSST (92.48 vs. 74.16), FD_PANNs (9.56 vs. 8.48), KL_PANNs (2.24 vs. 2.18), IS (11.30 vs. 11.62), and IB (0.29 vs. 0.31), which the authors attribute to causality: a streaming model must commit to each sound from the frames seen so far.

• Subjective listening test agrees. LiveFoley leads online systems in both audio quality (MOS-Q 3.23) and alignment (MOS-A 3.58) and approaches MMAudio-L in alignment (3.79). MMAudio (online) has competitive FD_PANNs and KL_PANNs but its chunk seams are clearly audible.

• Shrinking the window hurts alignment; sliding it does not. In a controlled V-AURA-backbone study on VGGSound-Curated, DINOv2 (0.08 s look-ahead) yields DeSync 1.10 and SigLIP2 (0.08 s) 1.14, while VideoMAE-V2 reaches 0.88 at 0.64 s. The trailing sliding window gets DeSync 0.61 at 0.08 s. As look-ahead drops from 0.32 s to 0.08 s, the chunked window degrades from 0.67 to 0.88, whereas the trailing window moves only from 0.60 to 0.61.

• One model serves all hop sizes. Across s = 1, 2, 4 the unified model matches single-hop experts in alignment (DeSync 0.47, 0.46, 0.45 respectively) with minor distribution-metric degradation. Removing the Synchronizer degrades DeSync by about 0.03 and FD_PaSST at all hop sizes. Setting s = 4 reduces frontend computation by 4× versus s = 1.

• Widening the denoising window helps in both directions. History LocDiT improves FD_PANNs from 21.08 to 17.55 and IB from 0.31 to 0.32 over Independent LocDiT. Future-Warmup LocDiT reaches the best scores (FD_PaSSST 143.92, IS 12.13, DeSync 0.46), versus 158.04 and 0.47 for History. Low-Rate LocDiT, which halves the backbone rate, is worse than Independent on alignment (DeSync 0.54) and FD_PaSSST (180.52).

• Latency is under the perceptual threshold. On a single A800 GPU, end-to-end latency at s = 1 is 156 ms; Future-Warmup LocDiT brings it to an estimated 236 ms, still below the measured thresholds.

Methodology in Plain English

The first step was measurement: the authors played clips with naturally synchronized sound and progressively delayed the audio, asking participants to say when the sound started lagging. This gave concrete latency budgets per sound category.

For the visual side, the problem is that the best audio-visual encoders (Synchformer) look at 16-frame segments all at once — 0.64 s of video — before emitting features, which is too slow. Past fixes simply used shorter chunks, but then the encoder sees less temporal context and alignment collapses. LiveFoley instead keeps the full 16-frame window but slides it so that the newest frame sits at the end of the window, meaning the model only needs the past. Repetition of the first frame covers the warm-up period. Because the window can slide every 2s frames while still producing s new tokens, the delay is set by the hop size rather than window length — a knob the user can turn. A small transformer (the Synchronizer) then normalizes the features from different hop sizes into consistent tokens, so a single audio model handles all latency settings.

For the audio side, the system works like a language model that writes continuous audio: a 28-layer causal LM (MiniCPM architecture, trained from scratch) consumes one visual sync token and the previous audio patch's embedding at 12.5 Hz, and a lightweight diffusion transformer (LocDiT) refines the current patch. Rather than denoising that patch alone, the authors let it see the two previous clean patches (History) or also a provisional next patch (Future-Warmup); the provisional patch is discarded and regenerated properly at the next step. Training uses a conditional rectified-flow objective with noise injection on historical latents to reduce exposure bias, and inference uses attention sinks with a rolling KV cache so streaming can run beyond the 9.92 s training sequence.

Why This Matters

Impact on research. The paper reframes the streaming V2A latency problem: the trade-off is not inherently between latency and alignment but between latency and compute, once the window slides instead of shrinks. It also provides a reusable perceptual-threshold measurement and a curated, strictly aligned evaluation subset (VGGSound-Curated, 1,485 clips) that exposes visual-frontend differences that the full VGGSound test set obscures.

Real-world applications:

• Video games — footstep, impact, and weapon sounds triggered by live rendered frames rather than pre-authored samples, with hop size tuned per scene (shooters need the smallest hop; open-world roaming tolerates larger ones).
• AR/VR and mixed reality — spatialized feedback for user actions in passthrough camera video.
• Live streaming and broadcast — automatic foley for live camera feeds with no post-production step.
• Robotics and embodied agents — acoustic event feedback tied to what a robot's camera sees.

Industry relevance. The system is deliberately source-agnostic and plug-and-play: it conditions on arbitrary pixel streams rather than on an internal video generator, so it can sit behind a game engine or camera pipeline. Running at 156 ms end-to-end on a single A800 GPU, with a mode switch that trades delay for compute, maps directly onto the deployment constraints of interactive products.

Future Directions

• Closing the remaining quality gap to offline models. The authors explicitly call this an open direction: under causality, a model must commit to each sound from the frames observed so far, whereas offline models jointly denoise all patches with global bidirectional attention.

• Extending look-ahead in the backend. Future-Warmup LocDiT shows that access to the upcoming patch improves quality; how much further the denoising window can be widened while staying under the perceptual threshold is unresolved.

• More latency modes and finer-grained adaptivity. The Unified Varhop Framework is demonstrated at s ∈ {1, 2, 4}; whether more modes, or per-event rather than per-scene switching, can be supported by one set of weights is an open question.

• Multimodal control. The appendix reports that a single checkpoint extends to text prompts and outperforms the state-of-the-art streaming system in both video-only and video-text modes, inviting further work on controllable streaming foley.

Target Audience

Researchers and engineers working on audio generation, multimodal learning, and generative models for interactive media; practitioners building real-time audio for games, AR/VR, and live video pipelines; and anyone studying the latency-quality-alignment trade-offs of streaming generative systems. Readers without a background in diffusion models or autoregressive audio generation will find the introduction and user study accessible, but the method sections require intermediate-to-advanced familiarity.

Authors’ abstract

Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/