EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation

AI summary

Overview

Research area: Audio-visual generative modeling — specifically video-text-to-audio (VT2A) generation, controllability, and multimodal event reasoning.

Technical level: Advanced. The paper assumes familiarity with diffusion/transformer audio generators, CLAP-style audio-text embeddings, VideoLLM agents, and evaluation with automated plus human metrics.

Scope: The paper defines a new task (event-centric, hierarchically controllable video-grounded sound generation), releases a benchmark for it, and proposes a training-free agentic system that outperforms existing VT2A models on that benchmark.

What This Paper Is About

Existing video-to-audio systems are conditioned mostly on the video, with text treated as an optional short label or caption, so they tend to produce sound that matches what is visible while ignoring detailed written instructions. The paper argues this makes fine-grained creative control impossible — for example, you cannot cleanly say "turn the second meow at 00:07 into a lion roar" when a video has several meows.

The authors therefore define EchoFoley, a task built around a symbolic "sounding event" representation that states when, what, and how each sound is produced, together with a benchmark (EchoFoley-6k) and a training-free agentic method (EchoVidia) that follows such instructions.

Key Contributions

1. A new task formulation. EchoFoley defines event-centric hierarchical control for video-grounded sound generation, with a symbolic sounding-event tuple e = (t, d, p) where t = (t_start, t_end) is the temporal span, d is a semantic description of <subject, action, object> (object optional), and p holds controllable audio properties such as timbre, pitch, intensity, and spatialization. Control is organized into three levels (instance, group, video) and three independent types (temporal, timbre, volume).

2. A benchmark and evaluation suite. EchoFoley-6k is described as an expert-curated dataset of video–instruction–annotation triplets with dense event annotations, drawn from VGGSound and the PE Video Dataset, plus automatic metrics (TempCtl, TimbCtl, VolCtl) and human ratings (Instruction Adherence, Audio–Visual Coherence, Perceptual Quality).

3. A method: EchoVidia. A training-free agentic framework with a slow–fast thinking strategy, built around a VideoLLM agent that plans and edits a symbolic event plan using an action pool of 12 atomic operations spanning visual reasoning, sound design, and synthesis.

4. An event-awareness evaluation. The same data is repurposed to test nine VideoLLMs and omni-modal models on sounding event detection (Task 1) and temporal localization (Task 2), and to ablate the slow–fast strategy.

Main Findings

• Baselines fail on controllability. Across the evaluated VT2A models, temporal controllability stays low (0.18 to 0.43), timbre controllability clusters around 0.21 to 0.24 with only ThinkSound (0.34) and HunyuanVideo-Foley variants (0.46 to 0.48) showing moderate gains, and volume controllability is comparatively higher (0.50 to 0.69). Human raters reported Instruction Adherence below 2.60 out of 5.0 for these models.

• A visual-dominance bias. Baselines cluster in the upper-left of the Instruction Adherence vs. Audio–Visual Coherence plot: reasonable visual synchronization (Audio–Visual Coherence 3.2 to 3.6) but poor instruction following (Instruction Adherence 1.4 to 2.0). When visual cues and text conflict, the models favor the visual stream.

• Finer control is harder. Controllability drops sharply as the control level becomes more fine-grained, from video-level control down to group-level and then individual-event control.

• EchoVidia leads on every headline metric. It reaches 0.72 TempCtl, 0.78 TimbCtl, and 0.75 VolCtl; in human evaluation it attains 3.80 Instruction Adherence, 3.93 Audio–Visual Coherence, and 3.79 Perceptual Quality, outperforming the strongest baseline by +1.20, +0.40, and +0.32 respectively. The abstract states this corresponds to a 40.7% gain in controllability and 12.5% in perceptual quality over recent VT2A models, while Section 6.2 describes roughly 55% average improvement over the strongest baseline on the controllability metrics (+0.29, +0.30, +0.19).

• Audio aesthetics also improve. Under AES, EchoVidia scores 7.32 Production Quality, 4.29 Production Complexity, 4.33 Content Enjoyment, and 6.50 Content Usefulness, the best values in Table 2.

• Balance replaces bias. Unlike prior systems, EchoVidia sits high on both Instruction Adherence and Audio–Visual Coherence simultaneously, which the authors present as eliminating the visual-dominance effect.

• Slow–fast thinking helps event awareness. With SF, Gemini-2.5 Pro improves recall from 0.66 to 0.83 and F1 from 0.59 to 0.74 on Task 1; Qwen3-VL-30B-Thinking gains +0.09 recall and +0.18 F1. On Task 2, Gemini-2.5 Pro's IoU rises from 0.510 to 0.842 and Qwen3-VL-30B from 0.484 to 0.650 — a more than 60% relative improvement in temporal precision. Base models still show notable boundary drift.

• Dataset scale is reported inconsistently. The abstract and Section 4.2 describe "over 6,000 video–instruction–annotation triplets" and "42,000 fine-grained sounding event annotations," whereas Table 1 lists 937 videos, 14 video topics, an average video duration of 11 (range 6 to 30 seconds), 5,622 total sounding events, 6 sounding events per video on average, 6,018 total instructions, and 12 instructions per video on average. The paper does not reconcile these two sets of numbers.

• Human evaluation protocol. Human ratings used 50 randomly sampled video–instruction pairs, 6 participants, and achieved an inter-annotator agreement of 0.62 (Cohen's kappa).

Methodology in Plain English

1. Define a machine-readable middle layer. Instead of letting text go straight into an audio model, the authors define a "sounding event" — a structured record of when a sound happens, what it is, and how it should sound. Instructions, whether explicit ("a 1-second magical explosion at 0:03") or implicit ("the second meow"), are mapped into these records.

2. Build the data by combining AI proposals with human correction. Candidate videos are filtered for visually evident sound-producing interactions. Metadata and frame-level captions are generated, then a large language model proposes an imaginative story and a first draft of sounding events. Human annotators then convert the story into concrete instructions, fix event boundaries frame by frame, delete hallucinated events, and annotate attributes like pitch, timbre, volume, intensity, and spatialization.

3. Measure control along three axes. Temporal control is scored with IoU between the ground-truth event span and the generated audio's detected active region (TempIoU). Timbre control uses CLAP similarity between the audio segment and the event description. Volume control classifies each segment's relative loudness into low, medium, or high using two thresholds and checks agreement with the ground-truth label.

4. Run an agent instead of training a model. EchoVidia uses a VideoLLM agent that first finds candidate events and rough timings, builds a symbolic event plan, refines it through add/remove/modify actions, and finally hands the plan to a sound generation module that renders audio conditioned on visual and textual context. The slow–fast strategy gives the agent a global 1 fps overview ("fast") and a stretched, 16x slower-motion pass for precise localization ("slow": downsample to 16 fps, then stretch 16x back to 1 fps).

Why This Matters

Impact on research. The paper separates the question of what to generate (an editable symbolic event plan) from how to render it (a sound synthesis module), which gives the field a testable formulation for controllability rather than a single scalar quality score. It also shows that current VT2A models have a measurable visual-dominance bias, and that prompting-level reasoning strategies can improve event awareness without retraining.

Real-world applications.

• Film and video post-production, where editors need to place, replace, or reshape individual sound effects at specific timestamps.
• Video editing and social content creation, where a creator may want to transform one moment (a meow into a lion roar) without regenerating an entire soundtrack.
• Accessible multimedia creation, giving non-experts a natural-language way to author sound design.
• Training-data synthesis for world models, since the pipeline produces videos with aligned, densely annotated multimodal labels.

Industry relevance. The work comes from ByteDance Intelligent Creation with academic collaborators, and it targets a workflow (sound design for video) that sits directly inside production tooling. A training-free agentic layer is attractive industrially because it can be layered onto existing audio generators rather than requiring new end-to-end training.

Future Directions

• End-to-end trainable models. The conclusion explicitly suggests integrating the event-centric formulation into end-to-end trainable models rather than the current training-free agentic pipeline.
• Extending the symbolic representation. The authors propose broadening the event representation to more applications to expand creative generation capability.
• Why timbre and volume remain hard. Baselines perform relatively better on volume than on timbre and timing; the paper does not explain the mechanism, leaving a clear open question.
• Boundary drift in event localization. Even the strongest models show notable onset/offset drift on Task 2, so precise temporal grounding remains unresolved beyond the reported SF gains.
• Reconciling dataset-scale reporting. The gap between the advertised triplet/annotation counts and Table 1's statistics is not addressed and would need clarification for reproducible benchmarking.

Target Audience

Researchers and engineers working on audio generation, video-to-audio synthesis, and multimodal generation who need fine-grained controllability rather than one-shot soundtracks. It is also relevant to practitioners in film, video editing, and content creation tooling, and to those studying multimodal agents, video event localization, and VideoLLM reasoning — though the dense metric definitions and agent architecture make it most useful to readers with prior exposure to generative audio and multimodal evaluation.

Authors’ abstract

Sound effects build an essential layer of multimodal storytelling, shaping the emotional atmosphere and the narrative semantics of videos. Despite recent advancement in video-text-to-audio (VT2A), the current formulation faces three key limitations: First, an imbalance between visual and textual conditioning that leads to visual dominance; Second, the absence of a concrete definition for fine-grained controllable generation; Third, weak instruction understanding and following, as existing datasets rely on brief categorical tags. To address these limitations, we introduce EchoFoley, a new task designed for video-grounded sound generation with both event level local control and hierarchical semantic control. Our symbolic representation for sounding events specifies when, what, and how each sound is produced within a video or instruction, enabling fine-grained controls like sound generation, insertion, and editing. To support this task, we construct EchoFoley-6k, a large-scale, expert-curated benchmark containing over 6,000 video-instruction-annotation triplets. Building upon this foundation, we propose EchoVidia a sounding-event-centric agentic generation framework with slow-fast thinking strategy. Experiments show that EchoVidia surpasses recent VT2A models by 40.7% in controllability and 12.5% in perceptual quality.