Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion

AI summary

Overview

Research area: Generative audio / spatial audio (cs.SD), specifically native First-Order Ambisonics (FOA) soundfield generation and instruction-guided editing.

Technical level: Advanced. The paper assumes familiarity with autoregressive sequence models, rectified-flow / diffusion transformers (DiT), variational autoencoders, contrastive audio–text embeddings (CLAP), and on-policy distillation.

Scope (one sentence): The paper introduces Ambit, a unified framework in which an autoregressive planner emits executable ScenePlans that are compiled into conditioning for a rectified-flow FOA renderer sharing the same Transfusion backbone, supported by an FOA-aware VAE and a joint on-policy self-distillation (OPSD) post-training stage.

What This Paper Is About

Generating or editing spatial audio requires two kinds of decisions to line up: what each sound source is and does (content, when it is active, how it moves in 3D), and how that is actually rendered into a multichannel soundfield. Editing adds a third requirement: change only the requested attributes and faithfully keep everything else. The paper's goal is a single model that makes these source-level decisions explicitly, writes them into a structured, executable plan, and then renders native four-channel FOA audio from that plan — while preserving the acoustics of untouched sources during edits.

Key Contributions

1. Executable ScenePlan planning coupled to flow rendering in one Transfusion backbone. The planner autoregressively predicts a grammar-constrained plan specifying scene duration, room type, and per-source type, description, optional transcript, active interval, and static or linear 3D trajectory (azimuth, elevation, distance). A deterministic compiler turns this into source-indexed semantic tokens and frame-aligned spatial controls for a rectified-flow DiT renderer. For editing, the planner predicts the complete target scene (including retained sources), with contrastive features grounding the requested change and full reference latents supplying residual acoustic detail for preservation.

2. An FOA-aware continuous VAE. The latent space is regularized for spatial audio through three additions: a shared-decoder auxiliary reconstruction that supervises only the omnidirectional W channel (reusing the decoder, adding no parameters), asymmetric grouped KL that gives the W latent group weaker regularization (β_W < β_R), and spatial covariance matching (SCM) that supervises relative channel energy, coherence magnitude, and inter-channel phase.

3. Joint Transfusion On-Policy Self-Distillation (OPSD). Two frozen teachers (a planning teacher with target-plan memory, a rendering teacher with target-audio reference tokens) supervise the shared student on student-generated prefixes and on sampling states reached by rendering the student's own plans, jointly updating shared and task-specific parameters.

4. A full empirical study across speech, music, and sound for generation, plus instruction-guided editing, with codec ablations, fixed-renderer comparisons, preservation metrics, and post-training comparisons against continued SFT, ReFL, GRPO, and single-branch OPSD variants.

Main Findings

• Codec quality: The Ambit VAE reaches W SI-SDR 6.39 dB, direction error 4.96°, intensity coherence error 0.04, and W paired CLAP 0.90, beating DAC (5.57 / 5.13 / 0.08 / 0.88), EnCodec (0.41 / 12.25 / 0.15 / 0.85), and WavTokenizer (−19.41 / 56.66 / 0.35 / 0.63) on all four metrics.

• SCM carries most of the spatial gain: Adding SCM to the base objective cuts direction error from 6.51° to 5.14° and raises W SI-SDR from 5.63 to 6.08 dB. Later cepstral and SI-SDR terms raise W SI-SDR to 6.25 dB with only a small further directional gain (5.08°). Grouped KL plus auxiliary W reconstruction then improves all four metrics to the full-model row.

• Generation content: Ambit leads on paired CLAP, KL-PANNs/KL-PaSST, and FAD in music (0.46 / 0.51 / 0.95 / 0.71) and sound (0.41 / 0.53 / 1.78 / 0.75). Caveat reported by the authors: AudioX-Turbo retains lower FD-PaSST and FD-OpenL3 (music 234.65 and 37.40 versus Ambit's 261.45 and 44.59; sound 213.60 and 40.56 versus 235.49 and 49.99).

• Speech: Ambit achieves the lowest WER (0.07) and CER (0.05) in its table, but lower UTMOS (3.52) than Qwen3-TTS 1.7B (4.18) and UniAudio2 (4.11). The authors attribute the lexical gains to plans that carry literal transcripts.

• Editing quality: Ambit improves all six reported content/spectral metrics over SwanWeave, the strongest baseline: paired CLAP 0.84 vs 0.68, FD-CLAP 0.05 vs 0.15, FAD 0.24 vs 0.57, FD-PANNs 6.01 vs 11.40, KL 0.46 vs 0.92, LSD 1.27 vs 1.65.

• Spatial generation: Ambit leads both scene-level and object-level metrics: scene GCC 12.59, CRW 8.75, FSAD 0.02, DOA 0.51, azimuth MAE 0.40, angular trajectory RMSE 0.63, wCS 0.56 (versus e.g. BEWO + AS at 34.71 / 73.22 / 0.03 / 1.04 / 1.00 / 1.13 / 0.11). Object-level: spherical MAE 0.531, azimuth MAE 0.415, trajectory RMSE 0.629.

• Planner vs. renderer matters: A fixed-renderer comparison (Qwen3.5-4B + DiT) beats AS-pipeline baselines on GCC, CRW, and wCS but has higher scene and object direction errors — the paper notes binaural gains need not imply accurate source placement, and Ambit's gains with the same renderer support the learned planner.

• Preservation in editing: Ambit gives unedited-source direction MAE 0.17, W NMSE 0.44, and absolute level change 1.65 dB, versus SwanWeave (0.90 / 1.49 / 8.40) and Qwen3-Omni + DiT (1.42 / 0.58 / 20.94). The latter improves scene/object and W NMSE over SwanWeave but worsens unchanged-source direction and level errors — motivating measuring preservation against the original source audio.

• Joint OPSD is the strongest post-training variant: It leads all nine reported metrics over continued SFT, ReFL, GRPO, AR-only OPSD, and DiT-only OPSD (e.g. paired CLAP 0.867, FAD 0.211, KL 0.351, LSD 1.076, GCC 11.130, CRW 5.854, FSAD 0.0115). DiT OPSD improves every metric over SFT, whereas AR OPSD improves only LSD and FSAD — supporting refinement of planning and rendering together.

Methodology in Plain English

The system has two "modes" running inside one shared Transformer: a language-model-style planner and an audio renderer.

Planning. A frozen small text model (Qwen3.5-0.8B) plus learned projections let the planner read the user's text. The planner writes a plan token by token, constrained by a grammar so that every plan is a legal, executable structure. Each source gets a type, a description, optional transcript, a time interval, and a 3D path. Because the plan is structured rather than free-form prose, a compiler can convert it mechanically into two conditioning signals: text-like semantic tokens for cross-attention, and frame-aligned spatial vectors (256-dimensional per latent frame) encoding activity overlaps and interpolated poses at frame midpoints.

Rendering. Clean target audio is encoded into a latent; noise is mixed in at a random flow time; the DiT predicts the velocity that would push the noisy latent back to the clean one, conditioned on the compiled plan (and, for editing, on the reference audio latent). Sampling integrates from pure noise to a latent, which the VAE decodes to four-channel FOA.

Editing specifics. The planner sees the reference audio's latent, Ambit CLAP features, and the instruction, and is asked to write the full target scene. A source decoder predicts edit targets, operations, and updated attributes, and its memory steers decoding through gated residual attention. Two contrastive heads are trained: a semantic head (content, source counts) and a scene head (also timing, spatial attributes, room properties), with content-preserving spatial edits treated as semantic positives but scene negatives.

Making the VAE care about space. Ordinary reconstruction treats all channels alike, so nothing forces the model to nail the omnidirectional W channel or the relationships between channels. The authors add a masked pass that reconstructs only W (reusing the same decoder), weaken the KL penalty on the W latent group so it can hold more information, and add a loss that compares local STFT statistics between reference and reconstruction — energy fractions per channel, coherence magnitudes between channel pairs, and stabilized phase factors.

Fixing the train/test mismatch. Training uses ground-truth plans and reference prefixes, but inference feeds the renderer the planner's own predictions, so the renderer sees different conditions and lands in different sampling states. OPSD closes this gap: teachers derived from a supervised model are frozen, then the student rolls out its own plans and sampling trajectories, and is trained to match teacher token distributions on its own prefixes and teacher clean-latent velocity predictions at the exact states it visited. Teachers, reference models, and pretrained backbones stay frozen.

Data. Training uses 1.6M synthetic and recorded FOA examples and 1M editing pairs; the VAE is pretrained on 1M FOA clips. Recorded material includes 80,000 FOA clips from Sphere360 and 3,809 clips from STARSS23 (segmented into consecutive 10 s clips), captioned with Qwen3-Omni; synthetic scenes are rendered with Pyroomacoustics at 44.1 kHz in ACN [W, Y, Z, X] order with SN3D normalization. Evaluation uses 8,000 generation requests, 5,000 editing pairs, and 2,000 VAE test clips. The model has 15 shared Transfusion blocks (width 1,024, 16 attention heads) and 332.6M trainable parameters.

Why This Matters

Research impact. The paper makes a case that source-level symbolic decisions and multichannel acoustic synthesis should be learned together rather than pipelined, and it supplies concrete machinery for doing so: a compilable plan schema, a VAE objective that explicitly supervises inter-channel structure, and an on-policy distillation recipe that handles the planner/renderer coupling. The preservation metrics are also a methodological contribution — the authors show that a system can score well on target-scene metrics while badly damaging untouched sources, which argues for evaluating editing against the original audio.

Real-world applications:

• Immersive media and VR/AR content, where a text brief must become a moving 3D soundscape rather than a stereo mix.
• Game and interactive-environment audio, where sources need timed activity and trajectories rather than static placement.
• Virtual acoustics prototyping, using room type (dry, moderate, reverberant, outdoor) as a controllable plan field.
• Spatial audio post-production, where a user wants to move, retime, or restyle one element of an existing ambisonic recording without re-recording the rest.

Industry relevance. Native FOA output (rather than mono/stereo followed by a spatializer) removes a cascade stage; the modest 332.6M trainable parameter count and frozen text/CLAP/VAE backbones make the approach relatively deployable; and instruction-guided editing with measurable preservation is directly aligned with what authoring tools need.

Future Directions

• Longer scenes. The current ScenePlan supports scenes up to roughly 15 s (at most 648 latent frames at a 1,024-sample hop, about 15.05 s). Extending to longer soundscapes is stated as an open direction.
• Consistency across successive edits. The authors explicitly flag evaluating acoustic consistency when edits are applied one after another as future work — a chain of plausible single edits can still drift.
• Speaker-count and scene-complexity scaling. Plans support one to four local sources and durations within the ~15 s window; whether planning and rendering hold up for denser or longer scenes is untested here.
• Closing the perceptual gap. Ambit has the lowest speech WER/CER but lower UTMOS than Qwen3-TTS 1.7B and UniAudio2, and loses on FD-PaSST/FD-OpenL3 to AudioX-Turbo in music and sound; better reconciling semantic/lexical fidelity with perceptual and distributional quality is a natural next step.

Target Audience

Researchers and engineers working on generative audio, spatial/immersive audio, and multimodal planning — particularly those interested in Transfusion-style hybrids of autoregressive and flow models, structured intermediate representations for controllable generation, audio VAE design for multichannel data, or on-policy distillation for coupled discrete/continuous systems. It is also relevant to practitioners building spatial audio authoring tools who want to understand what edit-preservation metrics should look like, and to readers who want a template for converting free-form language into a strictly typed, compilable scene specification.

Authors’ abstract

Spatial audio authoring coordinates source decisions with soundfield synthesis. For editing, plans specify what to change and retain, while reference audio supplies further acoustic detail. We present AMBIT, a unified framework for native First-Order Ambisonics (FOA) generation and editing with three connected contributions. First, a shared Transfusion backbone couples autoregressive planning with rectified-flow rendering through executable ScenePlans that specify source content, activity, and 3D trajectories. For editing, the plan describes the full target scene, including retained sources; contrastive features ground the requested change, while full reference latents supply residual acoustic detail for preservation. Second, an FOA-aware continuous VAE combines shared-decoder omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching to preserve content and inter-channel structure. Third, joint Transfusion on-policy self-distillation (OPSD) refines planning on student-generated prefixes and rendering at sampling states reached under student plans, with target-conditioned teachers supervising both modes of the shared student. Evaluations across speech, music, and sound show improved generation content and spatial quality, acoustic fidelity during instruction-guided editing, and unedited-source preservation over the compared baselines. Project page: https://wjm-george.github.io/AMBIT/