FLOMSS: Generative Music Source Separation by Flow Matching

AI summary

Overview

• Research area: Music source separation (MSS) using generative modeling, specifically conditional flow matching (CFM) in the compressed complex-STFT domain.
• Technical level: Advanced. The paper assumes familiarity with continuous normalizing flows, optimal-transport and Schrödinger-bridge probability paths, ODE solvers, and STFT-domain audio processing.
• Scope: A systematic study of supervised conditional flow matching for separating full-band stereo music into four stems (Vocals, Bass, Drums, Other), evaluated on MUSDB18-HQ with signal-based, embedding-based, and subjective listening-test metrics.

What This Paper Is About

Music source separation has been dominated by discriminative models that map a mixture directly to source waveforms or spectrograms. Prior generative approaches for MSS (InstGlow, Multi-source Diffusion/MSDM, MGE-LDM) typically learn source priors or perform conditional inpainting rather than explicitly learning a mixture-to-source generative mapping, and they do not produce full-band stereo audio. This paper asks whether conditional flow matching, which has succeeded in speech enhancement and speech separation, can be systematically adapted to full-band stereo four-stem MSS, and how such models compare to discriminative ones beyond the usual SDR score.

Key Contributions

1. FLOMSS framework: An open-source generative framework (to be released upon acceptance) for full-band stereo MSS using conditional flow matching, separating Vocals, Bass, Drums, and Other at 44.1 kHz stereo.
2. Systematic ablation of flow-matching design choices: prediction targets (velocity vs. data prediction), flow paths (OT-CFM vs. SB-CFM), single-source versus joint multi-source modeling, and soft versus hard mixture-consistency constraints.
3. Extended evaluation protocol for generative MSS: beyond waveform/SDR metrics, the authors add spectral distances (Mel, STFT, waveform L1), perceptual similarity (LPAPS with CLAP embeddings), and distributional metrics (Fréchet Distance and MMD over VGGish-16k, PaSST-32k, OpenL3-48k, CLAP-48k, and MERT-24kHz), plus a listening test with 23 participants.
4. Benchmark against discriminative and generative baselines: including a same-backbone discriminative baseline (FLOMSS-S-Disc.), SCNet, HTDemucs, Open-Unmix, InstGlow, MGE-LDM, and SAM Audio, with explicit notes about differing training data and audio formats.

Main Findings

• Prediction target and path matter differently: In the single-source ablation (Table 1, average across stems), OT-CFM with velocity prediction (OT-CFM-v) gives the best embedding-based scores (e.g., FD VGG 0.77, FD CLAP 0.23, LPAPS CLAP 2.43, FD MERT 5.35), while OT-CFM with data prediction (OT-CFM-x) is better on all signal-based metrics (Mel 1.63, STFT 2.69, waveform L1 0.0226, SDR 3.77). SB-CFM-x is best among CFM variants on waveform distance (0.0203) and SDR (4.63) but worse on embedding metrics (FD VGG 2.88, FD MERT 6.87).
• The discriminative baseline wins on reconstruction, loses on distributional metrics: The same-backbone discriminative model achieves the best waveform L1 (0.0197) and SDR (4.95) in the ablation, yet is worse than both OT-CFM variants on Mel distance (2.30 vs 1.71/1.63), STFT distance (3.16 vs 2.81/2.69), and most embedding-based metrics (FD VGG 2.55 vs 0.77/0.87; LPAPS CLAP 2.58 vs 2.43/2.45).
• A distributional–reconstruction trade-off: Stepwise analysis of OT-CFM-x for Vocals (Figure 1, Midpoint ODE steps i = 0 to 9) shows early estimates approach the discriminative baseline in Mel distance and SDR, while later estimates improve spectral and embedding-based metrics at the cost of SDR.
• Joint multi-source modeling is efficient but costly in quality: The unconstrained multi-source model produces all four stems with nearly the same backbone size but drops average SDR from 3.77 (single-source) to 3.07 and worsens embedding metrics (FD VGG 0.87 to 2.30, FD PaSST 261 to 416, FD MERT 5.43 to 10.6).
• Soft mixture consistency partially recovers quality; hard consistency hurts: Soft MC brings SDR back to 3.69 while preserving Mel (1.73) and LPAPS (2.71). Hard MC in the waveform domain (SDR 2.11, Mel 2.74, FD PaSST 497) and in the uncompressed STFT domain (SDR 2.34, Mel 2.71, FD PaSST 597) degrades overall, except on MERT embeddings (9.61 and 8.83 vs 5.43 for single-source).
• FLOMSS beats prior generative baselines on most spectral and perceptual metrics: Under format-matched evaluation, FLOMSS-S (16M) reaches average Mel 1.63, FD VGG 0.87, FD PaSST 261, FD CLAP 0.31, FD MERT 5.43, LPAPS CLAP 2.45, and SDR 3.77, versus InstGlow's Mel 3.08, FD VGG 3.96, FD PaSST 535, FD CLAP 0.69, FD MERT 10.6, LPAPS 3.27, SDR 2.75.
• Scaling to FLOMSS-M (67M) improves every metric: Average Mel 1.55, SDR 4.34, FD VGG 0.65, FD PaSST 205, FD CLAP 0.24, FD MERT 5.15, LPAPS CLAP 2.30. It surpasses Open-Unmix on all metrics except SDR, and beats SCNet and HTDemucs on average Mel and FD while keeping a weaker SDR (4.34 vs 9.69 and 8.14).
• Subjective results are competitive despite lower SDR: In the listening test (23 participants, ten 8-second excerpts, 0–100 scale), FLOMSS-M scores 59.12 overall and ranks 2nd, behind SCNet (62.59) and ahead of HTDemucs (58.71), Open-Unmix (35.40), FLOMSS-S-Disc. (40.28), and InstGlow (26.6). It ranks 1st on Bass (51.71), 2nd on Vocals (67.16), and 3rd on Drums (61.21). Reference conditions score 94.57 (ground truth) and 78.24 (16 kHz ground truth).

Methodology in Plain English

The system works on short-time Fourier transform (STFT) representations of audio rather than raw waveforms. The magnitude of each spectrogram is compressed with a power law (compression factor α = 0.5, scaling β = 0.15) to make values more consistent with the Gaussian noise the model uses.

Training is framed as learning a path from noise to a clean source. In flow matching, the model is given a mixture plus a noisy, partially denoised version of the target source at some "time" t and asked to predict either the direction of motion (velocity prediction) or the clean target itself (data prediction). The path's mean is a straight line from the mixture spectrogram to the target source spectrogram, and the two variants differ in how variance is scheduled: OT-CFM uses σ_max = 0.5 and σ_min = 0, while SB-CFM uses a Brownian-bridge noise scale of 1, giving a maximum standard deviation of 0.5 at t = 0.5. At inference the model integrates the learned field with a 10-step solver (Midpoint ODE for OT-CFM, exponential integrator for SB-CFM) and then inverts the compression and the STFT.

The backbone is a 16M-parameter NCSN++ U-Net following SGMSE, scaled to 67M by widening channels; the discriminative baseline uses the same architecture but only sees the mixture. A multi-source variant expands only the input and output channels to carry all four sources, with a fixed stem order. Two mixture-consistency variants are tested: a soft version adding 0.1 × an MSE loss on the sum of estimated sources, and a hard version that forces the per-source states to sum exactly to the mixture at all times (defined in the waveform or uncompressed complex-STFT domain, since the compression is nonlinear and does not preserve additivity).

Training uses the official MUSDB18-HQ split at 44.1 kHz stereo, 97,920-sample segments, online remixing of 2–4 stems from the same song region, 93,600 steps with Adam at a learning rate of 10⁻⁴, and an effective batch size of 32 on 4 A100 GPUs. Inference uses the same segment length with 10% overlap and triangular crossfades.

Why This Matters

• For research: It provides the first systematic mapping of flow-matching design choices onto full-band stereo MSS, and it shows that SDR alone is an incomplete measure of generative separation quality. The paper documents a trade-off between waveform reconstruction fidelity and distributional/perceptual similarity, which reframes how generative separators should be evaluated.
• For evaluation practice: The paper contributes an evaluation suite combining signal metrics, LPAPS with CLAP embeddings, and FD/MMD over five pretrained audio encoders, giving future work a ready-made benchmark protocol.
• Real-world applications:
  • Stem extraction for music production, remixing, and DJ tools.
  • Karaoke and instrumental/acapella generation from existing recordings.
  • Catalog re-processing: upmixing or re-balancing legacy stereo masters.
  • Interactive/creative audio tools where a generative model produces plausible stems rather than strictly reconstruction-optimal ones.
• Industry relevance: The work comes from Sony Group Corporation and targets 44.1 kHz stereo full-band audio, the format professional music workflows actually require. The 67M-parameter model's second-place subjective ranking suggests generative separators can be perceptually competitive even without matching state-of-the-art SDR, which is relevant to products where perceived quality matters more than a distortion score.

Future Directions

• Closing the SDR gap: The authors explicitly leave further SDR improvement as future work; FLOMSS-M's average SDR of 4.34 trails SCNet (9.69) and HTDemucs (8.14).
• Making hard mixture consistency work for MSS: Because hard MC degraded most metrics except MERT embeddings, the authors suggest that it may require an MSS-specific flow path and training strategy, and note they deliberately omitted auxiliary signal-level losses (unlike the speech-separation work they cite).
• Solver efficiency and sampling steps: The study uses 10-step solvers throughout; the paper does not report results for other step counts, leaving the speed/quality trade-off of the flow path unexplored.
• Extending beyond four fixed stems and to other datasets: All experiments use the four-stem MUSDB18-HQ setup; generalization to larger catalogs and finer instrument taxonomies is not evaluated.

Target Audience

• Researchers in audio source separation and generative audio modeling who want a rigorous reference point for applying flow matching to music.
• Practitioners building music production, remixing, or karaoke tools who need to know how a generative separator's perceptual quality compares with discriminative alternatives.
• Evaluation-focused researchers interested in embedding-based and distributional metrics (FD, MMD, LPAPS) as complements to SDR.
• Readers should have graduate-level familiarity with diffusion/flow models and time-frequency audio representation; beginners will find the mathematics in Sections 2 and 3 demanding, though the results tables are readable on their own.

Authors’ abstract

Music source separation (MSS) has been conventionally addressed by discriminative methods. Recently, flow matching has emerged as a strong generative paradigm for modeling complex data distributions, with successful applications across diverse domains. These advances motivate us to systematically investigate flow matching for MSS in a four-source (Vocals, Bass, Drums, Other) setting in full-band stereo. Specifically, we investigate prediction targets, flow paths, single-source versus multi-source modeling, and mixture consistency constraints. Beyond the signal-level metrics commonly used for MSS, such as signal-to-distortion ratio, we further evaluate MSS methods using embedding-based metrics, as well as a listening test, providing a comprehensive benchmark. On MUSDB18-HQ, our proposed model exhibits competitive performance in both objective and subjective tests, highlighting the potential of flow matching for generative MSS. Demo: https://ik4sumii.github.io/FLOMSS/