FlowBind: Efficient Any-to-Any Generation with Bidirectional Flows

AI summary

Overview

• Research area: Multi-modal generative modeling — specifically "any-to-any" generation, where a single model translates between arbitrary subsets of modalities (text, image, audio, and, in an extension, 3D point clouds).
• Technical level: Intermediate. The paper assumes familiarity with flow matching, diffusion-style generative models, and multi-modal alignment metrics, though its central idea (a shared latent plus per-modality invertible flows) is conceptually simple.
• Scope: The paper proposes FlowBind, a flow-matching framework that replaces a fixed Gaussian prior with a learnable shared latent connected to each modality by its own invertible flow, and demonstrates on text, image and audio that it reaches competitive quality with up to 6× fewer parameters and 10× faster training than prior flow-based any-to-any methods.

What This Paper Is About

Any-to-any generation aims to translate between arbitrary subsets of modalities, so a single model can, for example, produce audio from an image or text from audio. Existing flow-based generalists are inefficient: they need large datasets with restrictive pairing constraints (CoDi anchors everything to text; OmniFlow relies heavily on fully paired data), they model expensive joint distributions, and they rely on complex multi-stage training pipelines. FlowBind's goal is to make any-to-any generation simple, data-flexible and cheap by factorizing multi-modal interaction through a learnable shared latent space.

Key Contributions

1. A factored flow-based framework for any-to-any generation. FlowBind replaces the fixed Gaussian prior with a learnable shared latent that captures cross-modal commonality, and connects each modality to that latent through its own invertible, modality-specific flow. This factorization lets the model train on arbitrarily paired data (not just fully paired or text-anchored data) with a low compute budget.

2. Single-stage joint training under one flow-matching loss. Both the shared latent (produced by an auxiliary encoder) and all modality-specific drift networks are optimized jointly with a single flow-matching objective, avoiding the multi-stage pipelines of CoDi (which separates modality alignment from joint generation) and OmniFlow (which requires a distinct post-training phase after merging components).

3. A gradient-stopping scheme that prevents collapse without extra loss terms. The paper shows analytically and empirically that stopping gradients through the auxiliary encoder for t ∈ (0, 1] while updating it directly at t = 0 is sufficient to prevent degenerate solutions, where prior direct-flow work used contrastive or other explicit regularizers.

4. Efficiency and generality demonstrated across modalities and tasks. Experiments on text, image and audio cover all six one-to-one tasks, plus many-to-one and one-to-many settings, with an extension to 3D point clouds showing that adding a modality requires only one additional drift network.

Main Findings

• Compute and parameters: FlowBind uses 568M trainable parameters and 48 GPU-hours, versus OmniFlow's 3.2B parameters and 480hr (reported only for its final joint training stage) and CoDi's 4.3B parameters (training time omitted due to absence of training code and details). The abstract states up to 6× fewer parameters and 10× faster training than prior methods.

• Data efficiency: FlowBind trains on 310K text–image samples, 96K text–audio samples and 180K image–audio samples, compared with CoDi's 400M text–image, 3.5M text–audio and 1.9M image–audio samples, and OmniFlow's 28M text–image, 2.4M text–audio and 2.2M text–audio–image samples. The paper states this is 0.15% of CoDi and 1.79% of OmniFlow's data.

• Fidelity, one-to-one generation (Table 2): FlowBind records FID 17.39 for text→image, CIDEr 46.26 for image→text, FAD 4.19 for text→audio, CIDEr 55.11 for audio→text, FAD 2.50 for image→audio, and FID 26.60 for audio→image. For comparison, CoDi records 24.80 / 16.40 / 9.84 / 6.62 / 14.58 / 50.4 and OmniFlow records 22.97 / 44.20 / 4.20 / 31.79 / 5.67 / 106.03. FlowBind achieves the best quality metric in all six tasks.

• Alignment, one-to-one generation (Table 3): FlowBind scores CLIP 28.35 (text→image), CLIP 29.74 (image→text), CLAP 29.08 (text→audio), CLAP 36.70 (audio→text), AIS 82.89 (image→audio) and AIS 78.17 (audio→image). It shows superior alignment on four of six tasks; OmniFlow leads on text→image CLIP (31.52 vs. FlowBind's 28.35), which the authors attribute partly to OmniFlow being initialized from SD3-Medium.

• Image–audio strength: FlowBind outperforms generalists and even dedicated specialists on image–audio generation (FAD 2.50 for image→audio vs. Seeing & Hearing's 5.31; AIS 82.89 vs. Seeing & Hearing's 75.11; FID 26.60 and AIS 78.17 for audio→image vs. Sound2Vision's 42.55 and 62.39). The authors attribute this to learning a shared latent directly from audio–image pairs rather than through text.

• Comparison with an LLM-based generalist: Against UnifiedIO2-L, FlowBind is described as comparable on text–image tasks and stronger in other settings.

• Many-to-one (Table 4): FlowBind reaches CLIP 27.83 (image→text) and CLAP 35.21 (audio→text) for (image+audio)→text, CLIP 25.57 and AIS 57.93 for (text+audio)→image, and CLAP 28.13 and AIS 76.02 for (text+image)→audio. The authors highlight that CoDi and OmniFlow tend to disregard the text input in (text+image)→audio (CLAP 4.85 and 7.68 respectively), while FlowBind incorporates both.

• One-to-many (Table 5): FlowBind scores CLIP 25.02 and CLAP 29.12 for text→(image+audio), CLIP 27.98 and AIS 74.34 for image→(text+audio), and CLAP 36.79 and AIS 59.99 for audio→(text+image).

• Fixed anchor vs. learnable anchor (Table 6): A data-controlled comparison shows a FlowBind variant trained without the image–audio pair scores image→text 30.04, audio→text 37.04 and image→audio 61.88, outperforming a text-anchoring baseline at 27.94, 36.72 and 55.48 respectively.

• Shared latent alignment (Table 7): Measured with the CKNNA metric using at most 1024 samples and neighborhood size k = 10, the shared latent scores 0.2872 (text–audio) and 0.3026 (audio–image), versus 0.1965 and 0.1343 for modality-specific latents.

• Theoretical result: At t = 0, the encoder's effective objective reduces to E[Var(z^i | z*)] — the "unexplained variance" — so the encoder is driven to retain predictive information about each modality. Proposition 1 extends this by decomposing the loss at t = 0 into unexplained variance plus the approximation error of the drift networks.

• Scaling to new modalities: Extending to 3D point clouds via Pix3D requires only an additional modality-specific drift network, so parameter count grows roughly linearly with the number of modalities N. Trained with only an additional paired image–point-cloud dataset, the model generalizes to unseen tasks such as text→point cloud and point cloud→text.

Methodology in Plain English

FlowBind starts from a simple observation: if every modality can be mapped into and out of one common space, then translating between any two modalities is just a trip out and back through that space.

During training, a small auxiliary encoder takes whatever modalities are present in a given sample and produces a single shared latent vector. For each modality present, a separate "drift" network learns to move data from that modality to the shared latent along a straight interpolated path, using the standard flow-matching objective (predict the displacement between the two endpoints). The encoder and all drift networks are trained together in one stage with one loss.

The catch is that this objective admits a trivial solution: the encoder could collapse to a constant (such as zero), and the drifts could still fit perfectly with no meaningful shared representation. FlowBind avoids this with a gradient-stopping trick: for all time steps except t = 0, gradients do not flow back into the encoder, so the drifts train stably; at t = 0 the encoder is updated directly alongside the drifts. The authors show that this single time step gives the encoder a well-defined objective — minimize how much of each modality's variance remains unexplained by the shared latent. Training alternates between these regimes by sampling t from a mixture of a uniform distribution on (0,1) and a point mass at t = 0.

At inference, the auxiliary encoder is discarded. Because the flows are invertible, a backward integration from a source modality yields the shared latent, and a forward integration from that latent yields the target. With multiple source modalities, the latent estimates from each are simply averaged before generating the target.

FlowBind operates in compact, semantic representation spaces rather than raw high-dimensional data: it uses frozen encoders — EmbeddingGemma for text, CLIP for images (with Stable-UnCLIP as the decoder) and CLAP for audio — and the drift networks and auxiliary encoders are MLP-based with residual connections and AdaLN-zero time modulation. Because the frozen encoders and decoders handle within-modality generation, the drifts only need to learn cross-modality correspondence.

Why This Matters

Impact on research. The paper reframes any-to-any generation as learning a shared latent anchor rather than anchoring modalities to text (as CoDi does) or modeling a joint velocity field across all modalities (as OmniFlow does). If the factorization holds up, the cost of adding a new modality becomes roughly one additional drift network instead of a quadratically growing joint model, and the whole system trains in a single stage under one loss instead of several. The paper also contributes a small analytical result — the t = 0 loss decomposition into unexplained variance and drift approximation error — that may be reusable in other direct-flow or encoder-coupled generative settings.

Real-world applications (implied by the modalities the paper evaluates):

• Cross-modal retrieval and search, such as finding or generating audio to match an image, or retrieving images from audio cues, where the reported image–audio gains are largest.
• Accessible content production, converting between text, images and audio to generate audio descriptions, captions or spoken summaries without separate specialist pipelines.
• Creative and media tooling that conditions on several inputs at once, such as generating sound design from a text prompt plus a reference image.
• 3D asset and scene workflows, given the demonstrated extension to point clouds and generalization to unseen text↔point-cloud tasks.

Industry relevance. The headline numbers — 568M parameters, 48 GPU-hours, and 310K/96K/180K training pairs versus 400M/3.5M/1.9M for CoDi and 28M/2.4M/2.2M for OmniFlow — speak directly to practitioners who cannot assemble large fully paired multi-modal datasets or pay for multi-stage training. Fewer parameters and a single training stage also reduce operational complexity, avoiding the brittle staged pipelines the authors criticize.

Future Directions

• Scaling the modality set. The paper demonstrates an extension only to one additional modality (3D point clouds on Pix3D). How well the linear parameter growth claim holds at many more modalities, and whether the shared latent becomes a bottleneck, is not established.
• Beyond averaging for multi-source conditioning. Multi-source inference aggregates latent estimates by simple averaging, which the authors analyze in Appendix E. Whether learned or weighted aggregation improves many-to-one results is left open.
• Systematic study of partial pairing. The method is motivated by training on arbitrarily paired data, but the reported data configurations are specific (310K text–image, 96K text–audio, 180K image–audio). How performance varies with different pairing patterns and dataset sizes is not reported.
• Standardized many-to-many benchmarking. The authors note that no standard benchmark exists for many-to-many generation and construct their own synthetic triplet dataset by extending AudioCaps text–audio pairs with FLUX.1-schnell images. Broader, community-agreed evaluation protocols would make comparisons with future methods more reliable.

Target Audience

Researchers and graduate students working on multi-modal generative models, flow matching, or diffusion-based cross-modal synthesis; practitioners who need any-to-any translation under tight compute and data budgets; and engineers evaluating whether a factored shared-latent design can replace text-anchored or fully paired joint-modeling pipelines in production systems. Readers unfamiliar with flow matching or ODE-based generation will need background reading, but the paper's core idea and efficiency claims are accessible at an intermediate level.

Authors’ abstract

Any-to-any generation seeks to translate between arbitrary subsets of modalities, enabling flexible cross-modal synthesis. Despite recent success, existing flow-based approaches are challenged by their inefficiency, as they require large-scale datasets often with restrictive pairing constraints, incur high computational cost from modeling joint distribution, and rely on complex multi-stage training. We propose FlowBind, an efficient framework for any-to-any generation. Our approach is distinguished by its simplicity: it learns a shared latent space capturing cross-modal information, with modality-specific invertible flows bridging this latent to each modality. Both components are optimized jointly under a single flow-matching objective, and at inference the invertible flows act as encoders and decoders for direct translation across modalities. By factorizing interactions through the shared latent, FlowBind naturally leverages arbitrary subsets of modalities for training, and achieves competitive generation quality while substantially reducing data requirements and computational cost. Experiments on text, image, and audio demonstrate that FlowBind attains comparable quality while requiring up to 6x fewer parameters and training 10x faster than prior methods. The project page with code is available at https://yeonwoo378.github.io/official_flowbind.