SplitFlow: Flow Decomposition for Inversion-Free Text-to-Image Editing

AI summary

Overview

Research area: Computer Vision — text-to-image editing with rectified flow generative models.

Technical level: Advanced. The paper assumes familiarity with flow matching, rectified flow ODEs, diffusion inversion, classifier-free guidance, and multi-task gradient-conflict methods.

Scope: The paper introduces SplitFlow, an inversion-free text-to-image editing framework that decomposes a target prompt into sub-prompts, computes an independent flow for each, and recombines them into a single editing trajectory.

What This Paper Is About

Text-to-image editing systems must modify a real image to match a new text prompt while keeping everything else unchanged. Existing rectified-flow editing pipelines struggle because inverting an image back to noise is inaccurate, and when a target prompt contains several attributes at once, the gradients guiding the edit become entangled and pull in conflicting directions. SplitFlow attacks both problems by splitting the editing task into several smaller, independently controlled flows and then merging them into one coherent trajectory.

Key Contributions

1. Flow decomposition improves editing fidelity. The authors demonstrate that splitting the editing flow into sub-target flows improves the fidelity of the edited image, particularly in preserving background consistency.

2. SplitFlow, a decompose-then-aggregate framework. They propose SplitFlow, which progressively approximates the target latent through flow decomposition followed by flow composition, rather than mapping source to target in a single step.

3. Two aggregation mechanisms: LTP and VFA. They introduce Latent Trajectory Projection (LTP), which aligns sub-target latents with the full target direction, and Velocity Field Aggregation (VFA), which adaptively weights sub-target velocity fields to suppress redundancy and emphasize distinct semantic directions.

4. A mathematical justification for VFA. They prove that the weighted aggregation produces a vector whose inner product with the simple average is at least the squared norm of that average, using Gibbs' inequality and Jensen's inequality.

Main Findings

• Outperforms prior zero-shot editing methods on PIE-Bench. On the SD3 backbone, SplitFlow reaches Structure Distance 25.96, PSNR 22.45, LPIPS 102.14, MSE 81.99, SSIM 83.91, CLIP Whole 26.96, and CLIP Edited 23.83 — beating FlowEdit on SD3 (27.24, 22.13, 105.46, 87.34, 83.48, 26.83, 23.67) and iRFDS on SD3.

• The fidelity-enhanced variant beats a stronger backbone. SplitFlow† on SD3 (Structure Distance 14.55, LPIPS 68.53, CLIP Whole 26.23, CLIP Edited 23.01) outperforms FTEdit on SD3.5 (18.17, 80.55, 25.74, 22.27) in Structure Distance, LPIPS, and both CLIP metrics, even though FTEdit uses a stronger model.

• Best overall result comes from SD3.5. SplitFlow on SD3.5 records Structure Distance 11.68, PSNR 27.12, LPIPS 52.93, MSE 30.61, SSIM 89.76, CLIP Whole 26.29, and CLIP Edited 22.89, edging out FlowEdit on SD3.5 (11.80, 26.97, 53.68, 31.23, 89.70, 26.18, 22.88) across every reported metric.

• Naive averaging already helps fidelity. The ablation shows that averaging sub-flows (AVG) keeps CLIP similarity on par with the baseline (26.81 Whole, 23.67 Edited) while strongly improving background preservation (PSNR 23.36 vs. 22.13, LPIPS 92.00 vs. 105.46, MSE 68.26 vs. 87.34, SSIM 85.00 vs. 83.48, Structure Distance 22.28 vs. 27.24).

• LTP trades background fidelity for prompt alignment. Adding LTP alone yields CLIP Whole 26.93 and CLIP Edited 23.82 with Structure Distance 26.22 — higher CLIP than averaging, but lower background preservation than AVG, while still beating the baseline on all metrics.

• LTP plus VFA gives the best balance. Combining both components reaches Structure Distance 25.96, PSNR 22.45, LPIPS 102.14, MSE 81.99, SSIM 83.91, CLIP Whole 26.96, and CLIP Edited 23.83.

• Aggregation timestep matters. Testing η_dec of 30, 29, 28, 27, and 26 shows all configurations beat the baseline; decreasing η_dec (a longer decomposition period) improves editability at the cost of background preservation. The authors choose η_dec = 28.

• Cost trade-off is explicit. SplitFlow needs roughly 3 × (33 − 28) + 33 = 48 inference steps. FlowEdit takes 57 minutes for inference on PIE-Bench, whereas SplitFlow takes 83 minutes for 700 images, plus about 20 minutes for LLM prompt decomposition.

• Qualitative gains on hard edits. SplitFlow successfully adds a hat to a Mozart portrait where other methods fail or distort the image, and converts "three giraffes" to "three elephants" with a tiger replacing another animal while preserving the phrase "in a line," which FlowEdit does not.

Methodology in Plain English

The starting point is an existing inversion-free editing scheme: instead of recovering noise that would reconstruct the original image, the method drives the image through a path in clean-image space using the difference between the model's velocity predictions for the target prompt and the source prompt. SplitFlow keeps this inversion-free formulation but changes how the target prompt is handled.

First, an LLM (Mistral-7B) is given an instruction prompt, the source prompt, and the target prompt, and returns a short sequence of sub-prompts, each capturing one semantic piece of the change. In most cases the number of sub-prompts is at most three. For example, turning "A german shepherd dog stands on the grass with mouth closed" into "A german shepherd dog with black sunglasses jumping on the grass with mouth opened" produces sub-prompts such as "A black sunglasses dog jumps grass," "A dog with open mouth jump grass," and "A german shepherd with black sunglasses."

Each sub-prompt gets its own editing flow with its own relative velocity field, so the attributes are handled separately instead of all at once. This decomposition runs from timestep η_max down to η_dec, spanning 5 steps in the final configuration.

At η_dec, the separate flows are recombined. Latent Trajectory Projection normalizes the full-target latent and projects every sub-target latent onto that direction by taking an inner product along the channel dimension, then averages the projected latents. Velocity Field Aggregation then compares the sub-target velocity fields by cosine similarity, converts those similarities into softmax weights per spatial location, and blends the velocity fields with those weights. The updated latent is the projected latent plus the aggregated velocity times the timestep increment. The intuition, borrowed from gradient-conflict resolution in multi-task learning, is that redundant, aligned directions get damped while distinct directions get emphasized.

Experiments use Stable Diffusion SD3 and SD3.5 rectified flow models, with T = 50 steps, η_max = 33 (skipping the first third of steps), classifier-free guidance values of 3.5 for the source and 13.5 for the target, and η_dec = 28. Evaluation is on PIE-Bench (700 images spanning ten categories), using LPIPS, SSIM, MSE, PSNR, Structure Distance, and CLIP similarity, compared against RF-Inversion, RF-Solver, FireFlow, iRFDS, FTEdit, FlowEdit, and DDIM-based diffusion baselines.

Why This Matters

Impact on research. The paper shows that semantic decomposition of a prompt is itself a useful lever for improving editing, independent of which aggregation scheme is used — even naive averaging beat the baseline on background preservation. Embedding flow aggregation in the style of multi-task gradient conflict resolution is described by the authors as the first application of flow decomposition and aggregation to a text-based image editing framework. The accompanying proof that adaptive weighting dominates uniform averaging gives future work a formal handle on why such schemes help.

Real-world applications:

• Photo editing tools that let users change clothing, objects, or scene attributes through natural-language prompts while leaving the rest of the photograph untouched.
• E-commerce and product imagery, where catalog photos need consistent background preservation while attributes such as color or accessories change.
• Content creation and advertising, where complex multi-attribute prompts (subject, action, and setting at once) must be rendered faithfully from a reference image.
• Creative iteration in design workflows, where diversity of plausible edits matters alongside prompt alignment.

Industry relevance. The method builds on Stable Diffusion 3 and 3.5 rectified flow backbones, which are widely used in production image pipelines. The measured cost — 83 minutes for 700 images plus about 20 minutes of LLM decomposition — is slower than FlowEdit's 57 minutes, so deployment would require weighing the quality gains against latency. The reliance on an external LLM for prompt splitting also adds a dependency that industry adopters would need to manage.

Future Directions

• Better prompt decomposition. The authors identify dependence on the LLM's decomposition as the main limitation and call for more effective decomposition techniques using LLMs or vision-language models.
• Optimization-based conflict resolution. They suggest exploring optimization-based approaches to mitigate gradient conflicts during flow composition, beyond the current projection and weighting scheme.
• Reducing inference cost. SplitFlow's higher runtime than FlowEdit is acknowledged; narrowing that gap while retaining quality is an open engineering question.
• Extreme-case behavior. The paper states that additional discussion of extreme cases appears in the supplementary material, implying robustness boundaries of the decomposition-aggregation scheme are not yet fully characterized in the main text.

Target Audience

Researchers and graduate students working on diffusion and rectified-flow generative models, particularly those focused on text-guided image editing, inversion-free methods, and controllability. It also suits practitioners who build editing pipelines on Stable Diffusion 3/3.5 backbones and need to understand the fidelity-versus-editability trade-off, as well as readers interested in transferring multi-task learning ideas such as gradient conflict resolution into generative modeling.

Authors’ abstract

Rectified flow models have become a de facto standard in image generation due to their stable sampling trajectories and high-fidelity outputs. Despite their strong generative capabilities, they face critical limitations in image editing tasks: inaccurate inversion processes for mapping real images back into the latent space, and gradient entanglement issues during editing often result in outputs that do not faithfully reflect the target prompt. Recent efforts have attempted to directly map source and target distributions via ODE-based approaches without inversion; however,these methods still yield suboptimal editing quality. In this work, we propose a flow decomposition-and-aggregation framework built upon an inversion-free formulation to address these limitations. Specifically, we semantically decompose the target prompt into multiple sub-prompts, compute an independent flow for each, and aggregate them to form a unified editing trajectory. While we empirically observe that decomposing the original flow enhances diversity in the target space, generating semantically aligned outputs still requires consistent guidance toward the full target prompt. To this end, we design a projection and soft-aggregation mechanism for flow, inspired by gradient conflict resolution in multi-task learning. This approach adaptively weights the sub-target velocity fields, suppressing semantic redundancy while emphasizing distinct directions, thereby preserving both diversity and consistency in the final edited output. Experimental results demonstrate that our method outperforms existing zero-shot editing approaches in terms of semantic fidelity and attribute disentanglement. The code is available at https://github.com/Harvard-AI-and-Robotics-Lab/SplitFlow.