DOS: Directional Object Separation in Text Embeddings for Multi-Object Image Generation

AI summary

Overview

Research area: Computer vision, specifically text-to-image (T2I) generative modeling, CLIP text embedding analysis, and multi-object image generation.

Technical level: Intermediate. The paper assumes familiarity with CLIP text encoders, diffusion model conditioning, and embedding arithmetic, but its core idea is described in accessible geometric terms.

Scope: The paper identifies four failure-prone scenarios in multi-object T2I generation and proposes a text-embedding-only correction method that improves success rates across four benchmarks on two base models without altering the image generation process.

What This Paper Is About

Text-to-image models produce high-quality images from prompts but frequently fail when a prompt names multiple objects, either dropping an object entirely (object neglect) or blending two objects into one (object mixing). The authors trace these failures to information mix-ups inside CLIP text embeddings and to the fact that there are no explicit directional signals telling the model how to pull one object's representation away from another's. Their goal is to compute such directional signals and add them to the CLIP embeddings before generation, so that the T2I model separates objects more reliably without any change to the diffusion process itself.

Key Contributions

1. Identification of four problematic scenarios. Through extensive experiments with state-of-the-art T2I models, the authors isolate Similar Shapes, Similar Textures, Dissimilar Background Biases, and Many Objects as settings where inter-object relationships consistently intensify object neglect and object mixing, even when no descriptive attributes are present in the prompt.

2. A preliminary quantitative analysis of these scenarios. Contrasting prompt sets (20 prompts per condition, 10 random seeds, 200 images per condition) show success-rate gaps of 26.5% to 60.5% between the two contrasting conditions of each aspect, confirming that these are the dominant failure modes.

3. The DOS method. DOS constructs a separation vector for every object pair and every CLIP embedding type — semantic token embeddings of object nouns, EOT embeddings, and pooled embeddings — then aggregates them by a difficulty-weighted average into DOS vectors that are added to the original embeddings.

4. An adaptive strength mechanism. Each separation vector is scaled by a strength derived from shape/texture similarity and background-bias dissimilarity, computed from cosine-similarity profiles over 42 representative shape/texture words and 36 background phrases, with offsets precomputed over object pairs from MS-COCO.

Main Findings

• Four failure-prone scenarios are confirmed quantitatively. The four identified scenarios produce success-rate differences of 26.5% to 60.5% between contrasting conditions, with "lower success rate" indicating more frequent object neglect or object mixing.

• DOS achieves the highest success rate (SR) and lowest mixture rate (MR) on every benchmark and both base models. With SDXL as the base model, DOS reaches 64.00% SR / 3.50% MR on Similar Shapes, 71.50% / 3.50% on Similar Textures, 68.50% / 17.00% on Dissimilar Background Biases, and 48.00% / 15.50% on Many Objects.

• DOS outperforms the other text-embedding modification method. It beats TEBOpt by 10.50%–24.00% in SR across the benchmarks.

• DOS also improves the newer SD3.5 model. From an SD3.5 baseline of 75.50% / 4.00%, 79.00% / 6.00%, 78.00% / 17.50%, and 70.00% / 16.50%, DOS raises results to 81.00% / 3.00%, 87.50% / 3.00%, 85.50% / 13.50%, and 76.50% / 10.50%.

• Human evaluation strongly favors DOS. Across four benchmarks, DOS received 26.24%–43.04% more votes than the second-best result, with its own vote shares at 50.52%, 43.12%, 53.44%, and 55.18% respectively.

• Inference overhead is minimal compared to latent modification methods. DOS runs in 13.87s versus 12.97s for the SDXL baseline, while A&E and CONFORM take 58.83s and 58.48s — roughly 4-5 times the baseline, and about four times slower than DOS.

• All three embedding types matter. Applying DOS only to semantic token embeddings or only to EOT/pooled embeddings each improves SR, but MR drops most sharply with EOT/pooled updates, and modifying all three types gives the best overall SR and MR.

• Adaptive strengths outperform a fixed strength. Setting all strengths to a fixed value of 0.5 already improves substantially over the baseline, but adaptive strengths yield further gains — for example, 48.0% SR on Many Objects versus 41.0% with the fixed value.

• Adaptive strengths are not always optimal. In rare cases shown in Figure 6, the adaptive strength fails to scale separation vectors properly, causing object neglect; the authors compare against an optimal strength found by grid search over 121 settings (an 11×11 grid with a 0.1 interval).

Methodology in Plain English

The authors leave the image generator untouched and only edit the text embeddings that condition it. For a prompt with N object nouns, they work on three embedding types: the semantic token embeddings of the object nouns, the end-of-text (EOT) embedding, and the pooled embedding.

For each pair of objects, they build a separation vector that points from one object toward the other. How they build it depends on the embedding type. For semantic token embeddings, they use "pure" prompts of the form "a {object}" for each object separately and subtract the two embeddings — this avoids contamination from other tokens. For EOT and pooled embeddings, they deliberately exploit the causal-masking information mix-ups: they subtract the embedding of "a {object A} mixed with a {object B}" from that of "a {object A} separated from a {object B}", so the difference encodes the direction of separation.

Not all object pairs are equally hard. To reflect this, each pair's separation vector is scaled by an adaptive strength. The authors compare how strongly an object aligns with 42 representative shape/texture words and 36 background phrases, using cosine similarity, then compute Pearson correlations between the similarity profiles of the two objects. Pairs whose shape/texture profiles correlate highly, or whose background profiles correlate weakly, receive stronger separation. A shifted tempered sigmoid with temperature T = 0.6 (fixed across base models) maps these correlations to strengths, with offsets precomputed over MS-COCO object pairs.

Finally, the scaled separation vectors for all partners of a given object are averaged (weighted by N-1), forming the DOS vector. DOS vectors are added to the corresponding semantic token embeddings, while for EOT and pooled embeddings the DOS vectors are summed across all objects to counter compounded inter-object effects. Evaluation uses a VLM-based metric adapted from EnMMDiT: GPT-4o-mini classifies each prompt object in the generated image as fully intact, mixed, or absent, yielding success rate (all objects intact) and mixture rate (one or more objects mixed). Experiments ran on a single NVIDIA RTX 3090 GPU, with a guidance scale of 5.0 and 50 denoising steps for SDXL, and 7.0 with 28 steps for SD3.5.

Why This Matters

Impact on research. The work reframes multi-object generation failure as a property of the text embedding rather than the diffusion latent, showing that directional information extracted from CLIP embedding differences can be injected cheaply. It also positions text-embedding modification as a faster alternative to the latent-modification line of work (Attend-and-Excite, CONFORM), which requires iterative gradient updates and can introduce artifacts from departing the trained latent distribution.

Potential real-world applications (the paper itself does not enumerate applications; these follow from the method's behavior):

• Creative and design tooling where users prompt for several distinct items in a single scene, such as mood boards or product concept sketches.
• E-commerce and catalog imagery, where a prompt must render multiple named products side by side without blending them.
• Educational and scientific illustration, where prompts list several organisms, materials, or parts that must each appear distinctly.
• Consumer image generation apps that run on limited hardware, since DOS avoids the multi-fold latency cost of latent modification methods.

Industry relevance. DOS is model-agnostic in the sense that it only edits CLIP text embeddings, so it can be layered onto existing pipelines without retraining — demonstrated on both SDXL and the newer SD3.5. The near-baseline inference time (13.87s vs. 12.97s on SDXL) matters for production deployments where per-image latency is billed.

Future Directions

• Better strength estimation. The authors' own limitation section notes that adaptive strengths do not always yield optimal scaling, and suggest incorporating criteria beyond shape, texture, and background bias.
• Extending beyond the four scenarios. The paper defines its benchmarks around Similar Shapes, Similar Textures, Dissimilar Background Biases, and Many Objects; whether other inter-object relationships produce distinct failure modes is not reported.
• Attribute binding. The authors deliberately chose prompts that primarily contain objects and avoid descriptive attributes, leaving open how DOS interacts with prompts that combine multiple objects and their attributes.
• Generalization across architectures. DOS was validated on SDXL and SD3.5, with additional SD3.5 comparisons against TEBOpt and Self-Cross relegated to the appendix; broader evaluation across other T2I architectures and text encoders is not reported in the main content.

Target Audience

Researchers and engineers working on text-to-image generation, particularly those interested in prompt-embedding manipulation, CLIP representation analysis, and compositional or multi-object generation. It is also relevant to practitioners deploying T2I models who need a low-latency fix for object neglect and object mixing, and to readers interested in how properties of a shared embedding space can be exploited to patch downstream model limitations.

Authors’ abstract

Recent progress in text-to-image (T2I) generative models has led to significant improvements in generating high-quality images aligned with text prompts. However, these models still struggle with prompts involving multiple objects, often resulting in object neglect or object mixing. Through extensive studies, we identify four problematic scenarios, Similar Shapes, Similar Textures, Dissimilar Background Biases, and Many Objects, where inter-object relationships frequently lead to such failures. Motivated by two key observations about CLIP embeddings, we propose DOS (Directional Object Separation), a method that modifies three types of CLIP text embeddings before passing them into text-to-image models. Experimental results show that DOS consistently improves the success rate of multi-object image generation and reduces object mixing. In human evaluations, DOS significantly outperforms four competing methods, receiving 26.24%-43.04% more votes across four benchmarks. These results highlight DOS as a practical and effective solution for improving multi-object image generation.