ConsistCompose: Unified Multimodal Layout Control for Image Composition

AI summary

Overview

• Research area: Multimodal image generation, specifically layout-controllable multi-instance image composition within unified multimodal (understanding + generation) models. arXiv category: Computer Vision (cs.CV), arXiv:2511.18333v3.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generation, Transformer-based unified multimodal architectures (MMDiT, MoT), classifier-free guidance, and standard layout-to-image evaluation protocols.
• Scope: A single paper proposing a language-embedded layout control paradigm (LELG), a unified generation framework (ConsistCompose) built on the Bagel backbone, a 3.4M-sample dataset (ConsistCompose3M), and evaluations on layout control, identity preservation, and general multimodal capability.

What This Paper Is About

Unified multimodal models have become good at visual grounding (linking words to image regions), but they still struggle to do the reverse: generate images that obey explicit spatial layout constraints, especially when several distinct, identity-specific instances must be placed at specified positions. Existing layout-controlled methods rely on layout-specific fusion modules, region-aware U-Net edits, or separate layout modalities, which are hard to fold into modern Transformer-based generative frameworks and rarely support multi-reference, identity-preserving composition. ConsistCompose addresses this by writing bounding-box coordinates directly into the text prompt as ordinary tokens, so that spatial control is learned through the same interleaved modeling the model already uses for perception and reasoning.

Key Contributions

1. LELG paradigm: The authors introduce Linguistic-Embedded Layout-Grounded Generation, which encodes instance-level layouts as textual coordinate tokens inside the language stream rather than as a separate layout encoder or geometric branch.
2. ConsistCompose framework: A unified multimodal model built on the MoT architecture of Bagel that consolidates layout-grounded text-to-image generation, multi-reference identity-preserving multi-instance composition, and general multimodal understanding/generation in one interface.
3. ConsistCompose3M dataset: A 3.4M-sample multimodal dataset with layout and identity annotations, split into 2.6M text-guided pairs (derived by reprocessing LayoutSAM) and 0.8M image-guided pairs (built from subject banks and crops from Subjects200K and UNO on the Objects365 taxonomy), filtered with CLIP/DINO-based similarity.
4. Coordinate-aware classifier-free guidance: A Coordinate-CFG mechanism with velocity renormalization that strengthens spatial adherence at sampling time without changing the backbone architecture.

Main Findings

• COCO-Position layout control: ConsistCompose achieves the best result on all reported COCO-Position metrics, with an average Instance Success Ratio of 92.6% (surpassing the second-best model by 4.8%) and an Image Success Ratio of 76.1%. Position Accuracy reaches mIoU 85.3, AP 70.9, AP50 89.1, and AP75 76.9.
• Gains concentrated at high instance counts: The improvement over baselines is described as particularly pronounced at L4–L6 instance levels, where cluttered layouts challenge prior methods.
• Reported headline gains: The abstract and conclusion state a 7.2% gain in layout IoU and a 13.7% AP improvement on COCO-Position relative to layout-controlled baselines.
• MS-Bench multi-subject results: On MS-Bench (1,148 multi-subject combinations of 2–3 subjects from 40 categories), ConsistCompose reaches DINO 0.660, mIoU 0.889, and AP 0.789. On MS-Bench-Random it reaches DINO 0.630, mIoU 0.878, and AP 0.756. Baselines show complementary weaknesses: GLIGEN has strong mIoU (0.868/0.858) but low DINO (0.454/0.431); MUSE improves DINO (0.619/0.607) with moderate mIoU (0.698/0.673); MS-Diffusion has the highest CLIP-T (0.336/0.334) but poor spatial metrics.
• General multimodal capability preserved: MMBench is 81.4 for Ours (w/ Coord) versus 81.4 for the Bagel Base; GenEval is 0.88 versus 0.86; GEdit is 6.31 versus 6.68. MMMU drops from 46.4 (Bagel Base) to 42.3 (w/ Coord) and 39.4 (w/o Coord); the paper describes the MMMU scores of its variants as staying "within a similar range" to the base model.
• DreamBench identity preservation: ConsistCompose achieves the best DINO and CLIP-I in both single- and multi-object settings (Single: DINO 0.677, CLIP-I 0.792, CLIP-T 0.314; Multi: DINO 0.506, CLIP-I 0.703, CLIP-T 0.335), obtained without any DreamBench-specific tuning.
• Coordinate-CFG scale trade-off: On MS-Bench-Random, raising the scale from 0.2 to 0.6 increases DINO from 0.497 to 0.630 and mIoU from 41.7% to 87.8%; going to 1.4 yields only marginal mIoU gains (87.8% to 89.4%) while reducing DINO (0.630 to 0.589). A range of 0.4–0.8 is recommended for multi-reference generation; 1.6 is used for COCO-Position and 0.6 for MS-Bench.
• Ablation on guidance: Table S6 in the supplementary material shows that without Coordinate-CFG, Bagel scores an average Instance Success Ratio of 13.1% and mIoU 23.1 on COCO-Position; adding Coordinate-CFG at SFT raises these substantially as the scale increases from 0.2 to 3.0.

Methodology in Plain English

The authors start from an existing unified multimodal model, Bagel, which uses a Mixture-of-Transformer-Experts design with one expert for understanding and one for generation, plus two visual encoders (a SigLIP-initialized ViT for perception and a FLUX-initialized VAE for generation). Instead of adding a layout module, they attach each object's normalized bounding box directly after the object's name in the prompt, inside a <bbox>[...]</bbox> tag, with each coordinate written to three decimal places. The model then learns to associate the numbers with spatial positions purely through self-attention over the interleaved text and image tokens — no coordinate regression loss is used.

At sampling time, they add Coordinate-CFG: the standard text-and-image classifier-free guidance is extended with a third branch that removes the coordinate tokens. The difference between the coordinate-conditioned and coordinate-unconditioned velocity predictions is scaled by a factor and added back, then the whole velocity is renormalized to match the magnitude of the standard text-guided prediction.

Training is two-stage. Stage 1 ("LELG alignment") runs 18K steps with a cosine schedule starting at 2×10⁻⁵, mixing general understanding data (FineVision, MAmmoTH) with ConsistCompose3M and high-resolution (>512 px) images mined from COCO2017 and Objects365. Stage 2 ("hybrid SFT") runs 6K steps at a constant 2.0×10⁻⁵ learning rate with a CE:MSE loss weight of 0.25:1, mixing text-to-image, multi-reference-to-image, LELG-conditioned variants, image editing, and understanding data, with layout-conditioned tasks upsampled. Both stages use AdamW (β1=0.9, β2=0.95, ε=1.0×10⁻¹⁵), gradient clipping of 1.0, a diffusion timestep shift of 4.0, and 64 GPUs.

Why This Matters

• Impact on research: The paper argues that layout control does not require a dedicated architecture — embedding it in language keeps the model unified, avoids task-specific branches, and preserves general multimodal ability. If this holds, it removes a long-standing split between layout-specific pipelines and general unified multimodal models, and gives the field a large-scale (3.4M) supervision resource for layout-and-identity-conditioned generation.
• Real-world applications:
  • Advertising and marketing creative generation, where products or people must appear at prescribed positions.
  • E-commerce catalog imagery, rearranging the same product subject into new layouts while preserving its identity.
  • Storyboarding and illustration, where a creator specifies a scene composition and expects each element to land where drawn.
  • Photo editing and retouching tools, supporting content-preserving rearrangement of elements already present in an image.
• Industry relevance: The work comes from SenseTime Research and is built on top of the publicly described Bagel backbone, suggesting the approach is intended as a drop-in extension to existing unified generation stacks rather than a bespoke pipeline. The dataset contribution also lowers the barrier for other labs to train comparable layout-aware models.

Future Directions

• Finer-grained spatial control: The paper explicitly names finer-grained spatial reasoning and part-aware manipulation as extensions, since the current formulation is limited to bounding-box control.
• Interactive layout-guided composition: The conclusion points toward interactive workflows where layouts are specified and adjusted by a user during scene composition.
• Closing the MMMU gap: The reported MMMU scores (42.3 with coordinates, 39.4 without, versus 46.4 for the Bagel Base) leave an open question about how to inject layout supervision without any loss of cross-domain reasoning.
• Generalizing beyond bounding boxes: Because LELG is described as "an extensible abstraction for injecting spatial structure," a natural next step is testing it with other spatial representations inside the same language interface.

Target Audience

Researchers and engineers working on controllable image generation, layout-to-image synthesis, and multi-instance/identity-preserving composition; practitioners building unified multimodal models who want spatial control without adding layout-specific modules; and teams interested in large-scale supervised datasets for layout-and-identity-conditioned generation. Readers need working knowledge of diffusion or flow-matching generation and Transformer-based multimodal architectures to follow the method section, though the core idea — putting coordinates in the prompt — is stated in plain terms.

Authors’ abstract

Unified multimodal models that couple visual understanding with image generation have advanced rapidly, yet most systems still focus on visual grounding-aligning language with image regions-while their generative counterpart, linguistic-embedded layout-grounded generation (LELG) for layout-controllable multi-instance generation, remains underexplored and limits precise compositional control. We present ConsistCompose, a unified multimodal framework that embeds layout coordinates directly into language prompts, enabling layout-controlled multi-instance image generation from Interleaved Image-Text within a single generative interface. We further construct ConsistCompose3M, a 3.4M multi-instance generation dataset with layout and identity annotations (2.6M text-guided and 0.8M image-guided data pairs) that provides large-scale supervision for layout-conditioned generation. Within this framework, LELG is instantiated through instance-coordinate binding prompts and coordinate-aware classifier-free guidance, which translate linguistic layout cues into precise spatial control without task-specific branches. Experiments on COCO-Position and MS-Bench show that ConsistCompose substantially improves spatial accuracy over layout-controlled baselines while preserving identity fidelity and competitive general multimodal understanding, establishing a unified paradigm for layout-controllable multimodal image generation.