CanvasComposer: Personalized Group Photo Generation via a Multi-Reference Canvas

AI summary

Overview

Research area: Computer vision, specifically personalized text-to-image (T2I) generation and multi-human image composition using diffusion transformer models.

Technical level: Intermediate to Advanced. The paper assumes familiarity with diffusion models, VAE latents, LoRA finetuning, attention mechanisms, and Rotary Position Embeddings, though the core idea (a Photoshop-like canvas of separate reference people) is easy to grasp.

Scope in one sentence: The paper introduces CanvasComposer, a framework that lets users place multiple RGBA reference cutouts of people onto a shared canvas and generates a single coherent, identity-preserving group photo from that canvas plus a text prompt.

What This Paper Is About

Personalized image generators work reasonably well for a single identity, but they break down when several people must appear together: the model has to figure out which reference corresponds to which generated person, so identities get swapped, mixed, duplicated, or dropped, and the computational cost grows with each added subject. CanvasComposer reframes the problem as an interactive composition task — the user arranges separate reference subjects on a canvas, and the model harmonizes them into one coherent group photo that follows a text prompt.

Key Contributions

1. An interactive personalization paradigm in which users place multi-human references — full-body, portrait, or cropped-head — on a shared canvas, with each identity individually adjustable by direct canvas manipulation.
2. The multi-reference canvas, an input representation that keeps each subject as its own RGBA cutout rather than a single flattened collage. It preserves reference content when subjects overlap, and pairs with transparent latent pruning, which retains only tokens from non-transparent regions so conditioning length depends on non-transparent content rather than on the number of subjects.
3. Subject-wise cross-reference training, a data sampling strategy that sources each subject from a different image within the same identity group than the target image, with geometric and lighting augmentations, to prevent pixel-level input-target correspondence and mitigate copy-paste artifacts.
4. The CanvasComposer framework itself, built on pretrained FLUX Kontext, with experiments reported on multi-human personalization benchmarks showing state-of-the-art identity preservation and generation quality.

Main Findings

• Four-person identity preservation: On the 4P benchmark, CanvasComposer reaches an ArcFace score of 0.533, versus 0.309 for UniPortrait, 0.086 for OmniGen2, 0.082 for ID-Patch, 0.077 for UNO, 0.251 for Overlay Kontext, 0.223 for Qwen-Image-Edit, 0.217 for FLUX Kontext, and 0.434 for Nano-Banana.
• Two-person identity preservation: On the 2P benchmark, CanvasComposer scores 0.547 ArcFace, compared with 0.542 for StoryMaker, 0.460 for UniPortrait, 0.443 for DreamO, 0.121 for ID-Patch, 0.121 for OmniGen2, and 0.072 for UNO.
• Aesthetic quality and prompt alignment remain competitive but are not always best: CanvasComposer records HPSv3 of 12.5 and VQAScore of 0.840 in 4P (against 13.0 HPSv3 for both OmniGen2 and Qwen-Image-Edit, and 0.895 VQAScore for Qwen-Image-Edit), and 11.6 HPSv3 / 0.865 VQAScore in 2P.
• Strong user preference: In the 4P setting participants favored CanvasComposer over all state-of-the-art 4P personalization baselines in over 96% of pairwise comparisons on average; across 4P and 2P it wins over 84% of comparisons against personalization methods and over 60% against recent image-editing models (FLUX Kontext, Overlay Kontext, Qwen-Image-Edit, Nano-Banana). Per-baseline 4P win rates range from 60.6% (Nano-Banana) to 98.0% (ID-Patch); 2P win rates range from 84.7% (OmniGen2) to 98.4% (ID-Patch).
• Efficiency gains from transparent latent pruning: On the 4P benchmark, the average number of conditioning tokens drops by over 85%, and inference latency falls from 91.8 seconds to 21.2 seconds. Full measurements: 2200.2 tokens / 21.2 s for CanvasComposer, 5120 tokens / 44.3 s without the multi-reference canvas, and 16384 tokens / 91.8 s without transparent pruning.
• Occlusion robustness: On a 2P Overlap benchmark, where two references are centered in the same canvas region, CanvasComposer keeps an ArcFace of 0.416 while the no-multi-reference-canvas collage variant collapses to 0.164; on the standard 2P benchmark the two variants are closer (0.547 vs. 0.525).
• An IP-Adapter alternative trained on the same data scores lower: 0.485 ArcFace, 10.2 HPS, 0.856 VQAScore on 2P (no 2P Overlap numbers reported for it).
• Ablations isolate each component: Removing cross-reference training produces copy-paste artifacts resembling naive outpainting; removing the multi-reference canvas loses occluded content and degrades identity preservation for occluded subjects, a failure the paper illustrates with the pom-pom on a red hat.
• Layout control: The canvas layout sets the relative spatial arrangement of subjects, while the text prompt controls pose, interaction, and scene context.
• Single-person behavior: In the 1P comparison, competing methods tend to copy the reference's pose and expression, whereas CanvasComposer follows diverse prompts such as smiling and laughing.
• Beyond humans: Although trained only on human-centric data, CanvasComposer can accept non-human objects placed on the canvas at inference and produce coherent person-object interactions.
• Stated caveat: The paper notes an input asymmetry — CanvasComposer and the editing baselines receive the composited canvas encoding the intended layout, whereas prior personalization baselines accept only individual reference images without spatial layout control.

Methodology in Plain English

The system builds on a pretrained FLUX Kontext diffusion transformer, kept frozen except for lightweight LoRA finetuning, so the base model's capacity is preserved rather than replaced with new attention branches or per-subject adapters.

Instead of feeding the model one flattened collage, each subject (and an optional background) is supplied as a separate RGBA image, where the alpha channel marks which pixels are valid. Each RGB element is encoded with the pretrained VAE into latent tokens, and every token gets a 3D positional embedding of the form [j, x, y] — the same 3D RoPE interface FLUX Kontext already uses. The x, y are latent-grid coordinates, and the index j identifies which canvas element a token belongs to; the base model's index 0 is reserved for noisy image tokens, and user-placed elements get j ≥ 1. This index is only an input-side separation signal and carries no depth or z-ordering meaning, which is why the model can tell overlapping subjects apart.

Transparent latent pruning then discards tokens whose downsampled alpha is not above 0.5, so only valid tokens condition the model. Because all subjects share one fixed canvas, adding more subjects shrinks each subject's footprint, and the conditioning sequence does not grow linearly with subject count. The surviving tokens are concatenated and joined with the noisy latents as the transformer's input. Training uses a flow matching loss with this canvas conditioning.

To avoid the model simply copying pixels from input to output, training constructs each canvas by sampling each subject's segment from a different source image in the same identity group than the target image, fitting it to the target's bounding box and applying geometric jitter, color jitter (brightness ±0.5, contrast in [0.5, 1.0], saturation ±0.5, no hue shift), and object-level Gaussian blur and grayscale augmentation at probability 0.1 each. Training data comprises 32M in-house images across 6M scenes, filtered so each scene contains at most 4 identities, with human cutouts produced by an in-house YOLO11-style instance segmenter (the paper notes a public YOLO11-seg model can substitute, and that the internal dataset cannot be released, though a similar one can be curated by sampling public videos).

The LoRA has rank 512 on all attention layers, applied to 3072 × 3072 attention projection matrices, trained for 200K iterations with a constant learning rate of 1 × 10⁻⁴, total batch size 32, and 512 × 512 resolution, taking 4 days on 4 nodes of 8 A100 GPUs. Evaluation is at 1024 × 1024 resolution using 128 FFHQ-in-the-wild identity images and 32 prompts per benchmark, with 28 denoising steps and no post-processing. FFHQ is a public single-frame dataset and is not included in training.

Why This Matters

Personalization research has largely been single-identity, but the paper argues personalization is inherently social — people want pictures with friends and family. By moving layout control to the user and keeping references separated on the input side, this work offers a different design axis than token-concatenation or adapter-based conditioning, and its pruning insight (conditioning length proportional to valid content rather than subject count) is directly relevant to scaling multi-subject generation. Its efficiency numbers give concrete evidence that multi-person conditioning cost need not grow linearly.

Real-world applications:

• Consumer photo and messaging apps that let users insert friends, family, or themselves into a shared scene.
• Social and event imagery where a group photo is composed from separately captured portraits.
• Marketing and advertising creatives that need the same set of people rendered in new scenes, poses, or contexts.
• Avatar and identity-consistent content pipelines where each person is stored as a reusable RGBA asset and arranged on demand.

Industry relevance: the work comes from Snap Inc. with academic collaborators and targets an interactive, Photoshop-like creative experience, a format well matched to camera and social-media products where users already expect direct manipulation of composition rather than prompt engineering.

Future Directions

• Output-side occlusion and depth control: The paper explicitly leaves explicit control over occlusion or depth ordering in the output image as future work, suggesting integration of training-free occlusion control such as LaRender or training with depth-conditioned inputs, which would require depth estimation during data generation.
• Extending the canvas beyond humans: Since the model already generalizes to non-human objects placed on the canvas at inference despite human-only training, broader object and scene-element conditioning is a natural extension.
• Reducing the input asymmetry: Prior personalization baselines receive only individual images without layout control, so evaluation protocols that equalize inputs, or methods that infer layout themselves, remain open.
• Comparing and combining with layout-predicting approaches: The paper contrasts its user-composed canvas with Ar2Can, whose canvas is predicted by an Architect module and rendered by an Artist module trained with GRPO under a spatially grounded face-matching reward, leaving open how user-composed and automatically predicted canvases might be combined.

Target Audience

Researchers and engineers working on diffusion-based image generation, personalization, and multi-subject composition; practitioners building creative or social-media image products who need identity-preserving generation of several people at once; and readers interested in efficient conditioning, latent pruning, and interactive interfaces for generative models. Readers without a diffusion-model background will follow the interactive canvas concept and the benchmark results, but the pipeline and training sections assume prior familiarity with VAEs, DiTs, LoRA, and RoPE.

Authors’ abstract

Existing personalized image generators still struggle to preserve multiple reference identities in natural and coherent multi-human generations. To address these limitations, we present CanvasComposer, an interactive framework for personalized group photo generation. Inspired by professional image-editing software, CanvasComposer allows users to place reference subjects on a shared canvas, where each subject keeps its own RGBA cutout of the input. This multi-reference canvas preserves reference content under overlap while providing an intuitive interface for organizing multiple identities; the subjects remain separate elements on the input canvas, and the model outputs a single personalized and harmonized image. To keep this representation efficient, transparent latent pruning retains only tokens from each subject's non-transparent region, and cross-reference training mitigates copy-paste artifacts by learning to harmonize references sampled from different images. Extensive experiments demonstrate that CanvasComposer achieves coherent generation and strong identity preservation compared to state-of-the-art methods in multi-human personalized image generation. Project page: https://snap-research.github.io/canvascomposer