Ar2Can: An Architect and an Artist Leveraging a Canvas for Multi-Human Generation

AI summary

Overview

• Research area: Computer vision, specifically text-to-image diffusion and personalized (identity-preserving) image generation for scenes containing multiple people.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching models, reinforcement learning (GRPO), vision-language models, and face-recognition embeddings (ArcFace).
• Scope (one sentence): The paper introduces Ar2Can, a two-stage framework in which an "Architect" model predicts where each person should appear and an "Artist" diffusion model renders the photorealistic image at those locations while preserving each reference identity.

What This Paper Is About

Current personalized image generation models fail when asked to depict several different people in one scene: they merge faces, duplicate the same person, lose identities, or produce the wrong number of people. The authors argue this happens because a single-stage model must simultaneously decide where people go and what they look like, which conflates spatial structure with appearance. Ar2Can splits these responsibilities into two stages — spatial planning (the Architect) and identity rendering (the Artist) — and is evaluated on the MultiHuman-Testbench and Multi-ID-Test benchmarks.

Key Contributions

1. A two-stage decomposition framework, Ar2Can (Architect, Artist, and Canvas), which separates spatial layout planning from identity-preserving rendering for personalized multi-human generation.
2. Two Architect variants with different trade-offs: Architect-A (Qwen-2.5-0.5B, supervised fine-tuning on bounding-box prediction with L1 + GIoU losses) and Architect-B (Flux-Schnell, fine-tuned with reinforcement learning, which also produces human pose).
3. An RL-trained Artist (Flux-Kontext) optimized with GRPO on compositional rewards: count accuracy, prompt/perceptual quality (HPSv3), a spatially-grounded face-matching reward combining Hungarian centroid matching with ArcFace identity similarity, and pose correction, combined with curriculum learning.
4. A token sharing and dropping strategy that reduces inference time by 2x while letting the model implicitly handle occlusion through shared positional encodings.
5. Results using primarily synthetic data: The method achieves gains in count accuracy and identity preservation without requiring real multi-human images.

Main Findings

• Count accuracy: Ar2Can with Architect-A reaches the highest count accuracy of 90.2 on MultiHuman-Testbench, outperforming WithAnyone (89.8) and GPT-Image-1 (87.9), as well as Architect-B (86.9).
• Identity preservation: Ar2Can with Architect-B achieves the best Multi-ID score of 68.2, exceeding the best prior open-source method, MH-OmniGen (54.5), by 13.7 points.
• Perceptual quality: Architect-B achieves the highest HPS of 30.8 among methods in Table 1.
• Unified metric: Ar2Can reaches 72.4 (Architect-B) and 72.2 (Architect-A) on the Unified Metric that balances identity preservation and prompt alignment, substantially outperforming baselines.
• Proprietary comparison: GPT-Image-1 and Nanobanana score strongly on Action-S (97.0 and 98.9) and Action-C (91.1 and 95.5) but underperform sharply on identity preservation (28.8 and 20.6 Multi-ID). Ar2Can surpasses them on Multi-ID while staying competitive on prompt alignment and quality.
• Architect trade-offs: Architect-A is better at counting (90.2 vs 86.9) due to stronger language understanding; Architect-B achieves higher Action scores (82.0 vs 77.6 on Action-C) due to 2D spatial priors.
• Second benchmark: On Multi-ID-Test, Ar2Can reaches Multi-ID (Ref) 54.3 versus WithAnyone's 50.1 (a 4.2-point gap) and Multi-ID (GT) 36.5 versus 33.8 (2.7 points), with prompt alignment of 30.1. The authors attribute the smaller gaps to that benchmark's bias toward 1–2 person scenes.
• Ablation — simple matching hurts: Naive location-based face extraction improves Multi-ID (14.5 to 55.2) but degrades count (80.7 to 75.6) and HPS (29.2 to 27.6) because of copy-paste artifacts and unnatural face sizes.
• Ablation — full model: Adding Hungarian centroid matching lifts Multi-ID to 60.3 and HPS to 30.9; adding the curriculum reaches 86.9 count accuracy and 68.2 Multi-ID.
• Human preference: In a study with 25 evaluators, 25 triplet samples (3–5 input identities), and complex scene prompts, Ar2Can won 88% of prompts overall, versus 8% for XVerse and 4% for DreamO. Per-criterion preference scores were 75.2 (prompt alignment), 79.8 (ID similarity), and 70.1 (quality).
• Efficiency: Token dropping reduces token counts by 2x on average, and the authors report a favourable latency-quality trade-off on an A100 GPU at 1024×1024 with 3 identities.
• Architect-C: A unified multimodal architecture (BAGEL) fine-tuned with the Architect-B RL procedure achieves comparable Artist performance but at significantly higher training and inference cost.

Methodology in Plain English

Given a text prompt (for example, "three people playing basketball") and N reference face images, the system first plans and then paints.

Planning (the Architect). Architect-A is an autoregressive language model, Qwen-2.5-0.5B. Because language models cannot directly emit continuous coordinates, the authors extend its tokenizer with layout tokens (<SoL>, <EoL>, <C>) and attach a coordinate regression head and a coordinate embedding head alongside the normal token head. Predicted coordinates are re-embedded as instance-specific tokens (<Ci>), and consistent coordinate ordering is enforced through data sorting to improve training stability. Training combines cross-entropy loss with a coordinate loss using gIoU and L1 terms.

Architect-B instead uses Flux-Schnell, a fast flow-matching text-to-image model with only 4 denoising steps. It is fine-tuned with GRPO so that generated draft images contain the correct number of people, using a reward that combines exact count matching (detected via blob analysis) with HPSv3 prompt alignment. The draft then yields bounding boxes and pose.

Rendering (the Artist). Flux-Kontext renders the final image conditioned on the layout, the prompt, and the reference faces. It is trained with GRPO using four rewards: count accuracy, HPSv3 quality/prompt alignment, the spatially-grounded face-matching reward, and pose correction.

The face-matching reward is the paper's key mechanism. Instead of simply extracting the face at a predicted box (which causes copy-paste artifacts), the authors first match architect-predicted centroids to RetinaFace-detected face centroids using the Hungarian algorithm, then compute ArcFace identity similarity between each reference face and its matched generated face. The reward averages these similarities, so unmatched references score zero and wrong counts are naturally penalized. Because matching is on centroids rather than exact boxes, the model only needs to place faces near the planned positions, allowing natural pose, scale, and positioning variation.

A frontal pose reward, based on roll angle and facial symmetry from facial landmarks, applies when the prompt includes "Everyone is looking at the camera."

Canvas and efficiency. Training samples are built by generating N-person scenes with DisCo, detecting face locations, and replacing each synthetic face on a blank canvas with a real reference face. Each sample contains the canvas, the reference images, and the DisCo target image. At inference, only tokens inside the required regions are fed to the model, and overlapping regions receive identical RoPE positional embeddings, which signals spatial competition so the model resolves overlaps through depth layering, rearrangement, or partial occlusion.

Curriculum. Training data is bucketed by face count N ∈ {2,...,7}. For the first τ epochs, sampling is split equally between 2- and 3-person scenes; after τ epochs, sampling is uniform across all buckets. This is motivated by Flux-Kontext's tendency to reliably generate only 1–2 faces.

Data. The authors use 60,000 prompts describing 2–7 person scenes, generated with GPT-5, with 30% of prompts including the "Everyone is looking at the camera" tag. Data sources are multi-view references (approximately 100K identities), single-view references (approximately 500K images, with some secondary views synthesized by PuLID and others augmented by rotation, flipping, and brightness changes), and synthetic faces from DisCo. Sampling probabilities for the canvas sources are p1 = 0.5, p2 = 0.4, p3 = 0.1.

Training specifics. Architect-A is trained for 20K steps with AdamW, learning rate 10^-5, batch size 128, a 100-step warm-up, cosine decay, and λ_coord = 0.8. Architect-B is fine-tuned for 240 GRPO epochs and the Artist for 300 GRPO epochs, with curriculum transition over τ = 100 epochs. Face detection uses RetinaFace with a ResNet-50 backbone and confidence threshold 0.5; ArcFace embeddings are 512-dimensional (Anetlopev2), and pose uses the rtmlib library.

Why This Matters

Impact on research. The paper reframes multi-human generation failure as a structural problem — the entanglement of spatial planning and appearance rendering — and shows that reinforcement learning with spatially-grounded, non-differentiable rewards can correct a pretrained model without large-scale paired annotations. It also demonstrates that competitive multi-human results are achievable with primarily synthetic scenes paired with real reference faces, rather than proprietary multi-human datasets.

Real-world applications:

• Photo and video editing tools that place several specific people into a single generated scene (family portraits, group shots).
• Storyboarding and advertising content creation where brand ambassadors or actors must appear together with consistent, recognizable faces.
• Virtual and augmented reality avatars, group scenes, and social applications requiring several distinct identities in one frame.
• Personalization features on mobile and consumer devices, where a modular, replaceable Architect allows a speed-versus-accuracy trade-off for on-device deployment.

Industry relevance. The work comes from Qualcomm AI Research, and the design reflects deployment concerns: a modular Architect that can be swapped without retraining the Artist, a 2x token reduction from dropping and sharing, explicit latency measurements on an A100 GPU, and a data strategy that avoids the proprietary multi-human datasets the authors note are confined to organizations with vast user-generated content.

Future Directions

• Extending beyond the tested person counts. The curriculum covers buckets of 2 to 7 faces, while MultiHuman-Testbench spans 1–5 people; scaling to larger groups is not demonstrated and remains an open question.
• Reducing dependence on synthetic scene generation. The pipeline relies on DisCo to synthesize multi-person scenes before real reference faces are substituted, so quality and diversity may be bounded by that generator.
• Improving the unified multimodal Architect. Architect-C (BAGEL) achieved comparable Artist performance at significantly higher cost; closing that efficiency gap while retaining unified reasoning is an obvious next target.
• Refining occlusion and pose control. The paper's token-sharing mechanism handles occlusion only implicitly, and fine-grained body pose control paired with pose canvases is explored in the appendix rather than the main results.
• Documented limitations. The paper states that limitations and future research directions are discussed in Appendix F; the specific content of that discussion is not included in the provided paper text.

Target Audience

Researchers and engineers working on diffusion-based image generation, identity-preserving personalization, and reinforcement learning for generative models. It is also relevant to applied teams building multi-person image editing or avatar products, and to readers interested in how synthetic data plus RL rewards can substitute for proprietary multi-human training sets. Prerequisites include comfort with diffusion/flow-matching architectures, policy-gradient methods such as GRPO, and face recognition embeddings.

Authors’ abstract

Despite recent advances in personalized image generation, existing models consistently fail to produce reliable multi-human scenes, often merging or losing facial identity. We present Ar2Can, a novel two-stage framework that disentangles spatial planning from identity rendering for multi-human generation. The Architect predicts structured layouts, specifying where each person should appear. The Artist then synthesizes photorealistic images, guided by a spatially-grounded face matching reward that combines Hungarian spatial alignment with identity similarity. This approach ensures faces are rendered at correct locations and faithfully preserve reference identities. We develop two Architect variants, seamlessly integrated with our diffusion-based Artist model. This is optimized via Group Relative Policy Optimization (GRPO) using compositional rewards for count accuracy, image quality, and identity matching. Evaluated on the MultiHuman-Testbench, Ar2Can achieves substantial improvements in both count accuracy and identity preservation, while maintaining high perceptual quality. Notably, our method achieves these results using primarily synthetic data, without requiring real multi-human images. Project page: https://qualcomm-ai-research.github.io/ar2can/.