Personalized Image Generation with Reasoning and Reflection

AI summary

Overview

• Research area: Computer vision — personalized image generation, multimodal reasoning, and benchmark design for generative models.
• Technical level: Intermediate. The paper assumes familiarity with diffusion-based text-to-image generators (SDXL), multimodal large language models, preference optimization (DPO), and retrieval metrics, but each component is explained at a conceptual level.
• Scope: The paper introduces a benchmark (PMH-IG) for generating images from rich, naturally occurring user histories and proposes a framework (Pearl) that reasons over that history, reflects on its own rendering, and re-renders a corrected image.

What This Paper Is About

Existing work on "personalized" image generation mostly means reproducing a specific visual concept — a face, an object, or a painting style — from a handful of curated reference images while following a new text prompt. This paper argues that real personalization requires inferring who a user is from the messy multimodal traces they leave behind (reviews, posts, captions, images) and generating images that fit their lifestyle and aesthetic. To that end, the authors build a two-task benchmark from Amazon review histories and Instagram post histories, and propose Pearl, a system that plans an image from the user history, looks at the first rendering, and revises the plan to fix personalization mismatches.

Key Contributions

1. PMH-IG benchmark. The authors introduce what they describe as the first unified benchmark for personalized image generation from realistic user histories, built exclusively on public user data, with two complementary tasks: Personalized Scene Generation (Amazon review histories) and Personalized Creative Generation (Instagram post histories).
2. Multi-axis evaluation protocol. The benchmark assesses generations along target fidelity, visual quality, user distinguishability, semantic alignment with the user's history, and task-specific utility, combining image-based measures with retrieval-based and MLLM-judge-based evaluations.
3. The Pearl framework. A reasoning-interleaved architecture that pairs a multimodal reasoner with a frozen image generator: a planner reasons over history into an explicit image-generation plan, and a reflector compares the first rendering against the history to identify mismatches and produce a corrected plan for a final render.
4. A two-stage training procedure. Silver personalization trajectory distillation from a teacher model with privileged access to a ground-truth identity signal, followed by render-in-the-loop alternating-policy DPO in which each policy is scored by the full downstream pipeline that includes the other policy and the frozen renderer.

Main Findings

• Average gain of 15%. The abstract reports that Pearl outperforms strong baselines across both tasks with an average improvement of 15% across personalization metrics.
• Best on every Personalized Scene Generation metric. On the Amazon e-commerce task, Pearl reaches H@5 = 0.2297, MRR = 0.1639, aesthetic score = 5.33, MLLM-judge content = 4.36, overall = 3.92, and style = 2.88. For comparison, the strongest baseline H@5 values are 0.2162 (LaVIT), 0.2143 (PMG), 0.1873 (LLaVA), and 0.0985 (Pigeon); the best baseline MRR is 0.1535 (PMG).
• Best on 7 of 10 columns in Personalized Creative Generation. On the Instagram task, Pearl is best on all retrieval and MLLM-judge metrics and strictly best on CIS (0.645) and DIS (0.289) among target-based image metrics. Its inter-category R@1 is 0.876 and intra-category R@1 is 0.701, versus the next-best 0.748 and 0.659 (Pigeon). MLLM judge: style 3.554, content 3.277, overall 3.541.
• Fidelity metrics alone can be misleading. The authors note that the methods winning LPIPS (0.729 for PMG, and 0.735 for both LLaVA and Pearl) or MS-SSIM (0.068 for LaVIT) score poorly on retrieval, which the paper offers as evidence that a single fidelity metric can be won without delivering personalization.
• Baselines fail without explicit reasoning. The paper states that current personalization methods struggle on both tasks because they cannot reason over complex user history and translate it into an appropriate image; PMG and Pigeon fuse history directly into the generator, leaving cues entangled in dense vectors.
• Preliminary human evaluation favors Pearl. A single annotator assessed 50 distinct test users. For fit to the displayed user history, Pearl was selected in 19/25 scene-generation pairs (76%) versus PMG in 4/25 (16%), with insufficient evidence in two cases; for creative generation, Pearl was selected in 16/25 pairs (64%) versus Pigeon in 9/25 (36%). The authors label these as preliminary, single-annotator judgments of history fit, not the target users' own preferences.
• Reflection helps, mostly on fine-grained judge metrics. In the ablation comparing Pearl against Pearl-Reflection (which replaces alternating-policy DPO with supervised fine-tuning alone), Pearl wins on four of the five aggregated dimensions, with the largest improvements on MLLM-judge style and content. The overall score drops marginally by under 1%, which the authors attribute to reflection concentrating on fine-grained personalization details rather than drastic adjustments.
• The two tasks test different identity axes. Personalized Scene Generation draws on Amazon review histories that encode what a user does through behavioral signals, while Personalized Creative Generation draws on Instagram post histories that encode how a user looks through visual aesthetic signals.

Methodology in Plain English

The authors treat personalization as identity inference rather than concept copying. For each user, they assemble a history of prior activity: for the Amazon task, pairs of written review and reviewed product image; for the Instagram task, pairs of post image and caption metadata. The generation condition is separated from the user signal — in scene generation the condition is the product image to be personalized, and in creative generation it is a topic or content specification.

Pearl works in two steps. A planner model reads the user history and the condition and writes a chain-of-thought reasoning trace followed by a concrete scene description; a frozen SDXL image generator then renders that description. Because the planner commits to a description before seeing how the generator interprets it, the paper adds a reflector: it receives the user history and the first rendered image, compares the two along concrete personalization axes (scene incongruities, missing lifestyle cues, aesthetic deviations), and emits a revised scene description that is rendered again for the final output.

Training happens in two stages. First, a teacher multimodal model that has privileged access to a ground-truth identity signal (a held-out target image or post, depending on the task) produces "silver" trajectories, and the planner and reflector are each fine-tuned on their own trajectories with standard cross-entropy. Second, the authors alternate between updating the planner and the reflector with Direct Preference Optimization while holding the other policy and the renderer frozen. For each update they sample K = 4 candidate trajectories, run the full downstream pipeline on each, score the resulting images with a task-aligned retrieval reward, form preference pairs from the highest- and lowest-scoring candidates, and update with the standard DPO objective. The reward is a weighted composite of recommendation rank, identity element coverage, and a Qwen2.5-VL pointwise judgment for scene generation, and a weighted composite of CLIP and DINO similarities to the user's held-out post for creative generation. Only one planner update and one reflector update are performed; the authors explicitly leave multi-round optimization to future work.

Implementation details reported: planner and reflector are both initialized from Qwen2.5-VL-7B, the frozen renderer is SDXL-1.0 at resolution <1024×1024>, silver trajectories come from a Gemini2.5-Flash teacher, both policies are fine-tuned with LoRA adapters, sampling temperature is 0.9, and all experiments run on a single 4090 GPU.

Evaluation uses shared and task-specific metrics. Shared metrics are the LAION aesthetic predictor for output quality and an MLLM-as-Judge ensemble of Gemini Flash 2.5 and Qwen2.5-VL-32B scoring style, content, and overall on a 1–5 scale (the average of the two models is reported). Scene generation, which has no naturally occurring ground-truth scene per (user, product) pair, is evaluated with a two-tower contrastive visual recommender trained on real (review history, purchased product image) pairs, reporting Hit@5 and MRR over a pool containing the user's held-out true purchase. Creative generation has a natural ground-truth target (the held-out post), so it adds CLIP image similarity, DINO image similarity, LPIPS, and MS-SSIM, plus a StyleDiscriminator that retrieves the correct user from a decoy pool at inter-category and intra-category R@1.

Dataset statistics reported: Personalized Scene Generation has 3,972 train users (39,720 reviews), 510 validation users (5,100 reviews), and 518 test users (5,180 reviews), totaling 5,000 users and 50,000 reviews. Personalized Creative Generation has 3,464 train users (40K posts, 957 pairs), 742 validation users (8K posts), and 207 test users (2K posts, 314 pairs), totaling 4,413 users, 50,656 posts, and 1,271 pairs. Baselines compared are PMG (SDXL conditioned on LLM-extracted style keywords), Pigeon (MLLM-based with a learned masking on user history), LLaVA (dense image features for reasoning, generating text by default), and LaVIT (discrete visual tokens guiding generation); all four are paired with SDXL as the frozen generator to match Pearl's renderer.

Why This Matters

Impact on research. The paper reframes personalization from "reproduce this concept" to "infer this person," and supplies a benchmark with two tasks, five evaluation axes, retrieval-based personalization metrics, and real user histories rather than curated exemplars. It also demonstrates that reasoning and reflection can be trained end-to-end against a frozen renderer using preference optimization, and its multi-axis protocol exposes cases where a method wins one fidelity metric while failing personalization entirely.

Real-world applications.

• Personalized product imagery in e-commerce, where the same product is rendered in a scene matching each shopper's inferred lifestyle (a flashlight on a hiking trail for one user, in an engine bay for another).
• Creator and social media tooling that generates new posts matching an influencer's established visual identity in composition, color, setting, and presentation style.
• Generative recommendation, using generations as a signal for which product to surface to which user.
• Content personalization for platforms that already hold rich behavioral and visual histories.

Industry relevance. The paper is co-authored by researchers at Vanderbilt University, Adobe Systems, University of Maryland College Park, and University of Georgia, and is explicitly motivated by engagement and click-through-rate gains in e-commerce and by creator-support workflows. The task setup — a frozen off-the-shelf generator plus a trainable reasoning policy — is attractive for deployment because it avoids per-user fine-tuning of the image model, which prior test-time optimization methods such as DreamBooth and Textual Inversion require.

Future Directions

• Multi-round optimization. The training procedure currently performs only a single round of planner and reflector updates; the authors state that alternating for additional rounds is straightforward, that they expect further gains, and that a systematic study is left to future work.
• Direct evaluation of scene personalization. Personalized Scene Generation lacks a naturally occurring ground-truth scene image per (user, product) pair, so user alignment is measured indirectly through retrieval and judge metrics rather than direct reference-based comparison.
• Stronger human evaluation. The current human study is a preliminary, single-annotator assessment of history fit across 50 test users, and the paper notes it is not the target users' own preferences.
• Reducing reliance on privileged teacher signals. Stage 1 depends on a teacher model with access to a ground-truth identity anchor; how far this distillation can be removed or replaced is not resolved in the paper.

Target Audience

Researchers and practitioners in generative vision, multimodal reasoning, and personalization, particularly those working on text-to-image systems, recommender-adjacent generation, and identity-aware content creation. It is also relevant to industry teams building e-commerce product presentation or creator tooling who need a benchmark that rewards genuine identity inference over concept reproduction. Readers without a background in diffusion models or preference optimization will need to consult the referenced prior work (SDXL, DreamBooth, IP-Adapter, DPO) to follow the methodology in detail.

Authors’ abstract

Personalized image generation has remained narrowly focused on conditional synthesis from curated visual exemplars, rather than capturing who a user is. In practice, however, a user's personal context is much richer, comprising reviews, posts, images, captions, and metadata accumulated over time. A truly personalized generator should leverage this history to produce images aligned with the user's lifestyle and aesthetic preferences. To this end, we introduce the first unified benchmark for personalized image generation from user histories. The benchmark comprises two complementary tasks and a multi-axis evaluation protocol that assesses target fidelity, visual quality, user distinguishability, semantic alignment with the user's history, and task-specific utility. Grounded in real-world e-commerce and social media settings, the benchmark includes: (1) Personalized Scene Generation, which places a given object in a scene that reflects a user's preferences and lifestyle, motivated by personalized product presentation; and (2) Personalized Creative Generation, which generates a novel image on a specified topic that is faithful to a user's aesthetic and visual identity, motivated by social media content creation. We further propose PEARL, which couples a multimodal reasoner with a frozen image generator in an interleaved reason-reflect loop optimized with differential data reward. Across both tasks, PEARL outperforms strong baselines, achieving an average improvement of 15% across personalization metrics.