Contextualized Visual Personalization in Vision-Language Models

AI summary

Overview

Research area: Computer Vision / Vision-Language Models (VLMs), specifically personalized multimodal understanding, reinforcement-learning-based post-training, and benchmark design.

Technical level: Intermediate. The paper is readable without deep RL theory, but it assumes familiarity with vision-language models, in-context learning, and reward-based post-training.

Scope: The paper introduces "contextualized visual personalization" as a formal problem, proposes the CoViP framework (personalized image captioning as a proxy task, trained with RL and paired with caption-augmented generation), builds a new benchmark, and evaluates VLMs from multiple vendors on diagnostic personalization tasks.

What This Paper Is About

Current vision-language models can recognize what is in an image, but they fail to connect that image to what a specific user has experienced before. For example, a model may describe "a man in a black suit" instead of recognizing "your brother, whom you last met at the airport." The authors formalize this gap as contextualized visual personalization and build a framework, benchmark, and evaluation suite to close it.

Key Contributions

1. A new problem formulation. The authors introduce contextualized visual personalization, where a VLM must reason jointly over a query image, a user prompt, and a contextual history of past interleaved image–text interactions. A use case paper defines this as generating responses grounded in user-specific visual experience rather than generic descriptions.

2. The CoViP framework. CoViP treats personalized image captioning as the core shared process underlying diverse personalization tasks. It combines RL-based post-training on a verifiable reward with caption-augmented generation (CAG), where the model's own generated caption is reused as an explicit conditioning signal at inference time.

3. A personalized image captioning benchmark. The benchmark contains 2.8K training samples and 1.3K test samples, built from synthesized query images (one to four concepts per image), dialogues grounded in factual information (locations, timestamps, events, scenarios), and interleaved positive/negative context images retrieved by CLIP-L/14 (top-2 visually similar images per positive image). Evaluation uses CapEval-QAs, a caption-based multiple-choice QA probing protocol that reports Positive Accuracy (Acc+) and Negative Accuracy (Acc−).

4. Diagnostic downstream evaluations. Three tasks — Last-Seen Detection (LSD), Last-Action Recall (LAR), and Instruction-Triggered Recall (ITR) — probe whether a model truly grounds visual input in user history or takes textual shortcuts. The paper also reports gains on four established multi-image benchmarks.

Main Findings

• Existing VLMs cannot generate context-grounded captions. On CapEval-QAs, Gemini-3.0 Pro is the strongest proprietary model (Acc+ 58.1 for 1-concept, dropping to 32.4 for 4-concepts). GPT-4o scores Acc+ 34.2 at 1-concept and 15.3 at 4-concepts. Open-source Qwen3-VL-8B scores 39.0 and 18.6 respectively. All models score high on Acc− (94.1 to 99.5 range across reported settings), meaning they avoid irrelevant content but fail at capturing relevant content.

• Prior personalization baselines barely help. Qwen3-VL-8B + RAP underperforms the zero-shot baseline on every concept count (20.5, 10.4, 9.9, 7.3 Acc+). Qwen3-VL-8B + RePIC improves only slightly (44.0, 31.7, 29.2, 24.0).

• CoViP yields large captioning gains. Qwen3-VL-8B + CoViP reaches Acc+ 77.4 (1-concept), 68.4 (2), 65.2 (3), and 59.7 (4), a per-setting increase of +38.4, +42.8, +41.9, and +41.1 over the base VLM — an average improvement of approximately 40% across all concepts. Acc− decreases slightly (to 94.8, 94.1, 94.8, 92.8).

• CoViP generalizes to downstream diagnostic tasks. Direct-inference recall improves from the base Qwen3-VL-8B's 29.8 to 37.2 on LSD (+7.4), from 17.4 to 34.8 on LAR (+17.4), and from 9.40 to 28.0 on ITR (+18.6). With CAG applied, CoViP reaches 58.2 on LSD (+9.4 over base with CAG), 49.2 on LAR (+29.6), and 42.8 on ITR (+36.0).

• Proprietary models are strong but unstable, and CAG can backfire. Gemini-3.0 Pro scores 76.2 direct / 89.3 with CAG on LSD and 89.4 direct on ITR, but only 9.40 direct / 44.0 with CAG on LAR. On ITR, applying CAG drops proprietary model recall below 20% across all VLMs (GPT-4o 13.5, GPT-5 10.5, Gemini-2.0-Flash 12.2, Gemini-3.0 Pro 19.0). The authors attribute this to generic captions lacking the fine-grained details that ITR's triggering keyword requires.

• Gains come from retrieval, not recognition. Analysis in Figure 6 shows entity-name F1 (recognition) increasing only moderately across models, while MCQA accuracy (retrieval) rises by a much larger margin at comparable F1 levels. Baseline models already recognize reasonably well; retrieval is the bottleneck.

• CoViP improves established multi-image benchmarks. MM-NIAH rises from 86.7% to 88.0% (+1.3%), MMNeedle from 47.3% to 51.3% (+4.0%), MuirBench from 79.8% to 81.8% (+2.0%), and MMIU from 48.3% to 49.8% (+1.5%), for an average of 65.5% to 67.7% (+2.2%).

• Human evaluation supports the metric. Captions preferred by CapEval-QAs obtain higher win-or-tie rates across both human criteria (Context Groundedness and New Image Description). CoViP shows a clear advantage on contextual grounding and performs on par with Qwen3-VL-8B on describing the new query image; specific win-rate numbers appear only in Figure 5 of the truncated content and are not tabulated in the main text.

Methodology in Plain English

Step 1 — Define the shared sub-problem. The authors argue that across all personalization tasks, a model first has to interpret an image relative to a user's past experience, then generate a task-specific response. They label these two stages as an encoder-like component and a generator-like component, and observe that only the first is common to every task. That common component lines up exactly with personalized image captioning, so captioning becomes the training target.

Step 2 — Build a benchmark. They generate query images containing one to four concepts using an image-generative VLM, filter them with a text-generative VLM that checks instruction adherence and visual faithfulness, and pair them with multi-turn dialogues containing only verifiable facts. Each context mixes positive images (concepts that actually appear in the query) with negative images (visually similar, retrieved by CLIP-L/14, but absent from the query). This forces the model to discriminate visually rather than guess from text.

Step 3 — Score captions automatically. For each dialogue, an LLM generator produces three factual multiple-choice QA pairs. A judge model sees only the generated caption and answers the questions. Correct answers on positive-concept questions give Positive Accuracy; correctly answering "cannot be determined" on negative-concept questions gives Negative Accuracy. This design rules out textual shortcuts.

Step 4 — Train with verifiable rewards. The model is post-trained with RL (using the GSPO algorithm) to maximize a reward with two parts: a recognition reward — an F1 score comparing the set of concept indices the model says appear in the query against the ground-truth set — and a retrieval reward based on how well the generated caption lets the judge answer positive questions while avoiding wrong answers on negative questions. A degeneration filter assigns a reward of −1 to degenerate outputs.

Step 5 — Reuse captions at inference. In caption-augmented generation, the model first writes a personalized caption from the query, context, and captioning prompt, then conditions on that caption to answer the actual downstream user prompt. Training uses LoRA; full hyperparameters are in the paper's appendix and are not reported in the main text.

Why This Matters

This work shifts visual personalization from retrieving surface-level attributes (such as a name) to reasoning over episodic, experience-level user history, and it contributes a benchmark and a diagnostic suite that explicitly penalize shortcut solutions. It also argues that captioning-focused post-training should come before downstream personalization adaptation, since trying to train every downstream behavior directly does not scale to the open-ended long tail of real interactions.

Real-world applications:

• Personal AI assistants that recognize people, objects, and places from prior conversations and respond accordingly.
• Augmented-reality or camera-based memory aids that surface what happened during the last encounter with a person.
• Proactive notification systems that trigger a user-defined keyword when a specific individual is seen again (the ITR scenario).
• Photo and content organization tools that generate captions grounded in the user's own history rather than generic descriptions.

Industry relevance: The paper evaluates models from OpenAI, Google, and Alibaba alongside in-house baselines, and the author affiliations include Upstage AI, Samsung Electronics, DGIST, and Seoul National University. The finding that adding a captioning post-training stage stabilizes downstream personalization has direct implications for anyone shipping personalized multimodal agents.

Future Directions

• Move beyond synthetic data. The authors note that developing personalization benchmarks grounded in real-world user signals — shopping histories, conversational voice recordings, and long-term user–model interaction logs — is a key next step.
• Extend to omnimodal settings. The paper points toward omnimodal personalization benchmarks as a follow-up direction.
• Privacy-preserving personalization. The impact statement calls for investigating mechanisms that prevent sensitive visual history from being exposed, since the framework relies on stored personal visual experience.
• Strengthen evaluation. The limitations section notes that synthetic dialogues and generated images may contain factual inconsistencies or visual artifacts, and that additional human verification of dialogue factuality and image fidelity would help.
• Address instability. The authors leave open why proprietary VLMs behave inconsistently across diagnostic tasks and why CAG degrades ITR performance for generic captions.

Target Audience

Researchers and engineers working on vision-language models, multimodal personalization, or RL-based post-training, as well as practitioners building personalized assistants and context-aware AI products. It is also relevant to benchmark designers interested in evaluation protocols that rule out shortcut solutions, and to readers tracking the gap between proprietary and open-source VLM personalization capabilities.

Authors’ abstract

Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user's specific experiences, as they lack the ability to associate visual inputs with a user's accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.