PerTouch: VLM-Driven Agent for Personalized and Semantic Image Retouching

AI summary

Overview

• Research area: Computer vision, specifically diffusion-based image retouching combined with vision-language-model (VLM) agents.
• Technical level: Advanced. The paper assumes familiarity with latent diffusion models, DDPM denoising, ControlNet conditioning, and multimodal agents.
• Scope (1 sentence): PerTouch is a unified diffusion framework that performs globally aesthetic and region-level image retouching from natural-language instructions, while a VLM-driven agent interprets strong and weak user prompts and stores scene-aware memory of long-term personal preferences.

What This Paper Is About

Most deep-learning retouching tools produce one fixed output for a given input, cannot edit specific regions cleanly, and ignore vague user instructions or repeated editing habits. PerTouch aims to fix all three by letting a diffusion model follow region-level "parameter maps" for four attributes (colorfulness, contrast, color temperature, brightness) while a VLM-based agent translates fuzzy language like "increase significantly" into those parameters. The goal is retouching that preserves global aesthetic quality but adapts to an individual user's taste over time.

Key Contributions

1. Semantic-aware region adjustment using diffusion priors. A parameter-map representation is injected through ControlNet so that global aesthetics come from the diffusion prior while regional edits come from explicit attribute scores.
2. A data preprocessing scheme combining semantic replacement and parameter perturbation. Semantic replacement swaps a randomly chosen region (chosen with probability proportional to semantic area) with a region from another sample having the most divergent attributes; perturbation adds channel shifts and blurring to the parameter maps so the network treats segmentation as soft guidance rather than a rigid structure.
3. A VLM-driven agent with a scene-aware memory mechanism. The agent handles both strong instructions (e.g., "Significantly increased eagle brightness.") and weak instructions (e.g., "Optimize this image."), and stores scene semantics alongside confirmed editing parameters to model long-term user preferences.
4. Feedback-driven rethinking. An iterative loop where the agent inspects its own output and revises the control value across steps, allowing it to interpret subjective intensity words and refine unsatisfactory results.

Main Findings

• PSNR gains across all five experts (Table 1, MIT-Adobe FiveK test set): PerTouch reports PSNR of 25.1430 (A), 27.4733 (B), 26.7510 (C), 25.9726 (D), and 25.6602 (E), higher than PIENet, TSFlow, StarEnhancer, and Diffretouch on every expert version. For comparison, Diffretouch reports 24.5082, 26.1473, 25.9148, 24.5087, and 24.7373 respectively.
• LPIPS results are mixed. PerTouch reports 0.0798 (A), 0.0687 (B), 0.0844 (C), 0.0823 (D), 0.0792 (E). Based on the numbers given, PerTouch has the lowest LPIPS only for expert A; TSFlow is lowest for C (0.0630) and Diffretouch is lowest for B (0.0672), D (0.0768), and E (0.0776). The paper states that region-level retouching is introduced while global performance is maintained or surpassed in objective evaluation.
• User study favors PerTouch. With 30 images randomly selected from the MIT-Adobe FiveK test set and 50 volunteers, participants chose the result best matching their personal preference among PerTouch and baselines including DiffRetouch, StarEnhancer, TSFlow, and JarvisArt. The paper reports that the majority of participants preferred PerTouch in nearly half of the test cases, significantly outperforming other approaches (shown as a Kernel Density Estimation plot).
• Semantic Replacement Module ablation. Removing it significantly degrades localization of retouching and produces spillover effects, where local edits undesirably affect global regions.
• Perturbation Mechanism ablation. Removing it causes overfitting to externally encoded segmentation boundaries and reduced global visual coherence.
• Qualitative comparison with agent-based Lightroom controllers. Because the MIT-Adobe FiveK dataset lacks detailed retouching path descriptions between low-quality inputs and ground-truth images, the paper states that direct quantitative evaluation of VLM-toolchain systems such as JarvisArt is infeasible, so only a qualitative comparison (Figure 5) is provided.
• Not reported: training details, dataset splits, image resolutions, and inference latency are deferred to the Supplementary Material; no runtime or efficiency numbers appear in the main text.

Methodology in Plain English

The system starts from Stable Diffusion, which denoises in a learned latent space, and adds ControlNet to inject extra spatial guidance. For each training image, SAM generates a coarse panoptic segmentation (with non-maximum suppression applied afterward), and each segmented region is scored on four attributes: colorfulness, contrast, color temperature, and brightness. Those scores are embedded into the segmentation map, and the channel dimension is expanded to match the number of attributes, producing a parameter map where each pixel carries the score of its region for a given attribute. Coefficient values are adjusted within the range [-1, 1], each corresponding to a visual style learned from high-quality images.

Two training tricks keep the model from blindly copying the map. Semantic replacement forces the model to notice regional discrepancies by splicing in a region from another image with very different attributes. Perturbation (channel shifts, blurring) keeps the map as a soft hint so the generative prior still governs the overall look.

On the interaction side, a VLM agent decides between instruction types. For weak instructions it initializes the multi-channel parameter maps at the midpoint of each attribute as a default, then tailors them using the user's history. For strong instructions it uses the VLM's object detection to find the region, calls SAM for a coarse mask, and modifies only that region while keeping adjustments from the weak mode elsewhere. Feedback-driven rethinking forms a closed loop: the agent estimates a control value, generates an image, inspects it, and revises the value. Scene-aware memory extracts scene semantics from each edited image and stores them with the final confirmed parameters in a personal memory bank; when a new image arrives, the agent samples a parameter vector from the scene-conditioned preference distribution to guide the edit.

Why This Matters

Research impact. The paper tackles a gap in retouching research: existing deterministic models cannot express subjective preference, reference-image style transfer burdens users, and mask-based regional editing is sensitive to segmentation quality. PerTouch offers one framework that couples region-level control with personalization, and reports that adding region-level retouching does not sacrifice global PSNR performance.

Real-world applications:

• Smartphone camera post-processing, where a phone could restyle a photo per user taste without manual slider work.
• Batch editing for e-commerce, real estate, or social-media content, where the memory mechanism can apply a consistent personal style across many images.
• Consumer photo apps that accept vague spoken or typed requests such as "make the sky pop a bit" and translate them into precise edits.
• Professional workflows where a photographer names a specific region and attribute and lets the agent handle parameter values.

Industry relevance. The author list includes Samsung R&D Institute China-Beijing and Samsung Electronics' Camera Innovation Group, with funding from Samsung R&D Institute China-Beijing (SRC-B), alongside academic support from Nankai University and Shenzhen. This points directly at commercial camera and gallery software. The agent design also competes conceptually with VLM-driven controllers of toolchains like Adobe Lightroom, though the paper could not benchmark those quantitatively on this dataset.

Future Directions

• Expanding the attribute set. The paper notes the framework is extensible: once a region-level score can be computed for a new attribute, controllability over that attribute can be added. Testing more attributes is a natural next step.
• Quantitative evaluation against tool-chain agents. The absence of retouching path descriptions in MIT-Adobe FiveK prevented quantitative comparison with systems like JarvisArt; building a dataset with edit trajectories would close that gap.
• Better evaluation of long-term personalization. Scene-aware memory is central to the claimed contribution, but the reported evaluation centers on FiveK experts, a user study with 50 volunteers, and ablations, rather than a longitudinal measure of how preference alignment improves over repeated sessions.
• Broader datasets and full experimental disclosure. All dataset details, training details, and parameter-injection specifics are deferred to the Supplementary Material, leaving open questions about generalization beyond the 5,000 RAW images and five expert versions of MIT-Adobe FiveK.

Target Audience

Researchers and engineers working on computational photography, image enhancement, diffusion-model conditioning, and agentic multimodal systems. It is most useful for readers already comfortable with latent diffusion and ControlNet, and for product teams in smartphone imaging or photo-editing software who want an architecture for personalized, region-aware retouching. Beginners would need substantial background reading before the methodology is accessible.

Authors’ abstract

Image retouching aims to enhance visual quality while aligning with users' personalized aesthetic preferences. To address the challenge of balancing controllability and subjectivity, we propose a unified diffusion-based image retouching framework called PerTouch. Our method supports semantic-level image retouching while maintaining global aesthetics. Using parameter maps containing attribute values in specific semantic regions as input, PerTouch constructs an explicit parameter-to-image mapping for fine-grained image retouching. To improve semantic boundary perception, we introduce semantic replacement and parameter perturbation mechanisms during training. To connect natural language instructions with visual control, we develop a VLM-driven agent to handle both strong and weak user instructions. Equipped with mechanisms of feedback-driven rethinking and scene-aware memory, PerTouch better aligns with user intent and captures long-term preferences. Extensive experiments demonstrate each component's effectiveness and the superior performance of PerTouch in personalized image retouching. Code Pages: https://github.com/Auroral703/PerTouch.