IECLA: Interpretable Emotion Contribution Learning for Affective Image Editing

AI summary

Overview

Research area: Computer Vision — affective image editing, emotion attribution, and attribution-guided generative image editing.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-based image editors, vision-language encoders (CLIP, SigLIP2, DINOV3), region proposal pipelines (Grounding DINO, SAM), and attribution methods (Grad-CAM family, RISE, counterfactual perturbation).

Scope: IECLA is a framework that learns signed, object-level "emotion contribution" scores from counterfactual interventions and uses those scores as an interpretable control interface for steering existing image editors toward a requested emotion.

What This Paper Is About

Affective image editing asks an editor to make a source image evoke a target emotion (for example, fear or contentment) while keeping the original content and structure intact. The problem the authors identify is that an emotion word says nothing about which parts of the source image should change and which should be left alone — especially in complex scenes where emotion is spread across people, objects, and context. IECLA addresses this by first learning which regions support or oppose the image's current (source) emotion, then handing that signed evidence to a planning agent that converts it into localized instructions for an off-the-shelf editor.

Key Contributions

1. Signed object-level counterfactual supervision. The authors formulate emotion contribution as a signed, per-region training target built from counterfactual interventions (blurring a candidate region) corrected against matched reference-mask interventions. Positive and negative values distinguish evidence that supports the source emotion from evidence that opposes it.

2. A context-conditioned contribution predictor. A predictor built on the SigLIP2 vision backbone learns to estimate these contributions from a single unmodified image plus region masks, so no repeated interventions or source-emotion labels are needed at inference.

3. A unified attribution-guided editing framework. Predicted contributions become an interpretable emotion heatmap, and a target-aware agent converts that heatmap plus the desired emotion into a budgeted, content-aware editing plan for existing editors — without retraining the editors' parameters.

4. Broad empirical validation. Evaluations cover a locked attribution benchmark (7,278 images, 53,180 SAM masks), five external datasets (3,370 images), six editing backbones (5,440 matched tasks), ablations, three external VLM references, an independent human attribution review, and a blinded user study.

Main Findings

• Attribution quality leads all baselines. On the locked test set, IECLA reaches Impression ρ 0.5885, Signed ρ 0.5088, Top-1 0.5808, NDCG@5 0.8915, Macro-F1 0.3703, MAE 0.0939, RMSE 0.1743, and R² 0.4005. The strongest learned vision-encoder baseline (SigLIP2) reaches 0.3819, 0.2314, 0.4445, 0.8050, 0.3454, 0.1020, 0.1833, and 0.3370 respectively. The best heatmap method (LayerCAM) reaches Imp. ρ 0.2176, Top-1 0.2991, NDCG@5 0.7078 and cannot produce signed outputs.

• Gains transfer across all six editing backbones. With Qwen-Edit, Target Success rises from 0.2996 to 0.3579 (+19.5%), Emo-A from 0.2588 to 0.3081 (+19.0%), and Emo-S from 0.1843 to 0.2173 (+17.9%). Simultaneously, PSNR rises from 15.4985 to 20.3046 (+31.0%), SSIM from 0.5790 to 0.6137 (+6.0%), and LPIPS falls from 0.3698 to 0.3440 (−7.0%). InstructPix2Pix, FLUX.2, LongCat, OmniGen2, and HiDream show the same direction on both affective and preservation metrics.

• Emotion and structure improve together, not as a trade-off. The authors stress that IECLA shifts the conservative-edit versus strong-edit trade-off rather than simply increasing edit strength, since affective scores and PSNR/SSIM rise while LPIPS falls.

• Cross-dataset transfer holds. On 3,370 images from COCO, EmoSet, FI, Emotion6, and EmoArt with Qwen-Edit, all three preservation metrics improve on every dataset. Notable shifts include Emotion6 Target Success +71.0% and Emo-S +240.3%, and COCO SSIM +41.9% with LPIPS −50.8%. EmoArt shows the smallest structural gains (+8.5% PSNR, +6.7% SSIM, −5.1% LPIPS), which the authors attribute to affect being distributed across palette, texture, and composition in artwork.

• Attribution is the most important module. In ablations on Qwen-Edit, removing attribution causes the largest affective drop (Target Success 0.2915 versus 0.3579 for the full model). Removing the learning module also lowers every affective metric, and removing the agent keeps the strongest affective scores among ablations (0.3540) but still lowers every affective metric relative to the full model.

• Focal sign supervision matters most at the loss level. The loss-level ablation reports that focal-sign supervision has the largest affective impact, with ranking terms preserving regional importance and direction and auxiliary supervision reinforcing affective semantics.

• External references agree. Under three external VLM references on the same 289 images and 1,032 aggregated SAM regions, IECLA leads in regional ordering, signed agreement, top-region selection, and localization. An independent Human Review also aligns more closely with both the location and direction of affective evidence.

• Human raters prefer the edits. Forty independent reviewers scored 120 paired editing tasks across six editors, and IECLA-guided outputs received higher mean ratings for both target expression and naturalness; 20 additional heatmap tasks assessed attribution-map quality.

• IECLA beats prior editing baselines jointly. In the appendix comparison on the 5,440 source–target tasks, Qwen-Edit + IECLA leads in five of six evaluations (PSNR 20.3046, SSIM 0.6137, LPIPS 0.3440, Emo-A 0.3081, Emo-S 0.2173); EmoEdit ranks second on LPIPS and AIF leads on CLIP-I (0.9932).

Methodology in Plain English

The framework splits affective editing into two decisions that are usually tangled together: what in the source image is producing the current emotion, and how to change it toward the target.

Step 1 — Build signed training labels without human annotation. The authors train an emotion evaluator on EmoVerse image-level supervision and freeze it. For each candidate region (found with Grounding DINO and SAM), they apply two interventions: blurring that region, and applying a matched reference mask of comparable area and position. The change in the evaluator's score for the inferred source emotion, corrected by the average reference-mask effect, becomes the region's contribution value. Positive values mark regions whose removal hurts the source emotion (supporting evidence); negative values mark regions whose removal helps it (opposing evidence).

Step 2 — Learn to predict those values from one image. A predictor on the SigLIP2 backbone looks at three views per region at both global and local scale — the candidate region, its complementary context, and the whole image — and fuses them with difference and product features alongside region semantics, evaluator state, emotion embedding, and geometric metadata. A prediction head outputs a nonnegative magnitude plus probabilities over negative, neutral, and positive signs, combined into a signed score. Training combines a weighted Huber term, an absolute-ranking term, a signed-ranking term, a focal term, and an auxiliary term (weights 1, 0.25, 0.10, 0.20, 0.10 for the four named terms respectively as listed in the objective).

Step 3 — Plan and execute the edit. At inference the frozen evaluator reads the untouched input to infer the source emotion, and the predictor produces signed scores without any interventions. These form an emotion heatmap. A target-aware agent checks the attribution against visible content, picks the primary emotional carrier, anchors the main edit on positive evidence, adds optional atmosphere or minor-object actions only when the scene supports them, and treats negative evidence as preservation constraints. A visual and structural critic checks executability, a deterministic compiler assembles the prompt, and an instruction auditor checks for conflicts before the prompt reaches an existing editor.

Why This Matters

Impact on research. The paper reframes attribution from a diagnostic visualization into a reusable control signal. It shows that signed, object-level counterfactual targets — not just saliency heatmaps — carry information that generic editors cannot infer from an emotion label, and it demonstrates that this signal transfers to six different backbones without touching their parameters. It also provides a benchmark protocol for evaluating signed attribution (ranking, sign agreement, and regression together), addressing the known problem that visually plausible maps need not be faithful.

Real-world applications:

• Content creation and social media: adjusting mood and atmosphere of photos or artwork while keeping subjects recognizable.
• Advertising and marketing imagery: tuning campaign visuals toward a specified emotional register without reshooting.
• Mental-health and wellbeing tools: guided, restrained image adjustments where preserving scene identity matters more than dramatic transformation.
• Creative tools for artists and designers: interpretable heatmaps showing which elements carry an image's emotional reading, usable as a feedback layer inside editing software.

Industry relevance. Because IECLA works as a plug-in guidance layer over existing editors (Qwen-Edit, InstructPix2Pix, FLUX.2, LongCat, OmniGen2, HiDream) rather than a new generative model, it maps naturally onto product architectures that already have generation backends. The interpretable heatmap is also a plausible user-facing explanation layer, and the consistent preservation gains address a practical blocker for consumer deployment.

Future Directions

• Better handling of distributed affect in art and abstract imagery. EmoArt shows the smallest structural gains, and the authors tie this to affect living in palette, texture, and composition rather than a localized region — an open problem for region-based attribution.

• Beyond the teacher's notion of emotion. The attribution benchmark measures fidelity to the frozen evaluator's counterfactual contributions rather than human causal attribution, a limitation the authors state explicitly. Extending supervision toward human-causal labels is a natural next step.

• Relaxing the frozen-backbone constraint. The framework guides editors without retraining them; whether jointly tuning editors with the contribution signal would yield further gains is unexplored.

• Richer emotional targets. The evaluation covers eight editing emotions and categorical labels; the auxiliary loss already carries VAD information, leaving continuous valence–arousal control and finer emotional granularity as open extensions.

Target Audience

Researchers and practitioners in computer vision and generative modeling who work on image editing, affective computing, or explainable AI. It is most useful to readers already comfortable with diffusion or flow-based editors and vision-language encoders, and to applied engineers who want an interpretable, model-agnostic control layer for existing generative pipelines. Readers seeking an introductory treatment of affective computing will find the attribution and editing machinery dense, though the framing of the problem — that an emotion word does not specify what to change — is accessible to a general technical audience.

Authors’ abstract

Affective image editing (AIE) aims to evoke a target emotion while preserving the content and structure of a source image. Existing methods improve emotional expression or visual fidelity, but the target emotion alone does not reveal which source regions should be changed and which should remain intact, especially in complex scenes. We introduce Interpretable Emotion Contribution Learning for Affective Image Editing (IECLA), a framework that separates source-emotion attribution from target-aware editing. IECLA constructs signed object-level supervision through counterfactual interventions and trains a context-conditioned contribution predictor to localize evidence that supports or opposes the source emotion. At inference, an image-level evaluator estimates the source emotion directly from the input, while the predictor identifies editable and preservation evidence without repeated interventions. A target-aware editing agent then translates this evidence and the desired emotion into localized, content-aware operations for existing editors. Extensive evaluations show that IECLA predicts counterfactual contributions more faithfully on both in-domain and cross-dataset tests, while external VLM and independent human references support the quality of its localized evidence. Across six editing backbones, IECLA consistently strengthens target-emotion expression while preserving image content and structure, and user studies further show improved perceived target expression and naturalness.