Barycentric Guidance: Turning Foundation Image Editors into Continuous Affective Controllers

AI summary

Overview

Research area: Computer vision and affective computing, specifically valence–arousal (VA) emotion control applied to pretrained diffusion-based image editors (FLUX.1 Kontext, FLUX.2 dev, FLUX.2 Klein, Qwen Image Edit 2511).

Technical level: Intermediate. The paper assumes familiarity with diffusion models, classifier-free guidance, latent spaces and text encoders (CLIP, T5, QwenVL), but the central idea is geometric and can be followed without that background.

Scope: The paper introduces a training-free guidance scheme that converts off-the-shelf foundation image editors into continuous controllers over the two-dimensional valence–arousal emotion plane, evaluated on human faces, non-human faces, artworks and scenes.

What This Paper Is About

Existing VA-based image editors are training-based and domain-specific, so they cover only the VA regions present in their training data and generalise poorly across datasets. Meanwhile, foundation image editors produce high-quality edits but are not continuous affective controllers, because non-linear text embedding spaces make prompt interpolation poorly calibrated. The goal is to achieve accurate, broad-coverage and continuous VA control without any finetuning, extra data or model modification.

Key Contributions

1. A training-free framework that turns pretrained foundation image editors into continuous affective controllers in VA space, targeting three properties of controllability: accuracy, coverage and continuity.
2. Barycentric Guidance, which approximates the VA model with a triangle mesh of discrete emotion anchors and maps any target VA point to simplex weights that are used to reweight classifier-free guidance rather than to interpolate text embeddings.
3. State-of-the-art VA alignment on facial expression editing across AffectNet, FFHQ and CelebA-HQ, with an optional geometry-aware latent interpolation module (face morphing plus partial re-diffusion) for structural consistency on faces.
4. Generalisation beyond human faces, showing the same framework edits animal faces, cartoon faces, 3D face models, artworks and scenes, including a VA-based generation variant that adds a base scene prompt.

Main Findings

• Reported headline gains: across three face datasets the method improves VA target accuracy by 25–29%, expands VA-plane coverage by 28–50% and increases expression diversity by 29–49% over the strongest baseline.
• VA accuracy: on AffectNet the method reports RMSE 0.433 versus 0.610 for EmoStyle; on FFHQ 0.418 versus 0.568; on CelebA-HQ 0.412 versus 0.550. GANmut is not scored on RMSE because it does not control VA directly (it steers emotion via polar coordinates).
• Expression diversity: VA entropy on AffectNet rises to 0.916 (Ours) and 0.813 (Ours with Qwen Image Edit 2511), compared with 0.423 for EmoStyle and 0.616 for GANmut. The paper also proposes measuring entropy of predicted VA points via Kernel Density Estimation as a better diversity measure than standard deviation.
• Image quality: FID on AffectNet is 11.02 (Ours), 10.67 (Ours, Qwen), 36.72 (EmoStyle) and 36.89 (GANmut); IS is 4.42, 4.71, 2.81 and 3.90 respectively. On FFHQ, FID is 13.85 versus 25.50 (EmoStyle) and 73.20 (GANmut); on CelebA-HQ, 11.80 versus 17.44 and 59.59.
• Background preservation: LPIPS on non-face regions is 0.096 (Ours) versus 0.276 (EmoStyle) and 0.128 (GANmut) on AffectNet; 0.062 for the Qwen-based implementation of the method.
• Identity trade-off: the method still lags on identity similarity in some settings (ArcFace 0.55 on AffectNet versus 0.76 for GANmut, and 0.54 on CelebA-HQ versus 0.65 for EmoStyle), which the authors attribute partly to its greater expression diversity. A study on MEAD (60 actors, three intensity levels; mid-frame images for 48 actors, split 0) confirms that ID similarity decreases as expression strength increases. The Qwen-based variant reaches ID similarity 0.72 on AffectNet.
• FID penalises diversity: on six 4k-image AffectNet subsets, FID rises from 8.36 (neutral only) to 8.56 (|VA| ≤ 0.25), 10.05 (|VA| ≤ 0.75 range bound 0.5), 11.64 (|VA| ≤ 0.75) and 13.40 (full VA range), suggesting the metric penalises expression distribution shifts in addition to measuring realism.
• Coverage of hard regions: the comparison reports RMSE down 0.337 versus EmoStyle in the top-left VA quadrant, and better handling of "sleepy" expressions, which prior work struggles to capture.
• Ablation (AffectNet): full approach RMSE 0.433, entropy 0.916, VA std 0.64/0.57, S2D 0.303, FID 11.02; removing scaled CFG gives RMSE 0.439, entropy 0.906; substituting prompt interpolation for Barycentric Guidance gives RMSE 0.461, S2D 0.342; removing face morphing gives RMSE 0.451, entropy 0.756, S2D 0.377, FID 8.94, ID similarity 0.66. Removing components improves FID and identity similarity by staying closer to the source, but the full approach is needed for VA coverage and control. (Note: the comparison table marks S2D as lower-is-better, while EmoStyle's reported AffectNet value of 0.147 is lower than the authors' 0.303; the paper does not discuss this.)
• One-dimensional continuous editing (FLUX.1 Kontext): the method reports RMSE 0.292, smoothness 0.643 and C-LPIPS 1.922, versus 0.310/0.707/3.679 for prompt interpolation, 0.316/0.720/3.987 for CFG, 0.476/0.686/3.490 for FluxSpace and 0.321/0.665/3.552 for Kontinuous Kontext.
• User study: 66 participants comparing against EmoStyle over 50 CelebA edits (Wilcoxon signed-rank test, p < 0.05). Participants preferred the method on VA accuracy (40 versus 9, p = 8.1e-6) and non-affective content preservation (36 versus 11, p = 6.4e-5), while EmoStyle was preferred for realism (33 versus 15, p = 2.9e-4) and identity (41 versus 8, p = 8.6e-7). EmoStyle's average VA change was |ΔVA| = 0.46 against a target of 0.86, versus 0.91 for the proposed method.
• Non-face results: on EmoArt the method reports RMSE 0.735 and entropy 0.327 versus 0.851 and 0.129 for prompt interpolation. On FindingEmo, against EmotiCrafter, it reports RMSE 0.783 versus 0.796, S*2D 0.497 versus 0.540, FID 221.51 versus 233.30 and IS 10.68 versus 8.85, with lower entropy (0.163 versus 0.298).
• Why prompt interpolation fails: PCA on CLIP text embeddings of the template "a portrait photo of a <emotion> face" largely recovers the valence–arousal axes on the first two principal components, and this holds across CLIP models and prompt templates, but not for T5 or QwenVL encoders. Since FLUX.1 Kontext uses CL

Authors’ abstract

Emotions are expressed on a continuum and are naturally described in the valence-arousal (VA) space. Existing VA-based image editors are typically training-based and domain-specific, limiting them to VA regions seen during training and making them brittle across datasets and domains. Meanwhile, despite producing high-quality edits, state-of-the-art foundation image editors are not continuous affective controllers: non-linear emotion representations in text encoders makes prompt interpolation poorly calibrated, so accurate and continuous VA control remains unreliable. To fill this gap, we introduce Barycentric Guidance, a training-free framework that turns pretrained foundation image editors into continuous affective controllers over VA, requiring no finetuning, extra data, or model changes. Our method maps any target VA point to simplex-based guidance weights, enabling VA controllability in three aspects: (i) accurate targeting of VA states, (ii) broad coverage across the VA plane, and (iii) continuous control. Across three face datasets, we improve VA target accuracy by 25-29%, expand VA-plane coverage by 28-50%, and increase expression diversity by 29-49%, over the strongest baseline. Unlike prior domain-specific methods, our training-free approach exploits the foundation model's inherent versatility, enabling generalization beyond human faces to animals, artworks, and complex scenes within a single framework. Code is available at https://github.com/harveymannering/BarycentricGuidance.