Preventing Shortcuts in Adapter Training via Providing the Shortcuts

AI summary

Overview

• Research area: Computer vision, specifically text-to-image (T2I) personalized and stylized image generation via lightweight adapters on frozen foundation models.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, LoRA, ControlNet, IP-Adapter, and single-image reconstruction training objectives.
• Scope: The paper introduces Shortcut-Rerouted Adapter Training (SR), a training paradigm that explicitly routes confounding factors through auxiliary modules during adapter training—then discards those modules at inference—to prevent identity adapters from absorbing pose, expression, and distribution-shift shortcuts.

What This Paper Is About

Adapters such as LoRAs and encoder-based modules are trained to inject a specific attribute (like a person's identity) into a frozen image generator, but they are trained with a single-image reconstruction objective that rewards reproducing everything in the input image. As a result, adapters entangle the target identity with incidental factors such as head pose, facial expression, lighting, background, and dataset-specific style, which hurts prompt adherence and image quality. The paper's goal is to train adapters that inject only the intended target factor, by deliberately providing the confounding factors through auxiliary modules during training so the adapter has no incentive to learn them.

Key Contributions

1. A general training paradigm called Shortcut Rerouting (SR). Instead of hoping an adapter self-disentangles, the method routes confounding factors $C$ through an auxiliary Shortcut-Rerouting module $\mathcal{S}_C$, which is frozen and then removed at inference time, leaving the adapter $\mathcal{A}$ to specialize only in the target factor $T$.

2. Two practical instantiations using established tools. SR-LoRA absorbs dataset-level distribution shift between the foundation model and the finetuning data. SR-CN uses a pre-trained ControlNet to absorb pose and expression leakage from input images.

3. Empirical validation in two distinct settings. The method is evaluated on facial personalization and full-body personalization against baselines including InfU, PuLID, an IP-Adapter (IPA) trained without shortcut rerouting, and the open-source InstantX/FLUX.1-dev-IP-Adapter.

4. Extension to background and combined shortcuts. Additional variants (SR-LoRA-CN, SR-LoRA-BG, SR-LoRA-CN-BG) show the framework supports different combinations of shortcut modules, isolating identity by aligning both pose and background with the prior.

Main Findings

• Head pose control improves substantially. In the "face" setting, SR-CN IPA achieves a Head Pose error of 12.6755, versus 16.1199 for IPA, 17.5345 for PuLID, and 17.7139 for InfU (lower is better).

• Prior preservation improves most. Prior (LPIPS) drops to 0.3937 for SR-CN IPA and 0.4330 for SR-LoRA IPA, compared with 0.4800 for IPA, 0.4584 for PuLID, and 0.4490 for InfU (lower is better).

• Expression alignment improves. SR-CN IPA reaches EMOCA Expr. 0.5800 and LLM Expr. 3.6934, versus 0.3470 and 3.0714 for the vanilla IPA. Only InfU has a higher LLM Expr. score (3.7664).

• Identity fidelity stays competitive but is not uniformly best. SR-CN IPA reaches LLM Id. 4.7941 (slightly above IPA's 4.7929 and PuLID's 4.2826), while its FaceNet Id. of 0.7118 is below PuLID's 0.7742 and InfU's 0.7402. SR-LoRA IPA records the lowest FaceNet Id. of the compared methods at 0.6708.

• Full-body personalization improves across every reported metric. SR-CN IPA records LLM Id. 4.6510, FaceNet Id. 0.5857, LLM Expr. 3.5263, EMOCA Expr. 0.4794, Head Pose 18.05, Body Pose 137.6888, and Prior (LPIPS) 0.4133—better than both IPA (4.5986 / 0.5733 / 3.3000 / 0.3466 / 20.70 / 167.4000 / 0.4566) and InstantX (2.9930 / 0.3533 / 3.4736 / 0.4687 / 25.97 / 186.7454 / 0.5075).

• Qualitative behavior matches the quantitative story. Without shortcut rerouting, a smiling reference image causes IPA to ignore a prompt-specified "snarling" expression; SR-trained adapters respond to prompt-specified expressions and head poses and produce more coherent full-body generations.

• Distribution shift can be absorbed with a LoRA shortcut. A LoRA pre-trained on the finetuning dataset absorbs dataset-specific style and lighting, letting the identity adapter generalize beyond the training domain.

• The framework accepts layered shortcuts. Combining LoRA, ControlNet, and background modules lets the system match pose and background of the prior simultaneously, isolating only identity.

• Code and data are not released. The paper's checklist states that code and dataset release was not possible at submission time due to legal and compliance constraints within the authors' organization, though the paper reports that detailed methodology, dataset, and evaluation descriptions are provided.

Methodology in Plain English

The authors start from a formal observation: an image $X$ depends on a target factor $T$ (identity or style) and confounding factors $C$ (pose, expression, lighting, dataset domain). Standard adapter training minimizes reconstruction error between the generated image and the input image, which pushes the adapter to encode both $T$ and $C$.

Shortcut Rerouting changes the training pipeline so that the confounds are supplied through a separate, dedicated module. The generator is trained to produce $\hat{X} = G(\mathcal{A}(X), \mathcal{S}_C(C))$, where $\mathcal{S}_C$ is a pre-trained frozen module that already explains the confounds. Because pose, expression, or domain style are already accounted for, the adapter gets no gradient benefit from duplicating them. At inference, $\mathcal{S}_C$ is deleted, leaving generation governed only by $G(\mathcal{A}(X))$—a disentangled adapter inside the original model.

Two module choices operationalize this. SR-LoRA pretrains a lightweight LoRA on the finetuning dataset (for example, studio-lit identity images) to absorb domain-specific style and low-level features, freezes it, and trains only the identity encoder. SR-CN uses a pre-trained ControlNet conditioned on pose and expression maps extracted from training images (via pose estimation and landmark detection), so the adapter absorbs identity alone.

Experiments use an internal dataset of a few million high-quality human images, filtered to single-subject photos with low-quality, NSFW, and watermarked content removed, bucketed by aspect ratio, and cached with landmarks, segmentation masks, and text embeddings. Captions come from Qwen2.5-14B (text) and InternViT-300M-V2.5 (vision). Face crops are extracted and aligned with facial landmarks; body crops come from segmentation with background removal. All methods are built in PyTorch with HuggingFace Diffusers on FLUX.1 [Dev] with a DiT backbone and a Conditional Flow Matching objective, trained on 8× A100 GPUs (80GB each) with AdamW at a learning rate of 5e-5 and global batch size 32 for 250K iterations. Inference uses IP scale 1.0, CFG 3.5, 28 steps, and 1024×1024 resolution, with openai/clip-vit-large-patch14 for identity encoding.

Evaluation uses seven metrics: FaceNet Id., LLM Id., LLM Expr., EMOCA Sim., Head Pose (via HopeNet), Body Pose (via HRNet), and Prior (LPIPS).

Why This Matters

The paper reframes disentanglement in adapter training as a design-principle question: rather than fighting spurious correlations with the reconstruction loss, deliberately build a pathway for them. If the principle generalizes beyond personalization, it could change how modular generative systems are constructed when pretrained controllers already exist for the nuisance factors.

Real-world applications implied by the work:

• Personalized avatar and portrait generation, where users want their identity injected while text prompts still control expression and head pose.
• Full-body character consistency for storytelling, advertising, or social media content, where body type, clothing, and limb proportions must persist while body pose changes.
• Stylized text-to-image synthesis that reuses a subject across scenes without leaking the lighting, background, or studio style of the reference photograph.
• Responsible deepfake mitigation research, given the paper's own ethical note that better identity preservation and expression control could lower the barrier for misuse, motivating model watermarking and usage restrictions.

Industry relevance: The work comes from Snap Inc. and targets the exact production setting of fine-tuning proprietary foundation models—such as FLUX.1 [Dev]—on curated internal datasets whose distribution differs from the backbone's unknown training distribution. The SR-LoRA shortcut is specifically motivated by this opaque-model, opaque-data scenario, which is common in commercial personalization pipelines.

Future Directions

• Extending beyond encoder-based adapters. The authors note that their current evaluation focuses on encoder-based adapters, and that the approach could in principle apply to LoRA training—for example, learning a style LoRA free of layout or content shifts.

• Applying SR to stronger baselines. The paper applies Shortcut Rerouting only to IP-Adapter, which the authors describe as "a fairly simple baseline"; stronger baselines might yield overall better performance.

• Composing richer shortcut-module stacks. The paper demonstrates SR-LoRA-CN, SR-LoRA-BG, and SR-LoRA-CN-BG variants and points to the Appendix for ablations of their complementary effects, leaving the full design space of combined modules open.

• Generalizing the principle past personalization. The authors argue the idea of absorbing confounding variation through targeted pathways has implications for modular, interpretable, and more controllable generative systems broadly.

Target Audience

Researchers and engineers working on text-to-image personalization, adapter fine-tuning, and controllability of diffusion or flow-matching generative models will gain the most. It also suits practitioners who fine-tune proprietary foundation models on narrow internal datasets and need identity and style injection that still respects text prompts, as well as researchers interested in shortcut learning and disentangled representation learning more generally. Readers need a working knowledge of LoRA, ControlNet, and encoder-based adapters such as IP-Adapter to follow the methodological detail.

Authors’ abstract

Adapter-based training has emerged as a key mechanism for extending the capabilities of powerful foundation image generators, enabling personalized and stylized text-to-image synthesis. These adapters are typically trained to capture a specific target attribute, such as subject identity, using single-image reconstruction objectives. However, because the input image inevitably contains a mixture of visual factors, adapters are prone to entangle the target attribute with incidental ones, such as pose, expression, and lighting. This spurious correlation problem limits generalization and obstructs the model's ability to adhere to the input text prompt. In this work, we uncover a simple yet effective solution: provide the very shortcuts we wish to eliminate during adapter training. In Shortcut-Rerouted Adapter Training, confounding factors are routed through auxiliary modules, such as ControlNet or LoRA, eliminating the incentive for the adapter to internalize them. The auxiliary modules are then removed during inference. When applied to tasks like facial and full-body identity injection, our approach improves generation quality, diversity, and prompt adherence. These results point to a general design principle in the era of large models: when seeking disentangled representations, the most effective path may be to establish shortcuts for what should NOT be learned.