Directional Textual Inversion for Personalized Text-to-Image Generation

AI summary

Overview

Research area: Personalized text-to-image generation, specifically embedding-based personalization (Textual Inversion) and the geometry of token embedding spaces in pre-norm Transformers.

Technical level: Advanced. The paper combines a diffusion-model personalization pipeline with manifold optimization (Riemannian SGD, von Mises-Fisher priors) and a theoretical analysis of pre-norm Transformer blocks. The core idea is intuitive, but the formalism assumes familiarity with Transformer internals and embedding-space geometry.

Scope: A single paper (arXiv:2512.13672v2, cs.LG) that diagnoses norm inflation in Textual Inversion, proposes optimizing only embedding direction at an in-distribution magnitude, and validates the method on SDXL and SANA 1.5 across subject-fidelity, text-alignment, ablation, human-evaluation and interpolation experiments.

What This Paper Is About

Textual Inversion (TI) personalizes a text-to-image model by learning a new token embedding for a user-supplied concept, but it often fails when prompts are complex — the subject appears while details like style, background or secondary objects are dropped. The authors trace this to embedding norm inflation: the learned token drifts to magnitudes far outside the pre-trained vocabulary's range, which degrades prompt conditioning. Their goal is a personalization method that keeps semantic information (which they argue lives in the embedding's direction) while eliminating the harmful magnitude drift.

Key Contributions

1. An interpretability-driven diagnosis of Textual Inversion. The paper empirically shows that semantic information in CLIP token space is carried primarily by direction (cosine neighbors of "apple" are "apples", "fruit", "peach", "pear", "egg", whereas Euclidean neighbors are semantically unrelated), and that standard TI produces token norms often above 20 versus roughly 0.4 for the pre-trained vocabulary.

2. A theoretical account of why large norms hurt. For a pre-norm Transformer block, the authors decompose a token as x⁽⁰⁾ = m·v + p and prove two effects: positional information contributes only O(1/m) to the normalized signal (Lemma 1), and residual updates cause directional change shrinking in proportion to 1/‖x⁽ˡ⁾‖ (Lemma 2). Proposition 1 and Corollary 1 bound the accumulated directional drift across L blocks and show it goes to 0 as the initial token is scaled up.

3. Directional Textual Inversion (DTI). DTI fixes the embedding magnitude to an in-distribution value (e.g., the average vocabulary norm) and optimizes only the unit-sphere direction using Riemannian SGD with tangent-space projection and retraction, replacing AdamW.

4. A MAP formulation with a von Mises-Fisher directional prior. The prior makes the objective L(v) = L_data(m*v) + L_prior(v) with L_prior(v) = -κ·μᵀv, whose Euclidean gradient is the constant -κμ — cheap to add, numerically stable, and interpretable as a constant pull toward a semantically meaningful direction.

Main Findings

• Direction encodes semantics; magnitude does not. For the token "apple", cosine-similarity neighbors are "apples", "fruit", "peach", "pear", "egg"; Euclidean neighbors are tokens such as "U+2069", "altrin", "lestwe", "heartnews", "samanthaprabhu". The same pattern is reported for "study" and "writing" in the appendix.

• TI inflates norms and drifts semantically. Learned TI embeddings reach norms often greater than 20 against approximately 0.4 for the pre-trained vocabulary, and learned tokens move directionally away from related concepts such as "cat" and "kitten".

• Effect I confirmed — positional information is attenuated. A 2-layer MLP classifier trained on the frozen text encoder reaches 100% accuracy at predicting a token's absolute position from LN(e + p) on unmodified inputs. Accuracy deteriorates rapidly once the scaling factor m (tested at 0.5, 1, 2, 4, 8, 16) exceeds the encoder's natural scale. TI embeddings with large norms collapse to near-zero positional accuracy, while DTI embeddings remain fully recoverable.

• Effect II confirmed — residual updates stagnate. Measuring the angle between hidden states entering and exiting each pre-norm block, the average per-block angular change was 21.33° for TI embeddings versus 33.52° for DTI embeddings, a 1.57× larger change.

• DTI raises text fidelity. On SDXL, DTI scores 0.522 on text alignment versus 0.292 for TI, 0.466 for TI-rescaled, and 0.464 for CrossInit. On SANA 1.5-1.6B, DTI reaches 0.744 versus 0.621 (TI), 0.655 (TI-rescaled), 0.614 (CrossInit). On SANA 1.5-4.8B, DTI reaches 0.757 versus 0.646 (TI), 0.548 (TI-rescaled), 0.622 (CrossInit).

• Subject similarity is maintained, not maximized. On SDXL, DTI's image score is 0.450 versus TI's 0.561, but above TI-rescaled's 0.243. On SANA 1.5-1.6B, DTI scores 0.479 versus TI's 0.480; on SANA 1.5-4.8B, 0.452 versus TI's 0.446. The authors state DTI primarily improves text prompt fidelity because it does not directly optimize subject similarity.

• Simply rescaling TI is not enough. Rescaling TI's learned embeddings to the average vocabulary norm improves text fidelity but degrades image similarity (for example, SDXL text 0.292 → 0.466 while image 0.561 → 0.243).

• Every design choice matters. In the ablation: AdamW with a projection back onto the sphere gives 0.335 image / 0.463 text, while RSGD with the same setting gives 0.450 / 0.522; setting the magnitude to the minimum collapses performance (0.030 / 0.074); an out-of-distribution magnitude of 5.0 gives 0.383 / 0.373; removing the prior (κ = 0) gives 0.507 / 0.436; a higher κ of 0.5×10⁻³ gives 0.278 / 0.688. The best balance is κ = 0.1×10⁻³ with the mean magnitude.

• Human evaluators prefer DTI. In a study with 100 participants on Amazon Mechanical Turk, each answering 20 questions (10 on subject fidelity, 10 on image-text alignment), with four non-compliant responses excluded, DTI scored 43.45 on image fidelity and 66.77 on text alignment, versus CrossInit's 42.87 and 22.40 and TI's 13.78 and 10.83.

• Hyperspherical parameterization enables interpolation. DTI supports spherical linear interpolation (SLERP) between learned concepts, producing coherent blends (dog–teapot, dog–cat, young boy–older woman); the linear interpolation used by TI often fails to produce coherent intermediates.

• Training cost is modest. All experiments ran on a single NVIDIA A6000 GPU, with approximately 7 minutes of training per subject for SDXL-base and 30 minutes for SANA 1.5-1.6B.

Methodology in Plain English

The authors start by asking where a learned personalized token goes wrong. They inspect the geometry of CLIP token embeddings and compare two ways of finding a token's nearest neighbours — Euclidean distance (sensitive to both length and angle) and cosine similarity (sensitive only to angle). Cosine wins on semantic coherence, which suggests direction carries meaning.

They then analyse a standard pre-norm Transformer block, where the input is added to a normalized sub-layer output. Because normalization divides by the signal's size, a token with an enormous magnitude makes the small additive positional term nearly invisible, and it also makes each layer's bounded update tiny relative to the token — so the token's direction barely changes as it passes through the stack. They test both predictions: one experiment trains a small classifier to recover a token's position after normalization, and another measures the angle by which hidden states rotate inside each block.

With the diagnosis in hand, they change the parameterization. Instead of learning a free vector, they write the embedding as a fixed magnitude times a unit direction, choose that magnitude from the frozen vocabulary's own statistics, and optimize only the direction. Because the direction lives on a unit sphere, ordinary Euclidean optimizers would step off the surface, so they use Riemannian SGD: compute the gradient, project it into the tangent space at the current point, and retract the step back onto the sphere by normalizing.

They also add a Bayesian flavour. Treating the direction as a parameter with a von Mises-Fisher prior — the sphere's analogue of a Gaussian — turns the objective into a data loss plus a simple regularization term -κμᵀv, where μ is the normalized embedding of a related class token (such as "dog") and κ controls the pull strength. The gradient of this term is just a constant vector, so it is added before projection and costs no extra computation. The authors grid-searched κ over 5×10⁻⁵ to 2×10⁻⁴ and fixed it at 1×10⁻⁴ for all experiments.

Evaluation uses all reference images from the DreamBooth dataset, 40 prompts (the full DreamBooth set plus 10 additional complex prompts), 4 random seeds per prompt for 160 samples per instance, DINOv2 cosine similarity for subject fidelity, and SigLIP for image-text alignment. Baselines are TI and CrossInit in the main tables, with P+, NeTI, CoRe and DCO in the appendix; models are SDXL, SANA 1.5-1.6B and SANA 1.5-4.8B (the latter using Gemma as the text encoder and DiT as the image generator).

Why This Matters

Impact on research. The paper reframes a widely used but under-theorized method. Rather than adding capacity to the embedding space — the route taken by P+, NeTI and similar variants — it identifies a specific optimization pathology and fixes it geometrically. The theoretical claims (positional attenuation scaling as O(1/m), directional freezing under scaling) are stated as general properties of pre-norm blocks, so they are testable beyond personalization. The observation that direction is the semantically meaningful component also connects embedding-based personalization to a broader line of work on hyperspherical constraints in representation learning.

Real-world applications:

• Custom image generation for individuals, brands or products where the user's prompt mixes a personal subject with specific style, setting and secondary objects.
• Concept blending for creative and advertising work, using SLERP to morph smoothly between two learned subjects such as a dog and a teapot.
• Face personalization workflows where age, appearance or identity must vary along a coherent progression.
• Style transfer and subject-style generation, which the authors report on in the appendix using StyleDrop and FFHQ imagery.

Industry relevance. TI-style personalization is attractive because it stores only a single token embedding per concept and integrates into existing pipelines. DTI keeps that storage and workflow advantage while making the pipeline more robust on complex prompts, and it trains in roughly 7 minutes per subject on SDXL-base on one A6000. The failure mode it eliminates — a personalized token that "shouts over" the rest of the prompt — is exactly the kind of controllability gap that blocks production deployment of prompt-driven customization.

Future Directions

• Generalization to other encoder architectures. The analysis assumes pre-norm blocks with LayerNorm/RMSNorm; the authors explicitly flag investigating encoders with different normalization or positional encoding schemes as open work.
• Recovering subject fidelity. Because DTI does not directly optimize subject similarity, the authors suggest combining it with complementary lightweight fine-tuning methods such as LoRA, and show this qualitatively in the appendix.
• Better prior selection. Estimating the vMF concentration parameter κ is described as non-trivial; the authors treat it as a hyperparameter, fix it at 1×10⁻⁴, and defer fuller discussion to the appendix.
• Beyond text-to-image. The introduction notes that the TI paradigm is shared with LLMs and VLMs, raising the question of whether direction-only optimization transfers to prompt tuning in those settings.
• Scaling behaviour. The paper reports that DTI's advantage over baselines becomes more pronounced as the model size increases (from SANA 1.5-1.6B to SANA 1.5-4.8B), which invites further study at larger scales.

Target Audience

This paper is most useful to researchers and engineers working on personalization and controllable generation with diffusion models, particularly those who work directly with Textual Inversion or prompt-tuning pipelines. It also speaks to anyone interested in embedding-space geometry, norm and hyperspherical constraints, or the internal behaviour of pre-norm Transformer blocks. Practitioners looking for a drop-in improvement to an existing TI workflow will find the algorithmic description and released code actionable, while readers seeking a conceptual, non-mathematical treatment of personalization will find the theoretical sections dense.

Authors’ abstract

Textual Inversion (TI) is an efficient approach to text-to-image personalization but often fails on complex prompts. We trace these failures to embedding norm inflation: learned tokens drift to out-of-distribution magnitudes, degrading prompt conditioning in pre-norm Transformers. Empirically, we show semantics are primarily encoded by direction in CLIP token space, while inflated norms harm contextualization; theoretically, we analyze how large magnitudes attenuate positional information and hinder residual updates in pre-norm blocks. We propose Directional Textual Inversion (DTI), which fixes the embedding magnitude to an in-distribution scale and optimizes only direction on the unit hypersphere via Riemannian SGD. We cast direction learning as MAP with a von Mises-Fisher prior, yielding a constant-direction prior gradient that is simple and efficient to incorporate. Across personalization tasks, DTI improves text fidelity over TI and TI-variants while maintaining subject similarity. Crucially, DTI's hyperspherical parameterization enables smooth, semantically coherent interpolation between learned concepts (slerp), a capability that is absent in standard TI. Our findings suggest that direction-only optimization is a robust and scalable path for prompt-faithful personalization. Code is available at https://github.com/kunheek/dti.