MapRoute++: Surrogate-Guided Semantic Routing for Visual Concept Unlearning

AI summary

Overview

• Research area: Computer Vision, specifically visual concept unlearning (concept erasure) in text-to-image diffusion models, with a focus on the Gen μ 2.0 Challenge Task 3.
• Technical level: Intermediate. The paper assumes familiarity with diffusion models, text encoders, cross-attention, and embedding-space manipulation, but the core idea is explainable in plain terms.
• Scope: The paper describes a compute-efficient, modular method for removing targeted concepts from Stable Diffusion v1.4 without retraining the underlying model, and reports results on the official Gen μ 2.0 challenge benchmark using the Erasing-Retention-Robustness (ERR) metric.

What This Paper Is About

Text-to-image diffusion models can generate harmful or unwanted content, such as copyrighted styles or identity-related imagery, and removing one concept without damaging the model's other abilities is difficult. This paper proposes MapRoute++, which inserts small per-concept "mapper" modules between the frozen text encoder and the U-Net denoiser to redirect a target concept toward a hand-chosen substitute concept while leaving everything else nearly unchanged. The goal is robust erasure under direct, indirect, and adversarial prompts while preserving unrelated and semantically adjacent concepts.

Key Contributions

1. Task-specific training objectives: A two-stage training scheme where the mapper first learns identity mapping across a broad concept vocabulary (Stage 1) and then learns target-to-surrogate mapping with regularization terms (Stage 2). The second regularization term, applied over a curated set of proper names, is retained for artistic styles to protect artist identities but omitted for objects.
2. Richer concept representations: Each target concept is represented by multiple synonyms and paraphrases rather than a single keyword, which the authors say encourages genuine semantic removal and better generalization to unseen or adversarial prompts.
3. Dynamic semantic routing: At inference, the framework computes semantic similarity between the input prompt and all target concepts using the mean of the prompt's token embeddings, then sequentially applies the top-k most relevant mapper modules. Mappers are trained on sentence-level embeddings but applied token-wise to the full embedding sequence, keeping them compatible with the pretrained diffusion model.
4. A new benchmark result: On the official Gen μ 2.0 benchmark, evaluated with ERR on Stable Diffusion v1.4, the method reaches an average ERR of 0.721, which the paper reports as a 12.1% improvement over the strongest baseline.

Main Findings

• Overall average ERR of 0.721: MapRoute++ scores 0.721 average ERR across the five categories, compared to 0.600 for FADE, 0.591 for MapRoute, 0.477 for ESD, 0.472 for CA, and 0.456 for FMN. The paper describes this as a 12.1% absolute improvement over the strongest baseline, FADE.
• Large gains on scenes: In the Scene category, MapRoute++ scores 0.707 versus 0.190 for MapRoute — the paper calls this a margin of more than 50% improvement. Scene baselines were 0.461 (ESD), 0.422 (CA), 0.443 (FMN), and 0.589 (FADE).
• Style remains the hardest category: MapRoute++ scores 0.574 in Style, versus 0.478 (ESD), 0.509 (CA), 0.482 (FMN), 0.609 (FADE), and 0.494 (MapRoute). The authors attribute this to styles relying on global texture, brushwork, and composition that semantic token embeddings alone do not fully capture.
• Improvements are category-dependent, not universal: MapRoute++ exceeds MapRoute on Style (0.574 vs. 0.494) and Scene (0.707 vs. 0.190) but scores lower than MapRoute on Object (0.863 vs. 0.919), Animal (0.604 vs. 0.796), and Action (0.857 vs. 0.871).
• Per-concept averages across 20 concepts: Target forgetting 0.7575, retention of unrelated concepts 0.9462, retention of adjacent concepts 0.7633, robustness to indirect prompts 0.8100, robustness to adversarial prompts 0.8725, with an average ERR of 0.7245 in the per-concept table (versus 0.721 in the category table).
• Best-performing individual concepts: Golf Ball reaches ERR 0.9738, Eating 0.9710, Dancing 0.9234, Labrador Retriever 0.9098, Apple Fruit 0.9035, Sleeping 0.9005, Wedding 0.9000, Aurora Borealis 0.8995.
• Weakest individual concepts: Blue Jay reaches ERR 0.2990 (adjacent-concept score 0.083), Doodle 0.3558, and Van Gogh 0.3926 (target forgetting 0.300, indirect 0.250, adversarial 0.300).
• Failure modes identified: Van Gogh imagery retains recognizable swirling brushwork and Starry Night-like structure under direct, indirect, and adversarial prompts, while Blue Jay mappers convert adjacent bird prompts into texture-like patterns and still allow bird-like output under adversarial prompting.
• Retention is consistently high: The retention axis (A_ret) ranges from 0.9342 to 0.9895 across the 20 concepts, indicating unrelated concepts are largely untouched.

Methodology in Plain English

The approach freezes the diffusion model entirely and trains only a small residual multilayer perceptron for each concept to erase. That module sits between the frozen text encoder and the U-Net denoiser and edits the text embedding before it conditions image generation.

Training happens in two stages. In Stage 1, the module is trained to output exactly what it receives — an identity function over a broad concept vocabulary — so it starts from a state of "change nothing." In Stage 2, it is trained to push the target concept's embedding toward the embedding of a chosen substitute concept. Two extra terms keep this from spilling over: one reinforces the identity behavior, and another does the same over a curated set of proper names to protect celebrity and identity-related concepts. The balance weights for these terms default to 1. For artistic styles, the proper-name protection term is kept; for objects, it is dropped.

Instead of a single keyword, each concept is described with multiple synonyms and paraphrases. At inference, the system measures similarity between the incoming prompt and every target concept using the mean of the prompt's token embeddings, and applies the top-k best-matching mappers in sequence. It does so token-wise across the whole embedding sequence, so the learned edits feed directly into cross-attention.

The substitution concept matters more in MapRoute++ than in the original MapRoute. The authors choose substitutes heuristically using only general semantic reasoning about the target — not using the challenge dataset's indirect prompts, adversarial prompts, or adjacent-concept prompts — which keeps the setting conservative and avoids evaluation leakage. A good substitute is visually plausible, does not contain the target, and is neither so close that target cues return nor so distant that neighboring concepts are damaged.

Evaluation uses Stable Diffusion v1.4. The authors generate 20 images per concept, giving 20 concepts in total, and score each image with LLaVA (llava-v1.6-mistral-7b-hf) using one constrained yes/no question for all axes: "Does this image show <concept description>? Answer with 'yes' or 'no'." For direct, indirect, and adversarial prompts, "no" is the success answer; for adjacent and retained prompts, "yes" is success. The five axes — target forgetting, retention of unrelated concepts, retention of adjacent concepts, robustness to indirect prompts, robustness to adversarial prompts — are combined via harmonic mean into the final ERR score. All experiments ran on two Nvidia GeForce RTX A6000 (48GB) GPUs using the official MapRoute repository's parameters.

Why This Matters

The work addresses a practical safety problem: diffusion models absorb undesirable content during pretraining, and removing it by fine-tuning the whole model is expensive and risks degrading everything else. MapRoute++ shows that lightweight, per-concept edits plus input-conditioned routing can achieve competitive erasure without modifying the underlying generator. Its honest reporting of failure cases — particularly artistic styles and adjacent-concept damage for Blue Jay — makes it useful as a diagnostic of where embedding-space editing breaks down.

Real-world applications:

• Copyright and artist-style compliance: Removing artistic styles such as Van Gogh, Monet, Doodle, Neon, and Sketch from a deployed image generator, using the proper-name protection term specifically to avoid collateral erasure of artist identities.
• Bias and responsible deployment: Selectively suppressing unwanted societal bias concepts before shipping a generative model.
• Content-creator and art-generation tools: Preserving general image-generation quality (retention scores of 0.9342–0.9895) while blocking designated concepts in a production pipeline.
• Data augmentation and content creation pipelines: Reusing frozen diffusion checkpoints with modular, swappable mapper modules rather than maintaining separately fine-tuned models per policy.

Industry relevance: The method is compute-efficient by design — no diffusion-weight fine-tuning, small MLP modules — and the router makes a single deployed model handle many concept policies at once. That combination is attractive to organizations that need to adjust content policies after deployment, and is relevant to the copyright, safety, and model-governance concerns the paper cites.

Future Directions

• Improve style erasure: Since style concepts depend on global texture and composition, and MapRoute++ edits token embeddings rather than the U-Net denoiser's cross-attention weights, an open question is whether limited cross-attention fine-tuning closes the Style gap (MapRoute++ 0.574 versus FADE 0.609).
• Better adjacent-concept preservation: Blue Jay's adjacent-concept score of 0.083 and the collapse of adjacent birds into texture-like images indicate that retrieval or generation of adjacent concepts needs a stronger constraint than the current identity-preservation terms.
• Stronger surrogate-selection procedure: The paper relies on heuristic, leakage-free surrogate choice; a principled method for picking substitutes could reduce both under-erasure and over-erasure failure modes without consulting benchmark prompts.
• Generalization beyond the studied setting: All experiments use Stable Diffusion v1.4 and the 20-concept Gen μ 2.0 dataset; whether the approach and its routing mechanism transfer to other diffusion backbones or much larger concept vocabularies is not reported.

Target Audience

Researchers and practitioners working on diffusion model safety, machine unlearning, and responsible generative AI will benefit most, particularly those familiar with embedding-space editing and cross-attention conditioning. It is also useful for engineers who need to implement post-deployment content policies in image generation systems, and for benchmark designers interested in the five-axis ERR protocol and its harmonic-mean aggregation.

Authors’ abstract

We present our submission to Task 3 of the Gen$μ$ 2.0 Challenge on visual concept unlearning. Building on MapRoute, we introduce task-specific training objectives, richer concept representations, and semantic routing for concept-specific mapper selection. Our approach improves robust concept removal while preserving unrelated and semantically adjacent concepts. On the official benchmark, evaluated using the Erasing-Retention-Robustness (ERR) metric on Stable Diffusion v1.4, our method outperforms the state-of-the-art baseline by 12.1\% on average across the five concept categories, achieving substantial gains.