X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesis

AI summary

Overview

• Research area: Text-to-image (T2I) diffusion models, controllable image generation, imaging factor disentanglement, and evaluation metrics for generative models.
• Technical level: Advanced. The paper assumes familiarity with diffusion models, Textual Inversion, ControlNets, and vision-language models.
• Scope: The paper proposes X-MULTI, which adds zero-shot vision-language model (VLM) supervision to the MULTI method so that text-to-image models can generate novel combinations of camera lenses, sensors, viewpoints, and domains, and proposes I-FAA, a replacement for MULTI's Factor Alignment Accuracy (FAA) metric that reduces cross-factor correlation leakage.

What This Paper Is About

Text-to-image models can follow natural-language descriptions but struggle to independently control low-level image acquisition properties such as camera lens type, sensor modality, viewpoint, and domain. The goal of this paper is to let a model synthesize valid new combinations of these factors that never appeared together in training data (for example, a fisheye lens paired with an event sensor). The authors also argue that the standard metric used to measure this ability, FAA, is itself misleading because its factor classifiers exploit correlations between factors, and they introduce a replacement metric, I-FAA.

Key Contributions

1. X-MULTI approach: Introduces zero-shot VLM-based supervision on synthetically generated unseen imaging-factor combinations, providing explicit factor-level supervision beyond reconstruction-only training.
2. I-FAA metric: Introduces an improved factor-alignment evaluation metric that reduces cross-factor shortcut learning through class-balanced training and factor-specific augmentations.
3. Diagnosis of FAA: Demonstrates through Cramér's V correlation analysis, Grad-CAM attention maps, and ground-truth factor prediction comparisons that FAA suffers from cross-factor correlation leakage, and that its classifiers are inaccurate on ground-truth DF-RICO images.
4. Empirical evaluation on DF-RICO: Reports improved factor alignment for novel combinations compared to MULTI and other baselines, both with and without ControlNet structural guidance.

Main Findings

• Novel combinations improve: Averaged I-FAA on novel factor combinations is 0.47 for MULTI and 0.53 for X-MULTI, a 0.06 absolute improvement, which the authors describe as 11% relative. The largest gains are reported for lens and viewpoint factors, with 21% and 18% relative improvement respectively; the paper states that sensor and domain each show a relative improvement of 0.01.
• Overall best disentanglement with and without ControlNet: Under I-FAA on novel combinations (Table 5), X-MULTI achieves the best overall factor disentanglement, the best CLIP alignment with factors, and the best Diversity Score, while other metrics remain mixed across methods. The table caption states that X-MULTI with ControlNet guidance achieves the best factor adherence.
• Qualitative failures of baselines: SDXL Zeroshot fails to reflect target factors. When substituting an event sensor with a gated one, DreamBooth retained event artifacts while the proposed method transformed the image cleanly; when shifting the domain from real to video-game, the proposed method eliminated real-world traces while MULTI and DreamBooth could not. X-MULTI shows less proficiency with the thermal sensor but handles other sensors competently.
• VLM reliability is uneven: Used zero-shot with structured factor-specific prompts, Qwen2-VL-7B-Instruct achieves an average classification accuracy of approximately 0.78 across factors. The rgb-thermal sensor is strongly confused with rgb, and viewpoints other than front are inconsistently predicted, so the supervision strength is set to zero for rgb-thermal and for all viewpoints except front.
• VLM backbone and prompt design matter: Qwen2-VL-7B-Instruct reaches 0.78 average accuracy versus 0.51 for LLaVA-1.6. Detailed factor-specific prompts with per-class descriptions reach 0.78, simple prompts without descriptions reach 0.70, and a single unified prompt classifying all factors jointly drops to 0.63.
• Supervision strength: Moderate (10⁻⁶) and weak (10⁻⁸) supervision both reach an I-FAA of 0.53 on novel combinations, while overly strong supervision (10⁻⁴) degrades performance to 0.47.
• I-FAA classifiers are far more accurate on real data: On ground-truth DF-RICO images, FAA scores 0.23 (Lens), 0.83 (Sensor), 0.50 (Domain), 0.17 (View), averaging 0.43, while I-FAA scores 1.00, 0.97, 0.98, 0.84, averaging 0.95. The largest gains are for viewpoint and domain.
• Correlation leakage is reduced but not eliminated: FAA exhibits strong coupling for lens-viewpoint, domain-viewpoint, and lens-domain factor pairs. I-FAA reduces most of these correlations, but residual correlations remain, especially for strongly coupled or sparsely represented factor pairs.
• Existing combinations are preserved: On combinations present in training data (Table S.2), X-MULTI achieves the highest I-FAA while remaining competitive on image generation metrics.
• Attention maps become more meaningful: Grad-CAM visualizations show I-FAA attending to lens boundary curvature and geometric distortion, perspective-specific regions for viewpoint, and modality-specific and scene-appearance cues, whereas FAA frequently attends to unrelated or inconsistent regions.

Methodology in Plain English

The authors build on MULTI, which represents each imaging factor value with a learnable token made of 15 embedding vectors inserted into a text prompt, and trains those tokens with the standard diffusion reconstruction loss while keeping the diffusion backbone frozen. MULTI learns in two stages: general factor embeddings, then adaptation to a target dataset's specific factor values. X-MULTI keeps this two-stage structure and adds VLM supervision only at Stage 1.

During training, after an initial diffusion-only warm-up, each step samples a batch of real data and constructs an extra synthetic factor tuple by resampling factor values across categories. The current diffusion model generates an image for that synthetic tuple, and a frozen zero-shot VLM is queried with factor-specific prompts to predict each factor; a cross-entropy loss between the predicted and requested factors is added to the diffusion loss, weighted by a strength parameter. The model is Stable Diffusion XL, trained for 10 epochs with batch size 4, AdamW with weight decay 10⁻², and a linear warmup plus cosine annealing schedule with maximum learning rate 10⁻⁴; VLM supervision starts at epoch 3 at strength 10⁻⁶, with each mini-batch containing 3 real and 1 synthetic sample. Unreliable VLM predictions are masked out.

For evaluation, I-FAA trains one classifier per factor category. Each classifier shares a frozen DINOv3 backbone with a factor-specific trainable linear head, optimized with AdamW at learning rate 10⁻⁴, cosine annealing, and batch size 32. Two changes remove shortcuts: class-balanced under-sampling, where every class is randomly under-sampled to the size of the smallest class in that category, and non-overlapping factor-specific augmentations (for example, horizontal and vertical flips, random rotation and translation, color jitter, grayscale, noise, Gaussian blur, cutout/erasing/hide-and-seek, radial distortion, and perspective warp, distributed differently per category). The supplement gives exact parameter ranges, such as rotation up to ±5° and affine translation up to ±3% for the lens classifier, and a complete exclusion of color deviations from the sensor classifier. The evaluation uses the DF-RICO benchmark, which spans 15 autonomous driving and surveillance datasets annotated with four factor categories: camera lens (normal, fisheye), domain (real, simulation, video-game), viewpoint (front, back, side, drone, pole), and sensor (rgb, thermal, rgb-thermal, gated, event). The total number of images in DF-RICO is not reported.

Why This Matters

The work matters because collecting real sensor, lens, viewpoint, and domain data for every acquisition condition is expensive, yet vision systems must remain robust across those conditions. It also matters as a caution about evaluation: the paper shows that a widely used classifier-based disentanglement metric can report numbers that largely reflect correlations in the dataset rather than true factor control.

Real-world applications:

• Generating training or validation data for autonomous driving perception across camera modalities such as rgb, thermal, rgb-thermal, gated, and event sensors.
• Simulating viewpoint changes (front, back, side, drone, pole) for surveillance and vehicle-mounted camera systems.
• Domain transfer between real, simulation, and video-game imagery for sim-to-real pipelines.
• Guided image-to-image editing, where the paper demonstrates changing sensor or domain factors while respecting ControlNet structure (Canny and Depth).

Industry relevance: the author list includes researchers affiliated with Bosch Research and ETH Zürich, and the benchmark consists of autonomous driving and surveillance datasets, so the results speak directly to automotive perception and camera-system development.

Future Directions

• Reducing the residual cross-factor correlations that remain in I-FAA, using stronger augmentations or dedicated correlation-reduction methods.
• Improving VLM reliability, since the current supervision must mask out rgb-thermal sensors and all non-front viewpoints; better prompts or better VLMs could unlock those factors.
• Incorporating richer visual context into the factor-specific prompts to strengthen the supervision signal.
• Extending the analysis to factors and datasets beyond the four categories in DF-RICO, and validating whether the two-stage design transfers to other generative backbones.

Target Audience

Researchers and engineers working on controllable text-to-image generation, diffusion model customization, and multi-attribute disentanglement, as well as practitioners in automotive perception and synthetic data generation who need to produce images under acquisition conditions that are rare or absent in their datasets. It is also relevant to anyone who uses classifier-based generative evaluation metrics and wants to understand their failure modes.

Authors’ abstract

Imaging factor disentanglement in text-to-image generation aims to independently control image acquisition properties such as types of camera lenses, sensor types, viewpoints, and domains to enable combinatorial generalization. This should let the model synthesize novel factor combinations unobserved in the training data, such as pairing a fisheye lens with an event sensor never observed in training data. Recent work, MULTI, introduced learnable, factor-specific embeddings to disentangle imaging factors, along with the Factor Alignment Accuracy (FAA) metric to evaluate disentanglement quality. We identify and address two independent limitations. First, MULTI's pixel-level reconstruction objective supervises the model only on observed imaging factor combinations, providing no direct training signal for novel combinations. We therefore propose X-MULTI, which uses a pretrained vision-language model (VLM) to supervise novel factor combinations synthesized during training. Second, we show the FAA metric exhibits severe cross-factor correlation leakage, misrepresenting true disentanglement quality. We therefore propose Improved-FAA (I-FAA), which employs factor-specific augmentation strategies to break these correlations and enables more rigorous evaluation. Experiments demonstrate that X-MULTI achieves improved factor alignment on novel combinations compared to MULTI. Moreover, we show that correlation leakage in FAA distorts the evaluation of true factor disentanglement and I-FAA reduces this leakage and therefore provides a more robust assessment of factor alignment.