LumiCtrl : Learning Illuminant Prompts for Lighting Control in Personalized Text-to-Image Models

AI summary

Overview

Research area: Computer vision and generative AI — specifically controllability of lighting/illumination in text-to-image (T2I) diffusion models, combining color science (illuminant modeling, Planckian locus) with parameter-efficient model personalization.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, CLIP text encoders, ControlNet, cross-attention fine-tuning, and color-constancy terminology, though the core ideas can be grasped without reimplementing the math.

One-sentence scope: The paper diagnoses why standard illuminant terms fail in T2I models and proposes LumiCtrl, a method that learns dedicated "illuminant prompts" from a single image so that generated subjects can be rendered under user-specified lighting while preserving identity and spatial structure.

What This Paper Is About

Text-to-image models generate images from text prompts, but they cannot reliably obey lighting instructions such as "tungsten" or "6500K" — they default to a daylight-like appearance regardless of what the prompt asks for. The authors show this is a semantic gap in the text encoder, not a lack of visual knowledge, and they build LumiCtrl to close it by teaching the model explicit illuminant tokens rather than relying on words the encoder already mishandles.

Key Contributions

1. Diagnosis of a semantic gap in text encoders: The authors demonstrate that standard illuminant terms ("tungsten", "6500K") are not grounded in CLIP-based text embedding spaces — illuminant names do not cluster with general lighting concepts, and Kelvin values embed near ordinary numerals rather than photometric quantities.

2. First illuminant prompt learning method: LumiCtrl learns dedicated illuminant tokens (for seven illuminants) from a single image of an object, enabling direct illuminant control in prompt space rather than requiring a separate relighting model or software.

3. Edge-Guided Prompt Disentanglement: A frozen ControlNet conditioned on Canny edge maps is used during training so that the learned prompts capture illumination color rather than structural detail, mitigating what the authors call "spurious detail leakage." ControlNet is discarded at inference.

4. Masked Reconstruction Loss and Contextual Light Adaptation: A loss that weights foreground (masked) pixels more heavily than background lets the model learn the foreground illuminant precisely while allowing the background to adapt contextually, rather than forcing a uniform color shift over the whole image.

Main Findings

• Illuminant prompting fails systematically in existing T2I models: Probing 400 text prompts across 20 objects under four canonical illuminants (tungsten 2850K, fluorescent 3800K, cloudy 6500K, shade 7500K), the authors find generated images show no systematic shift toward the target illuminant and remain biased toward a default daylight prior, with high MSE between estimated and ground-truth illuminants.

• The failure originates in the text encoder: Across four CLIP-based encoders, embeddings of standard illuminant presets and Kelvin temperatures do not cluster with general lighting concepts, and Kelvin values embed near generic numerals. Silhouette scores for "Standard vs. Kelvin" and "Standard vs. General Lighting" are consistently low or negative, while "Kelvin vs. General Numeric" yields high scores.

• Quantitative superiority across all reported metrics: On 20 concepts with 42 random seeds per method, LumiCtrl with ControlNet achieves an angular error of 4.51, MSE of 16.80, and SSIM of 0.77. The best baseline angular error is 10.39 (IC-Light) and the best baseline SSIM is 0.71 (DreamBooth); the best baseline MSE is 33.20 (PnP+P2P).

• Edge guidance materially improves results: The ablation without ControlNet ("Ours w/o CtrlNet") still beats baselines, with angular error 6.87, SSIM 0.74, and MSE 22.40, but its higher MSE confirms edge guidance is important for structural integrity.

• Baseline categories fail in distinct ways: T2I personalization methods (Textual Inversion, DreamBooth, Custom Diffusion, Break-a-Scene) preserve concept identity but retain lighting from training examples. Instruct Pix2Pix distorts images or introduces artifacts; IC-Light better controls light direction but fails to preserve spatial background under foreground conditioning and produces inconsistent lighting under background conditioning. RGB2X records the worst angular error at 20.68.

• Human preference favors LumiCtrl: A two-alternative forced choice (2AFC) study with 15 participants screened with the Ishihara color vision test, covering 20 concepts and 4 standard illuminants (320 questions), analyzed with Thurstone's Case V Law of Comparative Judgment, shows LumiCtrl outperforming competing methods. The paper reports z-scores with a 95% confidence interval but does not list the individual numeric z-score values.

• Physical augmentations are limited by design: Flat Light Adaptation produces extreme orangish or bluish images as the illuminant is scaled toward warm and cool conditions, and cannot create illuminance-adaptive shadows — the concept's shadow remains identical across illuminants. LumiCtrl generates illuminance-adaptive shadows while Flat Light Adaptation does not.

Methodology in Plain English

The authors start from an existing personalization approach (Custom Diffusion) built on Stable Diffusion v1.5 and change what gets learned.

First, they create training variants of a single input image by applying physics-based illumination changes along the Planckian (black-body) locus, using seven illuminants: four camera presets — Tungsten (2850K), Fluorescent (3800K), Cloudy (6500K), Shade (7500K) — and three intermediates at 3300K, 4500K, and 7000K. This step, which they call flat light adaptation, is a simple uniform color transform used only to generate reference training data.

Second, they introduce new learnable tokens into the text vocabulary: one token for the concept itself and one for each illuminant, trained with prompt templates such as "a photo of [v] dog in [c*] illuminant." Only the key and value projection matrices in the diffusion model's cross-attention layers, plus the modifier tokens, are optimized; the rest of the model stays frozen.

Third, to stop the illuminant tokens from absorbing the structure of the training image, they add a frozen ControlNet conditioned on Canny edge maps during training. Because structure is already supplied by the edge map, the tokens are pushed to represent only color.

Fourth, they apply a Masked Reconstruction Loss that upweights the foreground region (via a user-provided mask) and downweights the background, with a trade-off hyperparameter balancing the two. This lets the foreground learn an accurate illuminant while the background shifts based on the model's own prior knowledge — the behavior they call Contextual Light Adaptation.

Training used Stable Diffusion v1.5 on an NVIDIA A40 GPU with the AdamW optimizer, batch size 2, learning rate 10^-5, and 3000 steps; the masked reconstruction loss was computed at 64×64 resolution, matching the image latent size. Inference used 200 DDPM steps with a classifier-free guidance scale of 6.0, and ControlNet was discarded. Evaluation compares against T2I personalization methods and T2I appearance editing methods under identical conditions, using estimated illuminants (via white-balancing and per-pixel ratio aggregation over a foreground mask) compared to Planckian ground truth.

Why This Matters

Impact on research. The paper reframes lighting control as a text-encoder grounding problem rather than a relighting problem. It introduces Prompt-Space Illumination Modeling as a distinct paradigm from Image-Space Illumination Control, and it provides a diagnostic methodology (embedding probing plus silhouette scores) that others can reuse to find similar semantic gaps for other physical attributes such as material, weather, or time of day.

Real-world applications (bullets):

• Content creation and digital art: Designers can generate a personalized subject under a specified illuminant instead of manually correcting lighting afterwards.
• Photography and post-production workflows: The paper explicitly frames its work relative to the five default illuminant options (daylight, tungsten, fluorescent, cloudy, shade) found in software like Photoshop — though the authors note generation runs in the reverse direction, introducing an illuminant rather than removing it.
• Product and advertising visualization: A learned subject can be rendered consistently across lighting conditions while keeping its identity, texture, and pose intact.
• Concept personalization pipelines: Any workflow already using Textual Inversion, DreamBooth, or Custom Diffusion could adopt illuminant tokens as an add-on to gain lighting control.

Industry relevance. The method trains on a single image, tunes a small number of parameters (cross-attention key/value projections plus tokens), and drops ControlNet at inference, which keeps generation cost comparable to standard personalization pipelines. That makes it a plausible feature for consumer creative tools rather than a research-only technique.

Future Directions

• Beyond seven illuminants: The authors note they focused on seven widely adopted illuminants for reproducibility, but content creators may need rarer illuminants that were not considered.
• Continuous rather than discrete illumination: Because illumination exists on a continuous spectrum, adding discrete tokens to the CLIP vocabulary is described as a baseline that can be further explored; a unified training framework capturing continuous variability is flagged as an interesting direction.
• Extending prompt-space modeling to other physical attributes: The semantic-gap analysis is general enough to motivate similar learned prompts for other photometric or physical properties that text encoders fail to ground.
• Background priors under extreme lighting: Since Contextual Light Adaptation relies on the pretrained model's prior for the background, understanding how far that prior can be pushed — and where it breaks down — remains an open question, particularly given the authors' observation that raising the masked-loss trade-off parameter produced unrealistic lighting that also affected the background.

Target Audience

Researchers and engineers working on diffusion-based image generation, model personalization, and controllable synthesis will benefit most, along with color science and computational photography researchers interested in how photometric concepts are (mis)represented in multimodal text encoders. Practitioners building creative or product-visualization tools that need reliable lighting control will find the practical setup — single input image, small number of trained parameters, ControlNet removed at inference — directly relevant.

Authors’ abstract

Text-to-image (T2I) models have demonstrated remarkable progress in creative image generation, yet they still lack precise control over scene illuminants which is a crucial factor for content designers to manipulate visual aesthetics of generated images. In this paper, we present an illuminant personalization method named LumiCtrl that learns illuminant prompt given single image of the object. LumiCtrl consists of three components: given an image of the object, our method apply (a) physics-based illuminant augmentation along with Planckian locus to create fine-tuning variants under standard illuminants; (b) Edge-Guided Prompt Disentanglement using frozen ControlNet to ensure prompts focus on illumination, not the structure; and (c) a Masked Reconstruction Loss that focuses learning on foreground object while allowing background to adapt contextually which enables what we call Contextual Light Adaptation. We qualitatively and quantitatively compare LumiCtrl against other T2I customization methods. The results show that LumiCtrl achieves significantly better illuminant fidelity, aesthetic quality, and scene coherence compared to existing baselines. A human preference study further confirms the strong user preference for LumiCtrl generations.