LumiX: Structured and Coherent Text-to-Intrinsic Generation

AI summary

Overview

• Research area: Computer vision and graphics, specifically text-conditioned diffusion models for generating intrinsic scene representations (albedo, irradiance, normal, depth, and final color).
• Technical level: Advanced. The paper assumes familiarity with diffusion models, flow matching, self-attention mechanics, LoRA, and tensor decompositions.
• Scope: LumiX is a structured diffusion framework that jointly generates a coherent set of intrinsic maps from text, and can also perform image-conditioned intrinsic decomposition within the same model.

What This Paper Is About

Existing text-to-image diffusion models output a single RGB image, which hides the underlying geometry, lighting, and material properties of a scene. Standard intrinsic image decomposition can recover those properties, but only from an already-given image, so it cannot create new scenes from language. LumiX targets "text-to-intrinsic generation": starting from a text prompt and jointly producing five pixel-aligned maps (color, albedo, irradiance, depth, and normal) that describe one physically consistent scene.

Key Contributions

1. LumiX framework. A structured diffusion framework for coherent text-to-intrinsic generation that jointly produces multiple physically consistent intrinsic maps in a single forward pass.
2. Query-Broadcast Attention. A forward-pass mechanism that shares a single query matrix (taken from the color model) across all intrinsic maps while keeping attribute-specific key and value projections independent, aligning scene content across maps.
3. Tensor LoRA. A tensor-based low-rank adaptation for finetuning that models cross-map relations in a parameter-efficient, scalable way, extending conventional matrix-based LoRA to a tensor formulation.
4. Unified generation and decomposition. The same model that is trained with text-only conditioning also performs image-conditioned intrinsic decomposition at inference by keeping one property clean while denoising the others.

Main Findings

• Coherence and preference gains. LumiX achieves 23% higher alignment and a better preference score (0.19 vs. -0.41) compared to the state of the art.
• Best configuration in the attention/LoRA study. The tensor-based row of Table 1 shows LumiX (Tensor LoRA) with 2.34M trainable parameters per attention block, 12.1 G FLOPs for LoRA, an alignment score of 8.30, and an average preference of 0.19 ImageReward / 20.52 PickScore.
• Baseline comparison. The official IntrinsiX repository row reports alignment 6.73 and average preference of -0.41 / 19.78. IntrinsiX without its first training stage collapses to alignment 3.17 and -1.22 / 18.54.
• Vanilla attention is weakest. Vanilla FLUX attention with Separate LoRA produces the lowest alignment score reported in Table 1 (2.40).
• Zero-shot albedo on ARAP. LumiX reports RMSE 0.165 and SSIM 0.753. Stronger RMSE comes from NIID-Net (0.129), Colorful Shading (0.149), Ordinal Shading (0.162), and IID (0.160); Colorful Shading also reports higher SSIM (0.796).
• In-the-wild decomposition. On 50 in-the-wild photographs, LumiX achieves ImageReward 0.14 and PickScore 20.16, compared to RGB↔X (-0.20 / 20.01) and Colorful Shading (0.06 / 20.03), despite training on 3K images versus 900K.
• Rank ablation. Rank R=8 gives the best balance (alignment 8.30, preference 0.19 / 20.52); R=4 remains competitive with fewer parameters (0.68M, alignment 7.86, -0.18 / 19.79); R=12 increases cost (4.98M, 22.5 G FLOPs) and drops to alignment 8.10 (0.14 / 20.29).
• Tuning the query projection hurts. Adding W_Q finetuning increases parameters (2.46M) and FLOPs (14.1 G) but degrades quality (alignment 7.14, -0.09 / 20.04).

Methodology in Plain English

The authors start from FLUX.1-dev, a pretrained text-to-image diffusion model, and finetune it on a subset of the Hypersim dataset containing about 3K images. Captions are generated from color images with BLIP-2 and used as text inputs.

Each intrinsic property (color, albedo, irradiance, depth, normal) is encoded into the latent space and the latents are concatenated along the batch dimension, so the base model sees five "images" at once. When attention is computed, instead of letting each property run its own self-attention independently, the query matrix from the color map is broadcast to every other map. The reasoning, drawn from prior work on content and style separation, is that queries carry the scene's content (so sharing them aligns structure) while keys and values carry property-specific appearance (so keeping them separate preserves each map's character). This is compared against the alternative in IntrinsiX, which concatenates keys and values across all maps and is M times more expensive.

For finetuning, the frozen FLUX weights stay fixed and only low-rank updates are inserted into the key and value projections (not the query projection). Rather than separate LoRAs per map, a fused one, or a hybrid version, the authors reshape the update matrix into a 4th-order tensor and factorize it into three components (A, B, C) with tensor contractions, so cross-map interactions are captured while parameter cost stays close to linear.

Properties also receive independent diffusion timesteps, which acts as a soft mask and lets the model accept a clean conditioning image at inference for decomposition tasks. Training used the Prodigy optimizer at learning rate 1.0, batch size 16, 10K steps on four NVIDIA A100 (80GB) GPUs, taking about 40 hours, with images resized to 512×512. Evaluation uses ImageReward and PickScore for perceptual quality, plus Qwen3-VL for cross-map alignment scored on a [0,10] scale over 200 sampled Hypersim test images.

Why This Matters

• Research impact: The paper argues that stable multi-map generation comes not only from scaling supervision but from structured parameter sharing. It offers a middle ground between fully independent per-map models (good quality, poor consistency) and fully fused attention (better consistency, unstable training and quadratic cost).
• Unified understanding and generation: One backbone handles both text-to-intrinsic synthesis and image-conditioned decomposition, avoiding separate pipelines for generation and analysis.

Real-world applications:

• 3D content and asset creation. Producing albedo, normal, and depth maps from a text prompt feeds directly into relighting, material editing, and asset pipelines for games and film.
• Augmented and virtual reality. Physically decomposed scene maps let virtual objects receive correct lighting and geometry cues from real environments.
• Robotics and scene understanding. Normal, depth, and irradiance maps provide structure that a single RGB image does not expose.
• Image editing and relighting. Separating albedo from illumination allows changing lighting without changing surface appearance, and vice versa.

Industry relevance: the method finetunes a public base model (FLUX.1-dev) with only about 133.1M trainable parameters and a 3K-image dataset, and it is trained on four A100 (80GB) GPUs in about 40 hours, which is a modest compute footprint relative to large-scale alternatives. It avoids ControlNet-style conditioning designs while still supporting image-conditioned decomposition.

Future Directions

1. Scaling to more intrinsic properties. The authors plan to extend LumiX beyond the five properties studied (color, albedo, irradiance, depth, normal) toward a broader set of intrinsic factors.
2. Scaling the data. The current model trains on about 3K Hypersim images; the authors intend to move to larger datasets.
3. Further improving albedo accuracy. On ARAP, several baselines still report lower RMSE than LumiX (0.165), leaving room to improve decomposition fidelity.
4. Toward unified models of geometry, materials, and illumination. The stated long-term goal is a single model achieving unified understanding and generation of scene geometry, materials, and illumination.

Target Audience

Researchers and practitioners in diffusion-based generative modeling, inverse rendering, and intrinsic image decomposition; graphics engineers who need material and illumination maps for relighting or asset creation; and readers interested in parameter-efficient finetuning methods (LoRA variants and tensor decompositions) for multi-task or multi-modal generation. The paper is best suited to readers already comfortable with diffusion model internals and attention mechanics.

Authors’ abstract

We present LumiX, a structured diffusion framework for coherent text-to-intrinsic generation. Conditioned on text prompts, LumiX jointly generates a comprehensive set of intrinsic maps (e.g., albedo, irradiance, normal, depth, and final color), providing a structured and physically consistent description of an underlying scene. This is enabled by two key contributions: 1) Query-Broadcast Attention, a mechanism that ensures structural consistency by sharing queries across all maps in each self-attention block. 2) Tensor LoRA, a tensor-based adaptation that parameter-efficiently models cross-map relations for efficient joint training. Together, these designs enable stable joint diffusion training and unified generation of multiple intrinsic properties. Experiments show that LumiX produces coherent and physically meaningful results, achieving 23% higher alignment and a better preference score (0.19 vs. -0.41) compared to the state of the art, and it can also perform image-conditioned intrinsic decomposition within the same framework.