Inference-Time Scaling of Diffusion Models for Infrared Data Generation

AI summary

Overview

Research area: Generative computer vision — text-to-image diffusion models applied to the infrared (thermal) domain, with a focus on inference-time scaling rather than pretraining or training-time scaling.

Technical level: Intermediate. The paper assumes familiarity with diffusion sampling, CLIP-style contrastive encoders, LoRA finetuning, and FID, but the core idea — pick the best of several generated images using a trained scorer — is easy to follow.

One-sentence scope: The paper tests whether a domain-adapted CLIP verifier, used to search over noise latents at generation time, improves the quality of synthetic infrared images produced by a LoRA-finetuned FLUX.1-dev model trained on only 1,000 infrared image-caption pairs.

What This Paper Is About

Infrared imagery captures thermal information and is useful when RGB cameras struggle, but high-quality annotated infrared data is scarce, so training large generative models directly in the infrared domain is difficult. The authors instead take a pretrained RGB text-to-image model, lightly adapt it to infrared with a small sample of data, and then spend extra compute at inference time to select better generations. The goal is to raise the visual and semantic quality of synthetic infrared images without needing a large infrared training corpus.

Key Contributions

1. An application of inference-time scaling to infrared image generation — a direction the authors state has not previously been explored — built on the verifier-plus-search framework of Ma et al. [10].
2. A self-supervised, domain-adapted verifier: CLIP-B/32 is finetuned to distinguish true infrared images from grayscale counterparts, using paired infrared images and captions plus grayscale augmentations of RGB images in each mini-batch, with captions formatted as "An INFRARED photo of {caption}." and "A GRAYSCALE photo of {caption}."
3. A scoring function, IRScore, defined as (1 − α)·cos(Φ_T(c_IR), Φ_I(x)) − α·cos(Φ_T(c_gray), Φ_I(x)), where α ∈ [0, 1] balances alignment with infrared semantics against similarity to grayscale.
4. An empirical comparison of two noise-search strategies (random search and zero-order search) under a shared 336-NFE budget on the KAIST Multispectral Pedestrian Detection Benchmark, showing FID improvements over naive sampling.

Main Findings

• Random search gives the largest measured gain: FID drops from 74.58 with naive sampling to 66.74 with random search, a reduction of over 10% relative to the baseline, at 336 NFEs versus 28 NFEs for naive sampling. Both configurations used α = 0.5 on the test split.
• Zero-order search improves less at the same budget: FID of 69.15, described as a roughly 7% gain under the same NFE budget. The authors attribute this to the incremental update procedure converging more slowly than the broader noise-space exploration of random search, consistent with Ma et al. [10].
• Search raises verifier scores consistently: Mean IRScore across generated samples — 0.1187 for naive sampling, 0.6010 for random search, and 0.4214 for zero-order search (all multiplied by 10 in Table 1). Both search methods outperform naive sampling on this metric.
• The finetuned verifier separates quality levels where pretrained CLIP does not: In Figure 1, for a ground-truth infrared image the authors report IRScore (ours) 0.5062, IR Similarity 0.4879, Grayscale Similarity 0.3866, and IRScore (pretrained) 0.1016. A high-scoring synthetic image yields IRScore (ours) 0.5135, IR Similarity 0.4213, Grayscale Similarity 0.3186, IRScore (pretrained) −0.0414; a low-scoring synthetic image yields IRScore (ours) −0.3212, IR Similarity 0.3189, Grayscale Similarity 0.3831, IRScore (pretrained) −0.0125. The pretrained CLIP model assigns a higher score to the lower-quality image than to the higher-quality one, while the finetuned model does not.
• Unguided generations are inconsistent: The authors observe high variability in quality from the LoRA-finetuned model, with many samples physically implausible or resembling grayscale versions of RGB images, which they attribute to inductive biases inherited from large-scale RGB pretraining.
• The NFE budget is modest: The authors note that 336 NFEs is relatively limited compared with other inference-time scaling work [10, 11] and expect further gains from a larger compute budget.

Methodology in Plain English

The team started from FLUX.1-dev, a strong pretrained text-to-image model, and taught it about infrared by finetuning with LoRA at rank r = 16 on just 1,000 infrared image-caption pairs, rather than training a diffusion model from scratch on scarce infrared data.

Because that lightweight adaptation produced images of uneven quality, they added a scoring step. They finetuned CLIP-B/32 on the same KAIST training data, deliberately mixing in grayscale versions of RGB images so the model would learn that a true infrared image is not the same thing as a black-and-white photo. At generation time, this finetuned CLIP scores a candidate image by how well it matches an infrared-flavored prompt and how poorly it matches a grayscale-flavored version of the same prompt, combined by the α weighting in IRScore.

They then tried two ways of searching noise space using that score. Random search draws N independent noise latents from a Gaussian, denoises each, and keeps the highest-scoring image, so NFEs = N × (number of denoising steps). Zero-order search instead takes an initial latent, samples N − 1 neighbors around it, denoises all N, moves the "pivot" to the best one, and repeats for k iterations, so NFEs = kN × (number of denoising steps). Zero-order search is intended to reduce verifier hacking, where search overfits to biases of the verifier instead of genuinely improving quality. Evaluation used FID against real images, plus the mean IRScore, on the KAIST test split.

Why This Matters

The paper reframes a data bottleneck as a compute problem: instead of needing a large infrared corpus to train a generative model, it spends compute at sampling time to pick better outputs from a model adapted on very little data. If this holds up, synthetic infrared data could be generated more cheaply, which matters for any vision system that must work in darkness, glare, fog, or other low-visibility conditions.

• Autonomous driving and driver assistance: The KAIST benchmark used here is vehicle-mounted long-wave infrared and RGB pedestrian imagery, a direct match for night-time pedestrian detection.
• Surveillance and security: Passive thermal sensing works without illumination, so synthetic infrared data could supplement scarce annotated footage.
• Medical imaging: The introduction lists medical imaging among domains where thermal information is valuable and where data collection is specialized.
• Data augmentation for detectors and scene classifiers: The stated motivation is providing large-scale, diverse training data to train downstream object detectors and scene classifiers.

Industry relevance: The work sits at the intersection of generative model deployment and sensor-specific data pipelines. The practical takeaway for industry is that a small finetuning set plus extra inference compute can substitute, to some degree, for an expensive proprietary infrared dataset — at the cost of higher generation-time latency, which is quantified here as 336 NFEs versus 28.

Future Directions

• Bigger inference budgets: The authors explicitly expect performance to improve substantially by scaling beyond the 336-NFE budget, which they describe as limited relative to prior inference-time scaling work [10, 11].
• Alternative verifiers: Training physics-based or domain-specific verifiers instead of, or alongside, the adapted CLIP model.
• Generalization beyond one dataset and modality: Extending evaluation to other datasets and other sensing modalities to test whether the approach generalizes.
• Search-strategy trade-offs: Understanding why random search outperformed zero-order search here in terms of convergence, and whether verifier hacking remains a risk given the small-scale verifier training data.

Target Audience

Researchers and practitioners working on generative models for non-RGB sensing, synthetic data generation for data-scarce domains, and inference-time scaling of diffusion models. It is also relevant to engineers building perception systems for low-light or low-visibility conditions who want to know whether a small domain dataset plus extra sampling compute is a viable alternative to collecting a large infrared corpus. Readers looking for downstream detector accuracy numbers, comparisons against other infrared generation methods, or a full ablation over α and dataset size will not find them here: the paper reports FID and IRScore on KAIST, plus the Figure 1 verifier-score examples, and describes much of the rest as preliminary.

Authors’ abstract

Infrared imagery enables temperature-based scene understanding using passive sensors, particularly under conditions of low visibility where traditional RGB imaging fails. Yet, developing downstream vision models for infrared applications is hindered by the scarcity of high-quality annotated data, due to the specialized expertise required for infrared annotation. While synthetic infrared image generation has the potential to accelerate model development by providing large-scale, diverse training data, training foundation-level generative diffusion models in the infrared domain has remained elusive due to limited datasets. In light of such data constraints, we explore an inference-time scaling approach using a domain-adapted CLIP-based verifier for enhanced infrared image generation quality. We adapt FLUX.1-dev, a state-of-the-art text-to-image diffusion model, to the infrared domain by finetuning it on a small sample of infrared images using parameter-efficient techniques. The trained verifier is then employed during inference to guide the diffusion sampling process toward higher quality infrared generations that better align with input text prompts. Empirically, we find that our approach leads to consistent improvements in generation quality, reducing FID scores on the KAIST Multispectral Pedestrian Detection Benchmark dataset by 10% compared to unguided baseline samples. Our results suggest that inference-time guidance offers a promising direction for bridging the domain gap in low-data infrared settings.