Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning

AI summary

Overview

Research area: Computer vision — pixel-space (VAE-free) diffusion models for text-to-image generation, and generative priors for dense prediction tasks (monocular depth estimation, image restoration/super-resolution).

Technical level: Advanced. The paper assumes familiarity with latent diffusion, flow matching, transformer architectures, representation alignment objectives (REPA/iREPA/Self-Flow), and zero-shot depth evaluation protocols.

Scope: The paper builds pixel-space text-to-image backbones along both available routes — pretraining from scratch and converting a pretrained latent model — then tests whether a pixel-space generative prior transfers better than a latent one to detail-critical downstream tasks. arXiv:2610.09450v1 [cs.CV], published 2026-10-07, by Hanqiu Li Cai (project lead) and Chema Garabito, SperidLabs. Code and models at github.com/speridlabs/iris-3b. License: CC Zero.

What This Paper Is About

Latent diffusion models run generation inside the compressed space of a pretrained VAE, which is efficient but lossy: any detail the codec discards is unreachable by the generator. Since dense prediction tasks such as depth estimation and restoration require pixel-accurate fine detail, the natural expectation is that a pixel-space generator should transfer better to them than a latent one. The authors build pixel-space backbones both ways — pretraining a 3B-parameter model from scratch and converting a pretrained latent model — and test that expectation directly.

Key Contributions

1. A test of pixel-space priors for dense prediction. Iris-3B (pretrained from scratch) and a converted FLUX.2 Klein base 4B are fine-tuned for monocular depth estimation and image restoration/super-resolution and compared against latent counterparts. The paper reports no significant improvement from pixel space and documents the confounds that remain.
2. A controlled 256px pretraining ablation campaign. Prediction targets (x vs. v) and representation-alignment strategies (REPA, iREPA, DINOv3 targets and Self-Flow) are each compared with their own control at matched steps on a 1.3B dual-stream PixelDiT reproduction.
3. A latent-to-pixel conversion recipe. FLUX.2 Klein base 4B is converted to pixel space; the paper identifies the input interface as the obstacle and removes it with a gain-matched ridge initialization plus a scaled trunk learning rate.
4. A released model. Iris-3B, an open 3B-parameter pixel-space text-to-image model trained from scratch to 1024², is released with weights and training code.

Main Findings

• Pixel-space pretraining scales to 3B parameters. Under the official evaluators at 1024², Iris-3B reaches GenEval 0.798, DPG 86.52, LongText 0.857 and OneIG (English overall) 0.540. It surpasses FLUX.1-dev 12B (0.66 / 83.8 / 0.607 / 0.434), is level with i1 3B on DPG (86.7), and on OneIG-EN matches Qwen-Image 20B (0.539). It trails i1 3B and Z-Image 6B on GenEval (both 0.84), though those two use prompt rewriting. It is strongest on single objects, two objects and colors and weakest on position and counting; English long-text rendering is its clearest gap.
• The scaled recipe. Ablations kept v-prediction over x-prediction and REPA with a DINOv2 teacher over the alternatives. x-prediction showed no benefit: the two targets were tied on GenEval at every milestone and x-prediction was worse on DPG and FID — though the x arm also used JiT's logit-normal (0.8, 0.8) timestep distribution instead of the default (0.0, 1.0), so the contrast does not isolate the prediction target. Switching the REPA teacher to DINOv3 lowered GenEval at every milestone and worsened FID throughout. iREPA had better FID than the control at 150K and 200K but trailed on GenEval and CLIP at nearly every milestone, and its FID advantage had almost vanished by 250K. Self-Flow was far behind on all four metrics.
• No significant gain from a pixel prior on depth. With one matched direct-regression recipe at 10K steps, Iris-3B is level with the latent FLUX.2 Klein on the five-benchmark mean (AbsRel .071 vs. .072; δ1 .946 vs. .947) — better on KITTI and DIODE, worse on NYUv2, ETH3D and ScanNet. The converted pixel model trails the latent one on every benchmark except KITTI and on the mean (.081 / .934). Iris-3B has the lowest soft edge error on Hypersim (.496 vs. .528 latent and .526 converted pixel).
• No advantage on restoration either. On DIV2K with native 4× Real-ESRGAN degradations, the converted pixel model is worse than its latent twin on most metrics — most clearly SSIM (0.504 vs. 0.557) and NIQE (4.11 vs. 3.26) — and level on LPIPS (0.274 vs. 0.275).
• Iris-3B restoration is competitive but unmatched. Its scores (PSNR 20.68, SSIM 0.481, LPIPS 0.292, NIQE 3.94, MUSIQ 68.01, DeQA 3.87) put it best of all methods in the comparison table on MUSIQ, slightly below both FLUX.2 Klein arms on fidelity and DeQA, and between them on NIQE. Its parent checkpoint, batch, schedule and budget differ from the FLUX.2 Klein arms, and inference adds wavelet colour correction and tiling, so this comparison is not matched.
• A persistent patch grid in pixel-space outputs. Measuring the mean absolute second difference of predictions next to 16×16 patch borders against its mean elsewhere, the latent model scores near one (0.97–0.99), the converted pixel model 1.31–1.44, and Iris-3B 1.61–2.25. NYUv2 and ScanNet are omitted because the latent model already scores well below one there. PixelUMM independently reports the same grid.
• The conversion bottleneck is the input interface. Under the unmodified conversion recipe, a 1K-step probe still generated prompt-independent patch-grid noise, and the unmodified recipe needed more than ten times as many steps to approach the loss the modified recipe reached in 1K. For FLUX, even the best per-patch linear map from pixels explains only 14% of the variance of the trunk's latent embeddings. A gain-matched ridge initialization and a trunk LR scaled to 0.12× the head's rate each more than halve the 1K-step loss on their own, but neither alone yields coherent images; together they produce photorealistic, prompt-aligned images within 1K steps at global batch 8.
• The converted model's deficit is the one consistent gap. Its parent was adapted with a short conversion while the latent arm starts from the fully pretrained model, so its prior is likely weaker — pointing to the conversion rather than to pixel space. The paper has not measured the converted models' GenEval, DPG or FID, so this remains a hypothesis.

Methodology in Plain English

The work proceeds in three parts.

Choosing a recipe. Before scaling, the authors reproduce PixelDiT at 1.3B parameters and run ablations at 256² on a fixed subset of their data at global batch 256. Every arm is scored with one protocol (EMA weights, 25-step FlowDPM++ sampling, CFG 4.5, GenEval, DPG-Bench, FID-10K against a held-out split, and CLIP score) and compared with its own control at matched steps. They sweep the prediction target and four alignment objectives.

Scaling to Iris-3B. The selected recipe (v-prediction with DINOv2 REPA) is scaled to 3B parameters with architecture changes made for efficiency at scale. The image is split into 16×16 patches, so 1024² becomes a 64×64 token grid. The trunk follows FLUX.1's hybrid layout: 8 dual-stream MMDiT blocks with separate text and image weights, then 16 single-stream blocks, at width 2560, using GQA with 20 Q and 5 KV heads, a sigmoid output gate, and sandwich normalization with RMSNorm. A shared-bias adaLN modulation removes 28% of the parameters of the same model with per-block adaLN at equal FLOPs. A four-block pixel transformer (PiT) head from PixelDiT decodes to pixels. Text comes from a frozen Qwen3-VL-4B-Instruct encoder; 12 of its layers are read and fused with layerwise attention pooling plus a two-block refiner. Training uses hybrid Muon (momentum 0.95, Nesterov, RMS-matched LR) for hidden weights and AdamW (β1 0.9, β2 0.95) for embeddings, norms and biases, with a constant 1e-4 LR after a 2K-step warmup, EMA decay 0.9999 and 10% caption dropout. The curriculum runs 256 → 512 → 1024 across four stages (Tab. 1: 0–367K steps at batch 1024 for 375.8M samples; 367K–530K

Authors’ abstract

Pixel-space diffusion models avoid the lossy VAE of latent models, which suggests an advantage on downstream tasks where fine-grained detail matters. We test this claim along both routes to a pixel-space backbone. We pretrain Iris-3B, a 3B-parameter pixel-space text-to-image transformer, from scratch through a $256\to512\to1024$ curriculum, after first ablating the prediction target and representation alignment at $256^2$ to decide what to scale. We also convert a pretrained latent model, FLUX.2 Klein base 4B, to pixel space. We fine-tune both families for monocular depth estimation and for image restoration/super-resolution. We find no significant improvement from using a pixel-space generative prior. Fine-tuned for depth with one matched direct-regression recipe, Iris-3B is level with the latent FLUX.2 Klein and the converted pixel FLUX.2 Klein falls behind it, and on $4\times$ DIV2K restoration neither pixel model beats a latent FLUX.2 Klein fine-tune, the converted one trailing it slightly. We document the recipes, the failure modes and the remaining confounds behind this negative result. Nevertheless, Iris-3B shows that pixel-space pretraining with the pixel-transformer (PiT) head of PixelDiT scales to 3B parameters and to text-to-image quality competitive with latent models, matching Qwen-Image on OneIG under the official evaluators at $1024^2$. We release its weights and training code in the hope that they help pave the way for further work on pixel-space generation.