Image Generation, Editing, and Diffusion

A prompt such as “a red bicycle beside a canal” admits many valid scenes. Evaluation therefore has to separate prompt adherence, visual quality, diversity, consistency, and prohibited content. Those are not one axis under several names. Later in this lesson an automated image score and 1,633 blinded human comparisons rank the same two models in opposite directions. That is what it looks like when the axes are collapsed into one number.

Editing adds another constraint. Preserve specified parts of the source while changing others. That is a different contract from unconstrained generation. And in at least one published editor it is not one setting but two: a scale for how much of the input survives, and a scale for how hard the instruction is applied.

A simplified diffusion pipeline learns how to reverse controlled corruption. Draw a clean image from the training distribution. Corrupt it at a chosen step according to a known schedule. Train the network to predict the noise, or a related target, from the noisy image, the timestep and the conditioning. At inference there is no target image to start from, only noise, and a scheduler applies the conditional predictions repeatedly until an image emerges.

Latent diffusion moved that loop off the pixels. Rombach and four colleagues run the noising and the denoising “in the latent space of powerful pretrained autoencoders”. The point of doing so is cost. It is how you “enable DM training on limited computational resources while retaining their quality and flexibility”. Conditioning enters through attention. “by introducing cross-attention layers into the model architecture”, they write, the same models become “generators for general conditioning inputs such as text or bounding boxes”. That was 2022. Text-to-image as most people have met it starts there.

Guidance has authors too. Ho and Salimans jointly train “a conditional and an unconditional diffusion model” — “We use a single neural network to parameterize both models” — and then combine “the resulting conditional and unconditional score estimates to attain a trade-off between sample quality and diversity similar to that obtained using classifier guidance”. Trade-off is the authors' own word. Nobody described it as free.

Similar-looking outputs can make different claims about their relationship to source evidence, and the claim decides which tests are meaningful.

Unconditional or prompted generation creates a new sample from learned distributions. There is no single target image, so diversity and prompt fidelity are the things worth measuring. Concept art is the ordinary case.

Reconstruction or restoration estimates a latent source from a measurement. Fidelity to the evidence is the point, the forward model matters, and invented detail is a failure rather than a feature. Denoising is the ordinary case.

Editing modifies selected attributes while preserving protected content. Consistency with the source is the contract, masks or instructions define the scope of the change, and identity drift is the characteristic failure. A product mockup is the ordinary case.

The pixels do not tell you which of the three you are looking at. The same output can be excellent under one contract and a defect under another.

A trustworthy editor shows and tests the constraints on what must be preserved. Two published systems make those constraints concrete.

ControlNet adds spatial conditioning to a locked, pretrained Stable Diffusion backbone. The new path is wired in with zero-initialized convolution layers — 'zero convolutions' — so that no harmful noise reaches the finetuning. Edges, depth, segmentation and human pose are the conditions its authors test. On how much data this takes they are explicit: “We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets.”

Instruction editing needed a dataset that did not exist, so InstructPix2Pix built one. Its authors hand-wrote instructions for 700 input captions sampled from LAION-Aesthetics V2 6.5+, fine-tuned GPT-3 Davinci on them for one epoch, and generated a final corpus of 454,445 instruction-and-caption examples. At sampling time the model then exposes two classifier-free guidance scales instead of one: “Increasing sI results in edited images that more closely resemble the input image, and increasing sT results in more intense edits.” Preservation and change are literally two dials. An editing product is a decision about where they are set.

Offline image scores are one layer of evidence, and there is a published case of the score and the raters pointing in opposite directions. In 2024 Google Research put a full Muse model against a deliberately early-stopped one, over all 1,633 PartiPrompts, blinded, three independent raters per pair. The full model is Model-A. “We observed that Model-A was preferred in 92.5% of the comparisons, while Model-B was preferred only 6.9% of the time.” The raters were not hesitating: 0.6% were indifferent, and Cohen's kappa was 0.9937.

FID ranked the pair the other way round, 18.42 for the degraded model against 21.40 for the full one. So did FID-infinity, 17.19 against 20.16, and KID, 0.0080 against 0.0105. The authors' proposed CMMD agreed with the raters, 0.721 against 0.951, lower being better. Their own preprint states it plainly: “We empirically demonstrate that FID contradicts human raters”.

A memorization probe returns numbers too, and they are small without being zero. Somepalli and four colleagues searched 12 million LAION-Aesthetics v2 6+ images against Stable Diffusion outputs with an SSCD retrieval model. Generations exceeding 0.5 similarity to a training image came to roughly 170 images, about 1.88% of the random generations. The training images matched above that threshold had themselves been duplicated an average of 34.1 times. Then comes the caveat a product team has to carry: the searched set is less than 0.6% of the model's training data, so this is a lower bound obtained on a sliver. “While most of the generations from large-scale models do not contain copied content, a non-trivial amount of copying does occur”.

Around those two measurements sits the rest of the loop. Build prompt suites covering objects, attributes, counts, relations, rendered text, cultures and prohibited requests. Measure fidelity and diversity with automated scores and blinded human comparisons, knowing from the Muse comparison that the two can invert. For editing, quantify protected regions, identity, geometry and unintended change. Probe memorization with a retrieval model, and report what fraction of the corpus you actually searched. Then monitor the product itself: rejection, complaints, provenance, unsafe outputs and distribution shift.

A retailer wants to change backgrounds and lighting while preserving the exact product shape, label, and color. Every choice below is a place where one of the results above lands on the specification.