Generative Adversarial Networks and Learned Critics

A generator turns random variables, and optional conditions, into synthetic samples. A discriminator — a critic — learns to tell real observations apart from generated ones.

The generator never gets a target to copy. Its gradients arrive through the critic. And because both networks change during training, the objective landscape each player faces keeps moving. Nonstationary is the technical word for it.

The two networks improve against each other. Neither objective guarantees a stable equilibrium in practice — and that is a proved statement, not a complaint about tuning.

The strongest guarantee anyone has obtained is deliberately narrow. Give the discriminator and the generator separate learning rates — a two time-scale update rule — and something can be proved about where training ends up. Heusel and colleagues proved it in 2017. “Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium.” Read the qualifiers rather than the verb. The equilibrium is local, not global. It holds under mild assumptions. And it belongs to that one update rule, not to adversarial training in general.

The other boundary was drawn from the opposite side. In 2018 Mescheder and two colleagues built a prototypical counterexample. When the data and generator distributions are not absolutely continuous, unregularized GAN training is not always convergent. Wasserstein-GANs and WGAN-GP fail the same way when the discriminator gets a finite number of updates per generator update. That finite number is exactly the update ratio the fourth stage of this loop rebalances. It is why the loop has a rebalancing stage at all.

Changing the loss alters gradient geometry. It also alters how the critic should be read.

The first row is the original, and it names its own target. Goodfellow and seven colleagues published it in 2014. They train “a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G”. Then they characterise the equilibrium exactly. “This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere.” A critic at one half is the goal. A critic at one is a critic that has stopped teaching.

The rows are not equals under test, and the loss is not the only variable in the row. Google Brain ran a neutral large-scale study in 2019, comparing regularization and normalization across the non-saturating, Wasserstein and least-squares losses. The constraint placed on the discriminator travelled better than the choice of loss. Spectral normalization improved sample quality consistently across all three objectives. The gradient penalty showed no clear trend. Pick a row, then pick the constraint that goes with it. The figures are in the debugging section below.

A visually impressive page can coexist with serious distribution failures.

Mode collapse has a mechanism, and the people who proposed the standard countermeasure led with it. “One of the main failure modes for GAN is for the generator to collapse to a parameter setting where it always emits the same point.” That is Salimans and five colleagues, in 2016. The reason is structural rather than a matter of tuning. “Because the discriminator processes each example independently, there is no coordination between its gradients, and thus no mechanism to tell the outputs of the generator to become more dissimilar to each other.” So “all outputs race toward a single point that the discriminator currently believes is highly realistic”. Their minibatch discrimination lets the critic look at several samples at once. That is the smallest change that removes the independence assumption.

The failure also has a countable size. Stacked MNIST places three MNIST digits in the colour channels of one image, so the target distribution has exactly 1,000 modes. Coverage can be counted rather than eyeballed. Srivastava and colleagues counted it in 2017, in the paper that introduced VEEGAN. DCGAN captured 99 of the 1,000 modes, at a KL of 3.4. ALI captured 16 modes at KL 5.4, Unrolled GAN 48.7 modes at KL 4.32, and VEEGAN itself 150 modes at KL 2.95. A generator holding 99 modes out of 1,000 can still fill a sample grid that no reviewer would question.

A second laboratory ran the same benchmark and got a second number. The PacGAN authors re-ran it in 2018 with their own DCGAN implementation, over 10 trials. They measured 78.9 ± 6.46 modes, at KL 4.50 ± 0.127. Their own PacDCGAN2 reached 1000.0 ± 0.00 modes at KL 0.06 ± 0.003. The same architecture on the same benchmark lands at 99 modes in one lab and 78.9 ± 6.46 in another. That gap is the implementation variance to expect before you attribute a coverage gap to your idea. Lin and colleagues state the general problem in their abstract: “Despite remarkable advances in generating realistic images, a major shortcoming of GANs is the fact that they tend to produce samples with little diversity, even when trained on diverse datasets.”

The memorization bullet is not a worry either; it has a measured worst case. The first membership inference attacks on generative models were mounted by Hayes and colleagues in 2019, in a paper called LOGAN. With white-box access, the attack works completely. “We note that, for white-box attacks, the attacker successfully infers the training set with 100% accuracy on both the LFW and CIFAR-10 datasets, and 95% accuracy for DR dataset.” Access changes the number. With no auxiliary knowledge, black-box accuracy falls to 40% on LFW, 37% on CIFAR-10 and 22% on the diabetic-retinopathy set. The obvious remedies did not change it. The candidate defences they tested “are either ineffective or lead to significantly worse performances”. A later paper, GAN-Leaks, systematised the same risk across image, medical and location data in 2020.

Treat curves as clues rather than expecting one ideal loss trajectory.

Step four has a worked example, and a warning about stabilizers that look harmless. Instability in Wasserstein GANs was traced to one choice made in the code. “We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior.” That is Gulrajani and four colleagues, in 2017. They replace clipping with one term: “penalize the norm of gradient of the critic with respect to its input”. The payoff they report is exactly the property step four is testing for. The method “enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets”.

Step four also has a published ranking, which is the reason it says one at a time. Spectral normalization constrains the discriminator's weights, and Miyato and three colleagues proposed it in 2018. Google Brain then tested it against the gradient penalty in a neutral large-scale study in 2019. The finding is a figure caption: “We observe that spectral norm consistently improves the sample quality.” The gradient penalty did not earn the same sentence — “in some cases the gradient penalty can help, but there is no clear trend”.

The cost is the figure to budget against, because step four is expensive in exactly this currency. Reaching an FID below 100 with the non-saturating loss plus the gradient penalty required at least 6 hyperparameter settings. Spectral normalization reached a better, lower FID with 2. On a fixed compute budget, a stabilizer that needs six settings and one that needs two are not the same recommendation. Only a study that fixed the budget could tell them apart.

After training, a generator can often produce a sample in one forward pass. That claim now comes with a published latency. In 2023 Kang and colleagues scaled a GAN up for text-to-image synthesis and called it GigaGAN. Their abstract states the advantage as a measurement: “First, it is orders of magnitude faster at inference time, taking only 0.13 seconds to synthesize a 512px image.” The same system reports 3.66 seconds for a 16-megapixel image. That is what a one-pass generator buys, in seconds, at two resolutions.

The cost is on the same page rather than in a rival's paper. Scaling the architecture is where the difficulty moved. Naively increasing the capacity of the StyleGAN architecture, the authors report, “quickly becomes unstable”. So both halves of the trade-off are measured, one in seconds of inference and one in training difficulty. Neither is settled by looking at samples. Compare GANs with autoregressive, flow, VAE, and diffusion baselines under the actual latency and coverage requirements, and price the training game as part of the decision.