Skip to content
AI.info

Kinds of learning

Generative and Discriminative Modeling

Compare discriminative prediction with generative modeling, clarify conditional and joint objectives, and examine when generation supports simulation or inference.

By the end you can

Comparison

Two different questions about the same observations

Models can focus on predicting a target from inputs, or on representing how observations and targets are distributed. The same dataset supports both questions, and the answers are not interchangeable: one produces a decision, the other produces a distribution you can sample from, score against, or complete.

FigureComparison · 2 columns

Discriminative question

Given this input, which output should be predicted?

  • Models a decision boundary or conditional mapping
  • Optimized for target prediction
  • Can ignore irrelevant input variation
  • Example: classify an image species

Generative question

What observations are plausible, possibly under a condition?

  • Models aspects of a data distribution
  • Can sample, reconstruct, or score observations
  • Must represent more variation
  • Example: generate images for a species

“Generative” describes an objective, not one architecture

A transformer can be trained generatively to predict the next token, while another transformer can classify a document. A probabilistic model can generate tabular samples, images, audio, sequences, or latent representations.

The clearest demonstration of the split is dated. An encoder was pretrained on unlabelled text with a fill-in-the-blank masked-language-model objective — a generative-style task, predicting what had been hidden. Then the same weights were fine-tuned discriminatively, by adding one extra output layer. That was BERT, in 2019. Its abstract records what the move was worth: “It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).” Nothing about the network changed between the generative phase and the discriminative one. The objective changed.

The benchmark authorities registered the consequence within months. The GLUE maintainers at NYU introduced SuperGLUE later that year, precisely because the headroom was gone. The state of the art of early July 2019 stood at 88.4. A non-expert human estimate stood at 87.1.

Architecture alone decides nothing. Objectives, data, conditioning, and use are what determine the learning approach and define what the model learns.

Example

What generative models can support

Generation is broader than creating pictures or prose. Two of the entries below are no longer promissory: one has a registered trial and a lung-function endpoint, the other has a measured failure mode with a name.

  • Conditional synthesis: produce candidate molecular structures under specified properties. Rentosertib is one of those structures. It is a TNIK inhibitor from Insilico Medicine, formerly ISM001-055, and both its target and its molecule came from generative AI. It was tested in a phase 2a randomized, double-blind, placebo-controlled trial: 71 patients with idiopathic pulmonary fibrosis, across 22 sites in China. After 12 weeks the 60 mg once-daily arm showed a mean forced vital capacity change of +98.4 ml (95% CI 10.9 to 185.9), against -20.3 ml (95% CI -116.1 to 75.6) for placebo. Nature Medicine published the result on 3 June 2025, and the abstract opens “Despite substantial progress in artificial intelligence (AI) for generative chemistry, few novel AI-discovered or AI-designed drugs have reached human clinical trials.” The evidence here is a placebo-controlled endpoint, not a plausible-looking structure.
  • Simulation: create plausible scenarios for planning, stress testing, or scientific models — with the caveat, measured later in this lesson, that plausibility and mechanism come apart.
  • Missing-data modeling: infer distributions over unobserved values under assumptions that have to be stated and defended, not inherited from the sampler.
  • Representation learning: learn latent variables that summarize recurring structure, then hand those representations to a discriminative stage.
  • Density or likelihood work: compare how well a model accounts for observations, including observations no one thought to look at.
  • Data augmentation: create additional examples, while checking fidelity and coverage. What happens when nobody checks has been measured twice. Shumailov and colleagues fine-tuned OPT-125m on wikitext2, reaching 34 mean perplexity from a 115 zero-shot baseline. Then they retrained each generation on the previous generation's output. By generation 9 a prompt about English church architecture came back as a list of coloured jackrabbits. Their abstract, in Nature in 2024: “We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear.” The same thing happens in images. Alemohammad and colleagues at Rice University watched StyleGAN2 artifacts amplify visibly within nine generations, named it Model Autophagy Disorder, and stated it as a law: “without enough fresh real data in each generation of an autophagous loop, future generative models are doomed to have their quality (precision) or diversity (recall) progressively decrease”.

Comparison

Prediction focus versus distribution focus

Neither family is universally superior; the choice follows the task and evidence.

Where the crossover falls has been measured. Take one parametric family and fit it two ways. Naive Bayes is fitted on the joint distribution, logistic regression on the conditional. The objective is then the only thing that varies. Ng and Jordan ran that comparison in 2001, and the result has two halves. People usually quote one. The discriminative fit has the lower error once data is plentiful. The generative fit “may also approach its asymptotic error much faster than the discriminative model — possibly with a number of training examples that is only logarithmic, rather than linear, in the number of parameters”. Across fifteen datasets from the UCI repository they found both regimes. The generative model was ahead while labelled data was scarce. The discriminative model was ahead once it was not. Which family is better is a question about your sample size.

FigureComparison · 3 columns

Discriminative model

Concentrates capacity on predicting the target.

  • Often strong with sufficient labeled data
  • Evaluation aligns directly with task metrics
  • May need fewer distributional assumptions
  • Cannot automatically simulate full observations

Generative model

Represents broader structure in the observations.

  • Supports sampling and latent modeling
  • Can exploit unlabeled structure
  • May require greater capacity and assumptions
  • Realism does not guarantee coverage or truth

Combined system

Uses generation or representation to support prediction.

  • Pretrain then fine-tune
  • Generate candidates then rank
  • Model uncertainty before acting
  • Needs separate evaluation at each stage

Key idea

A convincing sample can hide missing modes

A generator may produce sharp, plausible examples while omitting rare populations, unusual conditions, or inconvenient outcomes. The usual reassurance is that a reviewer will notice. Measure the reviewer and the reassurance collapses.

Put 315 people in front of real photographs and StyleGAN2 faces and they classify them at 48.2% accuracy, 95% CI [47.1%, 49.2%], against a 50% chance baseline. That was Nightingale and Farid, in 2022. A further 219 participants, trained with trial-by-trial feedback, reached only 59.0%. And 223 participants rated the synthetic faces 4.82 for trustworthiness, against 4.48 for the real ones. Their conclusion, in PNAS: “Our evaluation of the photorealism of AI-synthesized faces indicates that synthesis engines have passed through the uncanny valley and are capable of creating faces that are indistinguishable—and more trustworthy—than real faces.” Miller and colleagues replicated it the following year in Psychological Science, and extended it: 124 participants judged White AI faces to be human 65.9% of the time, against 51.1% for actual human faces. The eye is not merely at chance. It can be pulled the wrong way.

So the reviewer cannot even do the easy job — realism, the one property generators are optimised for — let alone the hard one. Evaluate diversity, conditional fidelity, memorization, subgroup behavior, and downstream consequences. A gallery of attractive samples is not a distributional audit.

At 48.2% against a 50% baseline, the reviewer is at chance on realism, and realism was never the question: plausibility of individual samples and adequacy of the modeled distribution are different claims.

Case

Classifiers trained on GAN samples lost 27.9% top-1 accuracy

The audit can be turned into a downstream task. Train an ImageNet classifier on nothing but samples from a conditional generator, then test that classifier on real images. Ravuri and Vinyals did exactly that in 2019. Modern generators, they note, “produce nearly photorealistic samples and obtain scores similar to the data distribution on heuristics such as Frechet Inception Distance (FID)”. And yet classifiers trained on BigGAN-deep samples lost 27.9% top-1 and 41.6% top-5 accuracy against classifiers trained on the real data. Likelihood-based models that score worse on those heuristics, VQ-VAE-2 and hierarchical autoregressive models among them, “substantially outperform GANs on this benchmark”. Whatever the GAN's samples were missing was not visible in the samples.

Steps

Choose a generative objective for a concrete reason

Generation introduces additional evaluation burdens, so it should answer a specific need.

Step four is the one people treat as advice, and it is the one that has been run at industrial scale. Carlini and colleagues took the most duplicated captions out of Stable Diffusion's training set and made the model draw them back: “To evaluate the effectiveness of our extraction methodology, we select the 350,000 most-duplicated examples from the training dataset and generate 500 candidate images for each of these prompts (totaling 175 million generated images).” Out of those 175 million images, 94 were near-copies of training images under an l2 threshold of 0.15 — 107 counting manual inspection of the top-1000. On CIFAR-10 the best DDPM memorized about twice as many training images as a StyleGAN-ADA of similar FID. The better-scoring family was the leakier one.

The consequence is not only technical. The U.S. Copyright Office cites this work in its May 2025 report on generative AI training, when it reasons: “When, however, a specific model can generate verbatim or substantially similar copies of a training example, without that expression being provided externally in the form of a prompt or other input, it must exist in some form in the model's weights.” A memorization test is a search for something a government body has already said the weights are capable of holding.

FigureProcess · 5 steps

  1. 1. State the artifact

    Name whether the system must sample, reconstruct, complete, simulate, or represent.

  2. 2. Define conditioning

    Specify which inputs or controls should constrain the output.

  3. 3. Protect coverage

    List rare cases, groups, and modes that must not disappear.

  4. 4. Test memorization

    Check whether outputs reproduce sensitive or copyrighted training examples.

  5. 5. Evaluate the use

    Measure whether generated artifacts improve the intended downstream decision.

Case

9,000 ordinary captions, and about 170 near-copies of the training set

Step four has also been run without any adversarial intent. Nine thousand captions were sampled from the LAION Aesthetics dataset, fed one at a time to Stable Diffusion v1.4, and every generation was checked against the training images for its closest match. Somepalli and colleagues published that experiment in 2023. About 170 of the 9,000 — “(top 1.88 percentile)” by similarity — came back above their copying threshold, and “above this 0.5 threshold, we observe a significant amount of copying”: whole objects and backgrounds carried over from a training image, in one row with only the painting on the wall changed. None of those 9,000 prompts was written to extract anything. They were the captions of ordinary training images.

Position

Synthetic data is a testable claim, and the gallery is not the test

Generated data is sold on how it looks, which is the one property that cannot settle the question — and, on the evidence of the sample-quality section, the one property on which a human reviewer scores 48.2%. The test that can settle it is cheap to describe: train the downstream model on nothing but samples, then evaluate it on real data. Ravuri and Vinyals ran exactly that on ImageNet. Classifiers trained on BigGAN-deep samples lost 27.9% top-1 and 41.6% top-5 against classifiers trained on the real thing, and the generators supplying those samples were scoring like the data distribution on FID. The ordering even reverses: likelihood-based models that look worse by those heuristics substantially outperform GANs on this benchmark.

The second failure runs the opposite way and is equally invisible in a picture. Somepalli and colleagues put 9,000 ordinary LAION captions through Stable Diffusion v1.4, and about 170 generations came back above their copying threshold, whole objects and backgrounds carried across from a training image — out of prompts that were nobody's attempt to extract anything. Carlini and colleagues then showed the same property under pressure: 94 near-copies out of 175 million generations. One failure is what the samples leave out, the other is what they repeat, and a reviewer looking at samples can see neither. So when a synthetic dataset is proposed to you, ask for two numbers: downstream accuracy on real held-out data, and a nearest-neighbour search against the training set. A gallery is evidence of neither.

Train the model on the generated set, evaluate it on the real one, and report that number instead of the samples.

Analogy

Recognizing a person versus writing a biography

Two tasks sit side by side. One identifies which person appears in a photograph, while the other tries to describe the range of events, settings, and details that could belong in that person's biography.

Discriminative and generative modeling share this contrast between decision and broader structure. A statistical generator guarantees neither the understanding of a life that a biographer brings nor the factual coherence that comes with it.

A boundary can be useful without modeling everything, and a generator can be rich without being reliable.

Generation is not a substitute for causal simulation

A model trained on observational patterns can reproduce associations without ever representing an intervention or a physical mechanism. That is why synthetic scenarios may look coherent while responding incorrectly to changes outside the training distribution.

The size of that gap has been measured. Physics-IQ is 396 videos of 66 real physical scenarios, each one filmed twice, so that the disagreement between two genuine recordings of the same event fixes the 100% “physical variance” ceiling. The best model tested on it reached 24.1% of that ceiling. Motamed and colleagues report the finding in one line: “We find that across a range of current models (Sora, Runway, Pika, Lumiere, Stable Video Diffusion, and VideoPoet), physical understanding is severely limited, and unrelated to visual realism.” Unrelated is the load-bearing word. There was no significant correlation between visual realism and physical understanding, and Sora's videos were the hardest for a multimodal judge to tell from real footage. A separate group at UCLA and Google Research reached the same conclusion with a different benchmark. On VideoPhy, in 2025, the best model was CogVideoX-5B, and it produced videos obeying the caption and physical laws in 39.6% of instances.

When simulation supports policy or safety decisions, compare the model with mechanistic knowledge, controlled evidence, and domain-specific constraints. A generator that scores 24.1% against a physical ceiling is not a stand-in for the mechanism.

Key takeaways