Skip to content
AI.info

Unsupervised learning

Gaussian Mixtures, EM, and Soft Membership

Learn Gaussian mixture models, expectation–maximization, covariance choices, model selection, and the limits of probabilistic cluster membership.

By the end you can

Example

What soft mixture output can and cannot tell you

Responsibilities support useful diagnostics when their conditional nature stays explicit. Every failure mode below is a published result, not a folk warning. That is what turns a caution into a check you can actually run.

  • Boundary case: A point with responsibilities 0.52 and 0.46 lies in an overlap region rather than carrying a certain hard identity. The number is a posterior under the mixture that was fitted, not a probability of belonging to a natural class.
  • Component collapse: With unconstrained component variances the likelihood of a normal mixture is unbounded. Chen and Li proved that in 2009 in The Annals of Statistics; they did not merely warn about it. So a full-covariance component shrinking around a few points is the algorithm following its own objective, not misbehaving.
  • Model comparison: BIC may favor fewer components by penalizing parameter count, while downstream use may still justify a finer representation. Schwarz derived that penalty in 1978, and he derived it for regular models. Its consistency for choosing the number of components in a finite mixture had to be proved separately, by Keribin in 2000.
  • Density estimation: The fitted mixture can score new observations, but poor Gaussian fit makes low density difficult to interpret, and the density is reported on the model's terms rather than the data's.
  • Label switching: The problem has a name because Matthew Stephens gave it one in 2000: “This is due to the so-called 'label switching' problem, which is caused by symmetry in the likelihood of the model parameters.” Papastamoulis states the same invariance independently: “In case that the prior distribution of the model parameters is the same for all states, then both the likelihood and posterior distribution are invariant to permutations of the parameters.” Stephens also records that removing the symmetry “by using artificial identifiability constraints” fails in general. That is why Papastamoulis's label.switching package ships one probabilistic and seven deterministic relabelling algorithms instead of one rule.

A thousand crabs from the Bay of Naples

One histogram, two overlapping populations, and nothing written on any individual crab saying which population it came from. That was the problem in 1894, and Karl Pearson answered it by fitting a two-component normal mixture by the method of moments to one measured distribution. Weixin Yao describes that data set exactly: “The data set consists of the measurements on the ratio of forehead to body length of 1000 crabs sampled from the bay of Naples.”

The case has not aged out of the literature. Chen and Li open their 2009 paper in The Annals of Statistics with it — “More than a hundred years ago, Pearson (1894) modeled a set of crab observations with a two-component normal mixture distribution.” — and Yao refit the same crab data in 2010, following Pearson, with a two-component normal mixture of his own.

That is the whole move a Gaussian mixture makes, and it is the reason it is worth learning. A hard partition would have assigned each of the 1000 crabs to one group and hidden the overlap. The mixture assigns substantial responsibility to both components where the two densities meet. Soft membership can represent overlap, but only within the model's distributional assumptions. Probability language does not remove the need to question the components.

A responsibility is conditional on the fitted mixture, not a universal identity probability.

Key idea

The unbounded likelihood is a theorem, not a numerical nuisance

Put one component mean on a single observation, shrink that component's variance toward zero, and the likelihood diverges. That is not a bug in anyone's code. It is a theorem. Chen and Li state it in their 2009 abstract: “The likelihood function of the normal mixture model is unbounded based on a set of random samples, unless an artificial bound is placed on its component variance parameter.” Yao, in a different journal and at a different institution, reaches the same conclusion — “It is well known that the normal mixture with unequal variance has unbounded likelihood and thus the corresponding global maximum likelihood estimator (MLE) is undefined” — and names the degenerate path outright: “if we set m1 = x1 and let s²1 → 0, the likelihood value goes to infinity”. The global maximum likelihood estimator does not exist. Any run that stops somewhere sensible stops because something stopped it.

That something is a modelling choice with a published default. scikit-learn's GaussianMixture carries reg_covar float, default=1e-6, documented as “Non-negative regularization added to the diagonal of covariance. Allows to assure that the covariance matrices are all positive.” It carries n_init int, default=1 — “The number of initializations to perform. The best results are kept.” Call the class without arguments and you are fitting a ridge of 1e-6 from a single initialization. What you report is a constrained estimate. The constrained-EM alternative, bounding the ratio of component standard deviations, does not escape the problem either. Yao's abstract says so: “However, choosing the constraint parameter is a difficult issue and in many cases different choices may give different constrained global MLE.”

So monitor convergence warnings, smallest eigenvalues, effective component mass and sensitivity to starts. Report the bound and the number of initializations alongside the likelihood. A larger likelihood obtained by moving the bound is not a better fit. It is a different model.

With unconstrained component variances there is no maximum to find, so the bound you impose is part of the model you report.

Analogy

Mixing several overlapping radio stations

A received signal carries several stations at once, each with its own strength and noise pattern, and for each moment you estimate how much each station contributed, then refine the station models.

Mixture observations are not literal sums of component samples in ordinary clustering. Each point is modeled as arising from one latent component, while uncertainty remains about which one.

Soft assignment expresses model uncertainty about a latent source.

Steps

Fit and qualify a Gaussian mixture

Treat covariance family and component count as hypotheses rather than defaults. And before trusting your own comparison, read one that was published in full.

The wine data is 178 samples, 13 chemical measurements, three Italian cultivars — Barolo, Grignolino, Barbera. The UCI Machine Learning Repository, which holds it as dataset 109, describes it this way: “These data are the results of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars.” mclust 5 runs steps 2 and 4 on that data together.

Searching all 14 parameterisations against BIC, mclust 5's authors report the selected model as EVE — ellipsoidal, equal volume, equal orientation, variable shape — with three components. BIC −6873.257, log-likelihood −3032.45, 156 estimated parameters. The next model in their table stood at −6896.837, a gap of 23.58 points, and the one after it at −6906.375. Note which family did not win: the unconstrained full-covariance one. The constrained model split the data 63/51/64 and matched the true cultivars with an adjusted Rand index of 0.8803998.

That is what steps 2 and 4 are for. A run left at the library default would have produced a different answer without ever having made the comparison. The difference here was worth about 24 BIC points and a named geometry.

FigureProcess · 5 steps

  1. 1. Prepare numerical geometry

    Scale features, review outliers, and decide whether Gaussian tails are plausible.

  2. 2. Compare covariance families

    Fit simpler and richer shapes under several initializations.

  3. 3. Inspect diagnostics

    Review likelihood, convergence, component mass, covariance condition, and responsibility entropy.

  4. 4. Compare K carefully

    Use AIC, BIC, held-out likelihood, stability, and intended use together.

  5. 5. Test new points

    Evaluate density and responsibility behavior on later, shifted, and boundary observations.

Comparison

Covariance choices define component shape and parameter cost

The four families below are the ones a scikit-learn user meets, because that library exposes exactly four: “covariance_type{'full', 'tied', 'diag', 'spherical'}, default='full'”. That is not the size of the actual menu.

mclust 5 decomposes each component covariance through the eigen-decomposition Σk = λk Dk Ak Dkᵀ. Volume, shape and orientation become three separate switches, and each one may be held equal across components or left free. Its 2016 paper in The R Journal states the count plainly: “Thus, 14 possible models with different geometric characteristics can be specified.” Table 3 names all fourteen: EII, VII, EEI, VEI, EVI, VVI, EEE, EVE, VEE, VVE, EEV, VEV, EVV, VVV.

The Mixmod library reaches the same count from an independent codebase and journal. Its 2015 paper in the Journal of Statistical Software captions Table 1 “Some characteristics of the 14 models” and explains that “The parameter λk determines the volume of the kth cluster, Dk its orientation and Ak its shape.” It also gives the parameter counts rather than an adjective: from α+β for [λDADᵀ] up to α+Kβ for [λkDkAkDkᵀ], with β = d(d+1)/2. So the cost of letting every component keep its own full matrix is not vaguely high. It is one d(d+1)/2 term per component.

Read the four families below as coarse groupings of that menu, and report the parameterisation you chose by name.

FigureComparison · 4 columns

Spherical

Each component has one variance shared across directions.

  • Few parameters
  • Round component geometry
  • Stable with limited data
  • Cannot model directional spread

Diagonal

Each feature has its own variance without covariance terms.

  • Axis-aligned ellipses
  • Moderate parameter count
  • Misses correlated directions
  • Sensitive to feature rotation

Tied full covariance

All components share one full covariance matrix.

  • Captures common correlation structure
  • Reduces parameters relative to separate full matrices
  • Forces identical orientation and shape
  • Can stabilize smaller components

Full covariance

Each component has its own covariance matrix.

  • Flexible ellipsoidal components
  • High parameter cost
  • Can become singular
  • Needs sufficient support per component

Visual

The alternating logic of expectation–maximization

EM repeatedly estimates soft assignments, then updates component parameters. The 1977 paper that named it, by Dempster and two colleagues in the Journal of the Royal Statistical Society Series B, promised two things: “A broadly applicable algorithm for computing maximum likelihood estimates from incomplete data”, and “Theory showing the monotone behaviour of the likelihood and convergence of the algorithm”.

The first half held. The convergence conditions did not. In 1983 Russell A. Boyles gave an explicit counterexample, a GEM sequence that does not converge under those conditions, and stated what does survive: “Two general convergence results are presented which suggest that in practice a GEM sequence will converge to a compact connected set of local maxima of the likelihood function; this limit set may or may not consist of a single point.” In the same year C. F. Jeff Wu re-derived the actual guarantee in The Annals of Statistics. Limit points are stationary points of the observed-data likelihood. Convergence to the maximum likelihood estimate follows only when the likelihood is unimodal and a differentiability condition holds.

So the loop below is exact about what it does — improve the likelihood at each step — and silent about where it lands. Stationarity is what you are given. A maximum is a further condition, and with unconstrained variances there is no global maximum to reach at all.

FigureProcess · 5 steps

  1. 1. Initialize components

    Choose weights, means, and covariance parameters for K Gaussian components.

  2. 2. Expectation step

    Compute each point’s responsibility for every component using current parameters.

  3. 3. Maximization step

    Update weights, means, and covariances using responsibility-weighted statistics.

  4. 4. Evaluate likelihood

    Measure how well the mixture assigns probability density to the observations.

  5. 5. Repeat

    Continue until likelihood improvement or parameter change meets a stopping rule.

EM guarantees stationary points of the observed-data likelihood; reaching the maximum likelihood estimate is an extra condition, not a consequence.

Probabilistic output is only as honest as the mixture model

Gaussian mixtures are valuable when overlapping ellipsoidal components provide a credible approximation. They offer soft memberships, density estimates, and a clear generative contract. That contract can still be wrong.

The selection criterion deserves the same scrutiny as the model. Gideon Schwarz introduced BIC in 1978, in The Annals of Statistics, like this: “The problem of selecting one of a number of models of different dimensions is treated by finding its Bayes solution, and evaluating the leading terms of its asymptotic expansion. These terms are a valid large-sample criterion beyond the Bayesian context, since they do not depend on the a priori distribution.” Read that sentence for what it assumes. The expansion is derived for regular models. A finite mixture is not one, which is why the mixture case had to be proved on its own. A 2017 survey in Statistica Sinica records the result — “Keribin (2000) investigated the consistency of the maximum penalized likelihood estimator for an appropriate penalization sequence” — citing “Keribin, C. (2000). Consistent estimation of the order of mixture models. Sankhyā A 62, 49-66”. Nguyen and Nguyen state the price of that proof: “the Bayesian information criterion (BIC) was proved consistent in mixtures, but under strong regularity, including high moments and high-order derivatives of the component density”.

So report the whole decision, not half of it: the covariance parameterisation by name, the number of components, the regularization constant and the number of initializations, the degeneracy checks, and the alternative K values you rejected and by how many BIC points. Reporting K without naming the covariance model reports only half of the choice. Reporting a BIC gap without the parameterisation it ranged over reports none of it.

Soft labels deserve hard scrutiny of the distributional assumptions.

Key takeaways