Skip to content
AI.info

Unsupervised learning

NMF, ICA, and Interpretable Factorizations

Compare non-negative matrix factorization, independent component analysis, and PCA as distinct models of latent structure.

By the end you can

Visual

Three factorization contracts

The methods preserve different properties and require different data conditions. The last of the ICA conditions is not a caution but a theorem. Comon proved it in 1994, and the restriction sits in the theorem's own first line: “Let x be a vector with independent components, of which at most one is Gaussian, and whose densities are not reduced to a point-like mass.” At most one. Sources that are all Gaussian fall outside Theorem 11 entirely.

Hyvärinen and Oja repeated the restriction in 2000, under a section heading that leaves no room: “Why Gaussian variables are forbidden”. Their statement of it is flat: “The fundamental restriction in ICA is that the independent components must be nongaussian for ICA to be possible.” The reason is geometric rather than numerical. Mix Gaussian sources orthogonally and the joint density comes out completely symmetric. It carries no information about the directions of the columns of the mixing matrix. There is nothing left for an algorithm to estimate.

FigureComparison · 3 columns

PCA

Orthogonal linear directions ordered by variance.

  • Allows positive and negative loadings
  • Uses centered data
  • Optimizes squared reconstruction
  • Does not seek independence

NMF

Non-negative factors whose product approximates non-negative data.

  • Supports additive parts interpretation
  • Requires non-negative input
  • Can have multiple local solutions
  • Needs rank and regularization choices

ICA

Linear components chosen to be as statistically independent and non-Gaussian as possible.

  • Supports source-separation assumptions
  • Has sign and permutation ambiguity
  • Needs whitening or preprocessing choices
  • Can fail with Gaussian or dependent sources

Variance, non-negativity, and independence are different organizing principles.

Analogy

Disassembling a color mixture under different laboratory rules

Three laboratories take the same mixture apart under three different rules: strongest variation, additive pigments only, and chemically independent sources. Each rule produces a different decomposition.

Chemical substances have independent physical existence. Statistical factors can be non-unique and objective-dependent. Several decompositions may reconstruct the same matrix while supporting different stories.

The decomposition rule determines what kind of “source” can appear.

Three decompositions, three different meanings of a component

One document-term matrix gives three different answers. PCA returns signed variance directions. NMF returns additive topic-like parts. ICA returns statistically independent signals. Calling all three outputs “latent factors” hides their different contracts.

Factorization methods do not discover one shared set of hidden causes. Each imposes a structure that makes one kind of decomposition possible.

A component is defined by an objective and constraints, not by the word latent.

Key idea

Component uniqueness is often weaker than visual interpretability suggests

NMF can settle in different local optima, ICA components can permute or flip sign, and nearly equal PCA eigenvalues permit rotation within a subspace. The first of those is a complexity result, not a quirk of a solver. Vavasis posted the proof in 2007 and published it in 2009, and his abstract lists what it delivers: “Then we establish several results about exact NMF: (1) that it is equivalent to a problem in polyhedral combinatorics; (2) that it is NP-hard; and (3) that a polynomial-time local search heuristic exists.” Multiple restarts are not sloppiness. A global solver is not on offer, and the heuristic that is on offer is local.

The ICA side has been measured rather than asserted. Three researchers built a tool called Icasso for that purpose, and their 2004 paper opens with the problem itself: “An important problem with most ICA algorithms is, however, that they are stochastic; that is, their results may be somewhat different in different runs of the algorithm.” Their test case was a 122-channel MEG recording of 17,730 samples, reduced to 20 dimensions by PCA. FastICA was rerun at M = 15 randomizations per setting. The instability was not total. Across repeated runs under different settings the top four component estimates were always ranked in the first four places, with the first and second only occasionally changing places.

Ten years on, a second tool called RELICA reported the same behaviour. With random starting weights, “the IC scalp maps for even the same data set may differ slightly across runs”. Its authors ran 150 bootstrap ICA decompositions per data set to quantify it. That is the price list for stability: 15 reruns in 2004, 150 in 2014.

So refit across seeds and samples, align components, and report reconstruction plus downstream behavior. Avoid naming a component from its largest loading alone.

A readable component is not automatically an identifiable component.

Case

Scale and order are ambiguities of the ICA model itself

ICA cannot tell you how loud a source is, and it cannot tell you which source is which. Both are properties of the model rather than defects of an implementation. Hyvärinen and Oja set them out in 2000, in a section titled “Ambiguities of ICA”, and they list two. The first: “We cannot determine the variances (energies) of the independent components.” The reason is arithmetic — “any scalar multiplier in one of the sources si could always be cancelled by dividing the corresponding column ai of A by the same scalar”. The second: “We cannot determine the order of the independent components.” Same kind of reason: “a permutation matrix P and its inverse can be substituted in the model”.

Six years earlier, in 1994, the whole indetermination had already been reduced to a single matrix. Comon's Property 2 and his Corollary 13 on identifiability leave a matrix DP undetermined, with D diagonal with entries of unit modulus and P a permutation. Under the heading “Preliminary observations” he closes the door on narrowing it: “This latter indetermination cannot be reduced further without additional assumptions.” Two independent bodies of work, six years apart, reach the same statement. No implementation is going to recover the scale or the order. The model never contained them.

Steps

Choose and validate a matrix factorization

Match the factor model to the support, objective, and interpretation the domain can defend. Steps 2 and 3 have a published protocol behind them rather than a general instruction to try several settings. The protocol is from 2004, in PNAS. Run the factorization repeatedly from random initial conditions. Average the per-run connectivity matrices into a consensus matrix. Score the stability of the result with the cophenetic correlation coefficient: “In a perfect consensus matrix (all entries = 0 or 1), the cophenetic correlation coefficient equals 1.” Brunet and colleagues report that 20–100 runs typically suffice. Rank selection stops being a judgement call and becomes a measurement. Their leukemia dataset was 38 bone marrow samples on Affymetrix Hu6800 chips, restricted to 5,000 genes.

The protocol travels. Gaujoux and Seoighe reimplemented it as an R package in 2010 and reproduced the rank estimation, plotting the cophenetic correlation coefficient for r in the range 2–5. Each point came from 50 runs, on the same 5000 x 38 Golub matrix of 27 ALL and 11 AML patients.

FigureProcess · 5 steps

  1. 1. Inspect data support

    Check non-negativity, centering, sparsity, scaling, and plausible mixing assumptions.

  2. 2. Fit simple baselines

    Compare PCA with NMF or ICA under several ranks and initializations.

  3. 3. Align components

    Handle sign, order, and split–merge variation before stability comparisons.

  4. 4. Review reconstructions

    Inspect residuals by entity, feature, rare slice, and held-out period.

  5. 5. Test interpretation

    Use domain exemplars, perturbations, and downstream tasks rather than attractive component plots.

Comparison

Interpretation traps across factor models

Readable component patterns are not automatically unique, causal, or stable.

FigureComparison · 3 columns

Parts-based story

NMF factors add together without cancellation.

  • Often intuitive for counts and images
  • Depends on non-negative encoding
  • Can split or merge topics across runs
  • Does not prove physical parts

Source-separation story

ICA seeks statistically independent latent signals.

  • Useful for mixed sensor sources
  • Requires a linear mixing approximation
  • Components can flip sign or order
  • Independence may not match domain sources

Variance story

PCA summarizes dominant covariance directions.

  • Stable and computationally mature
  • Can mix several sources
  • Uses orthogonality rather than independence
  • May prioritize nuisance variance

Example

Where each factorization earns a different interpretation

The data-generating story should precede the algorithm choice.

  • Topic extraction: NMF decomposes a non-negative document-term matrix into additive document weights and word profiles.
  • Audio separation: ICA can recover approximate independent sources from linear microphone mixtures when assumptions are plausible.
  • Image parts: NMF may learn non-negative localized patterns, while PCA eigenimages use positive and negative contrast directions.
  • Financial returns: PCA summarizes common covariance factors, but interpreting them as independent economic causes requires more evidence.
  • Sensor mixtures: ICA components can change order and sign across fits, so automated lineage needs alignment rules.

Factorizations are compact explanations under declared constraints

PCA, NMF, and ICA can all reduce dimension and reveal recurring structure. They answer different questions about variance, additivity, and independence.

Choose the method whose assumptions make sense for the data and intended use. Preserve component uncertainty and alternatives when the decomposition is not unique.

Do not ask one factor model to provide every notion of latent structure.

Case

Why non-negativity produced parts of faces rather than whole ones

Non-negativity comes with a contract, and Lee and Seung wrote it out in Nature in 1999. Their abstract gives the mechanism in one sentence: “These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations.” They set that against “principal components analysis and vector quantization, that learn holistic, not parts-based, representations”, and demonstrated an algorithm “that is able to learn parts of faces and semantic features of text”.

Whether the recovered parts are the real ones is a separate question. Four years later, at NIPS in 2003, Donoho and Stodden took it up and named a condition rather than offering reassurance: “We show that under certain conditions, basically requiring that some of the data are spread across the faces of the positive orthant, there is a unique such simplicial cone.” To test it they built the Swimmer image library: 256 images of 32x32 pixels, each carrying a 12-pixel torso in the center and four 6-pixel arms that can each be in one of 4 positions. Run on that library, Lee and Seung's own code recovered the 16 part and articulation generators correctly. The one element that never varied, the torso, was not resolved, and its ghosts contaminated several of the recovered generators. The failure is exactly where the geometric condition is not met.

What followed was a condition, not a retraction. Moitra records it in a 2014 draft of his machine learning textbook: “Motivated by issues of uniqueness and robustness, Donoho and Stodden [54] introduced a condition called separability that alleviates many of these problems”. Non-negativity buys additivity. It does not, on its own, buy the parts.

Key takeaways