Random Variables and Probability Models

Each layer answers a different question, and the vocabulary is not loose talk. These are the definitions a measurement standard is willing to be held to. The international Guide to the expression of uncertainty in measurement — the GUM — gives each one its own numbered clause. A random variable there is a variable with an associated probability distribution: the variable is defined first, on the outcomes, and the distribution is attached to it afterwards. The distribution function, the density and the mass function each get a clause of their own. On the density the standard is careful. It calls f(x)dx the “probability element”, not a probability.

The layers are separable because each can fail on its own. A sample space can omit an outcome the world still produces. A random variable can be a perfectly good function of outcomes the model has mis-specified. A distribution can be internally consistent and empirically wrong. Only the last layer, summaries and decisions, has no clause in any standard: what the number is for is not a mathematical question.

Probability begins with a set of possible outcomes and a rule that assigns probabilities to events. A random variable maps each outcome to a numerical value. This separation matters. The uncertainty belongs to the modeled outcome, while the random variable is the quantity we choose to observe or summarize.

A model may be mathematically coherent yet empirically poor, and the cost of that gap is not always academic. In R v Clark the prosecution's headline figure was that the chance of two sudden infant deaths in one family was 1 in 73 million. The arithmetic behind it was a single multiplication. The Court of Appeal's judgment records Professor Meadow's evidence that “you have to multiply 1 in 8,543 times 1 in 8,543 ... approximately a chance of 1 in 73 million”. Squaring a rate is a legitimate operation inside a model in which the two deaths are independent events. Whether that model applies to families was never established.

The Royal Statistical Society said so in a news release in October 2001: “This approach is, in general, statistically invalid. It would only be valid if SIDS cases arose independently within families, an assumption that would need to be justified empirically.” The same release stated that the 1 in 73 million figure “has no statistical basis”. The judgment also records Professor Berry's objection that “simply squaring the figure was an illegitimate over simplification”. On 11 April 2003 the Court of Appeal held that it was very likely the figure “grossly overstates the chance of two sudden deaths within the same family from unexplained but natural causes”.

Probability theory tells us how conclusions follow from assumptions; data and model checking tell us whether those assumptions are useful. Nothing inside 8,543 × 8,543 could have raised the independence assumption. It had to be written down as an assumption, and then tested against families.

The three functions describe distributions in different ways. The distinctions are formal enough that a metrology standard writes them out one clause at a time. In the GUM the distribution function is “a function giving, for every value x, the probability that the random variable X be less than or equal to x”. That is the CDF, and the definition covers discrete and continuous variables alike. The probability density function is the derivative of that distribution function, with f(x)dx named the “probability element”. The probability mass function gives pi = Pr(X = xi) for a discrete variable.

A US federal agency states the same split independently. The NIST/SEMATECH e-Handbook of Statistical Methods writes the discrete case as P[X = x] = p(x), with the values summing to one. It writes the continuous case as p[a ≤ x ≤ b] = ∫f(x)dx, with ∫f(x)dx = 1 over the line. Two institutions, one boundary. The mass function hands you a probability. The density hands you probability per unit, and only its integral over an interval is a probability.

A density is probability per unit of the variable. If a uniform variable lies between 0 and 0.1, its density is 10, so that the area over the interval equals one. The number 10 is not a probability. The NIST/SEMATECH e-Handbook of Statistical Methods puts the whole point in three sentences: “Probabilities are measured over intervals, not single points. That is, the area under the curve between two distinct points defines the probability for that interval. This means that the height of the probability function can in fact be greater than one.”

How much greater is a question of units, and the GUM works a case where the answer is spectacular. It takes the coefficient of linear thermal expansion of pure copper at 20 °C, 16,52 × 10⁻⁶ °C⁻¹, and models the possible values by “this symmetric rectangular distribution of possible values” of half-width a = 0,40 × 10⁻⁶ °C⁻¹. The height of that density is 1/(2a) = 1.25 × 10⁶ °C. Over a million, in an official worked example, and its integral is still exactly one. The variable itself is measured in millionths.

Confusing density with probability leads to incorrect likelihood interpretations and unit errors. A likelihood of 1.25 × 10⁶ is not a certainty. A likelihood ratio between two densities is only meaningful when both are expressed per the same unit. Likelihood values can exceed one for continuous models, for exactly the reason the copper example shows.

One deployed setting can support several different probability models, and the choice of experiment decides what the resulting number even counts. Epic's proprietary sepsis prediction model was externally validated at Michigan Medicine over 2018–2019, and the verdict published in JAMA Internal Medicine in 2021 was blunt: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.” The cohort is described two ways in the same sentence of the paper's own methods — 27,697 patients and 38,455 hospitalizations. Every headline figure below belongs to one of those units, not to both.

Start with the uncertain process rather than a favorite distribution. The first four steps are the ones that get done. Step five, checking consequences against recorded frequency, is the one that gets skipped, and it is the only step that can catch a model that is coherent in every other respect.

That check was run on image classifiers in 2017, and the paper opens with the finding: “We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated.” Guo and colleagues bin predicted probabilities into 15 bins and report expected calibration error. On CIFAR-100 a 110-layer ResNet is 16.53% off. A 5-layer LeNet on the same data is 4.85%. The older and less accurate model is the better probability model — “the average confidence of LeNet closely matches its accuracy, while the average confidence of the ResNet is substantially higher than its accuracy”. A single fitted parameter, temperature scaling, brings the ResNet from 16.53% to 1.26%. The ordering of its predictions was never the problem. The probabilities attached to that ordering were.

Step five also has to be repeated, because its findings expire. In 2021 Minderer and colleagues revisited the same question at a different venue and found that “the most recent models, notably those not using convolutions, are among the best calibrated”. The check is the deliverable, not the conclusion drawn from one run of it.