Joint, Marginal, and Conditional Probability

A filter can manufacture a relationship out of nothing, and the size of the effect has been measured rather than assumed. Sackett measured it in 1979. He took household interviews of a random general-population sample — 2,784 people — and computed the relative odds between respiratory disease and disease of the bones and organs of movement. The answer was 1.06, which is to say nothing at all. He then restricted the same interviews to the 257 of those people who had been in hospital in the previous six months. Inside that subgroup the cross-product odds were 4.06.

Oxford's Catalogue of Bias records the result: “He analysed data from 257 hospitalized individuals and detected an association between locomotor disease and respiratory disease (odds ratio 4.06).”

Same people. Same two diseases. Same questionnaire. The only thing added was a filter, and the filter is what produced the fourfold association. This is Berkson's bias, and admission is the collider being conditioned on.

A joint distribution describes several random variables together. Sum or integrate a variable out and you get a marginal distribution. Restrict attention to what is already known and you get a conditional one. These operations change the question. P(Y) describes overall frequency; P(Y|X=x) describes frequency within a specified context. Machine learning leans hard on conditional distributions, because prediction asks how uncertainty about a target changes after observing features.

The two answers are not obliged to agree. The rest of this lesson works through files where they did not: graduate admissions counts in which the marginal favours one group and four of six conditionals favour the other, a surgical series in which the marginal and both strata point opposite ways, and a sample of 257 hospital patients in which a filter alone moved an odds ratio from 1.06 to 4.06.

A two-variable joint distribution supports several related calculations, and the six-department Berkeley table shows all of them on one object. The joint P(sex, department, admission) is the array of counts over 4,526 applicants. Marginalize department away and you get 1198 of 2691 men admitted, 44.5%, against 557 of 1835 women, 30.4%. Condition on department A instead, renormalize inside it, and you get 62.06% against 82.41%. Reverse the conditioning — ask what mix of departments a rejected applicant applied to — and Bayes' rule takes you back the other way through the same counts. No number was recomputed between those three statements. Only the denominator changed.

For variables X₁,…,Xₙ, the joint can be written as P(X₁)P(X₂|X₁)…P(Xₙ|X₁,…,Xₙ₋₁). This identity requires no independence assumption. Independence assumptions simplify the factors. A graphical model encodes which conditioning variables are retained and which are omitted, so a factorization is both a computational device and a claim about dependence structure.

The Berkeley table is a three-variable instance. P(sex, department, admission) factorizes as P(sex) P(department | sex) P(admission | sex, department), with no assumption at all. The middle factor is the one that carried the aggregate result: women applied more often to departments that admitted fewer of everyone. Drop it — assert that department is independent of sex — and the model predicts a single admission rate for each sex, which is the 44.5% against 30.4% margin. Keep it, and the model reproduces 62.06% against 82.41% in A and 5.90% against 7.04% in F. Choosing which conditioning variables to omit is choosing which of those two answers your model can express.

These terms express different levels of separation, and a surgical series shows why the distinction is not academic. A 1986 report in the BMJ compared four treatments for kidney stones: “Success was achieved in 273 (78%) patients after open surgery, 289 (83%) after percutaneous nephrolithotomy, 301 (92%) after ESWL, and 15 (62%) after percutaneous nephrolithotomy and ESWL.” Marginally, then, percutaneous nephrolithotomy removed kidney stones more often than open surgery: 289 of 350 against 273 of 350, 83% against 78%.

Condition on stone diameter and the ordering inverts in every stratum. In the stratified table reproduced from Julious and Mullee, open surgery succeeded in 81 of 87 cases (93%) against 234 of 270 (87%) for stones under 2 cm, and in 192 of 263 (73%) against 55 of 80 (69%) for stones of 2 cm or more. Open surgery wins in both strata and loses overall.

Marginal independence, conditional independence and zero covariance are three separate statements about the same joint distribution. 78 against 83, reversing to 93 against 87 and 73 against 69, is what it costs to confuse them.

Before simplifying a joint model, work through the sequence below: draw the variables, state the joint question, identify what is being conditioned on, test the proposed independences, and check the selection effects last.

The last step is the one with a published benchmark. UK Biobank is a selected sample by construction. Approximately 9.2 million people aged 40–69, living within 25 miles of one of 22 assessment centres in England, Wales and Scotland, were invited. Of those, 5.5% took part, giving a cohort of about 500,000. Set that against surveys with conventional response rates and the difference is stark — “Analytical sample of 499 701 people (response rate 5.5%) in analyses in UK Biobank; pooled data from the Health Surveys for England (HSE) and the Scottish Health Surveys (SHS), including 18 studies and 89 895 people (mean response rate 68%).” That comparison appeared in the BMJ in 2020.

A separate study in the American Journal of Epidemiology describes who the 5.5% are: more likely to live in less socioeconomically deprived areas, less likely to be obese, to smoke or to drink daily, and carrying fewer self-reported health conditions than the general population. A joint distribution estimated on that cohort is the joint distribution of the people who agreed to join it. That is the same structural fact as Sackett's 257 patients, at a scale of half a million.