Logistic Regression Explained

For binary classification, logistic regression combines features into a linear score. That score is then mapped through the logistic function. The result lies between zero and one and can be interpreted as a modeled conditional probability when the model is well specified and validated.

The decision threshold is a separate product choice. Training the probability model does not decide automatically which cases should be approved, alerted, or deferred.

The curve inside the classifier is eighty years old. It arrived from dose-response experiments rather than from computing. Joseph Berkson (1899–1982), chief statistician at the Mayo Clinic, published it in the Journal of the American Statistical Association in 1944. J. S. Cramer's history of the method records what happened there: “In 1944 he turned his attention to the statistical methodology of bio-assay and proposed the use of the logistic, coining the term ‘logit’ by analogy to the ‘probit’ of Bliss (for which he was initially much derided)”. Berkson was not quite first. Cramer notes that “Wilson was probably the first to publish an application of the logistic in bio-assay in Wilson and Worcester (1943), just before Berkson (1944)”. But it was Berkson “who persisted and fought a long and spirited campaign which lasted for several decades”. The method carries the name of the man who kept arguing for it.

The classifier contains several transformations. They should not be collapsed into one word such as prediction. An intercept plus weighted encoded features gives a linear predictor; the score is read on an additive log-odds scale; the logistic transform maps any real score into a value between zero and one; the result is an estimated probability that still needs calibration and shift checks; and a decision rule then applies thresholds, costs, capacity, or an abstention policy.

The last step is the one most easily mistaken for a formality. It is not, and there is a theorem to say so. Three fairness conditions on a risk score cannot be held together at once. Kleinberg and two co-authors proved it in 2017, and their abstract states it flatly: “We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously.” Their Theorem 1 names the constrained special cases. They are the two that never arrive in deployment: groups with equal base rates, or perfect prediction. Short of those, calibration within groups and equal average scores for the positive and negative classes cannot all hold at once. Alexandra Chouldechova reached the same collision independently that year, working on recidivism prediction instruments.

So the boundary drawn above between an estimated probability and a decision rule is not tidiness. Two reasonable requirements on one score provably collide as soon as the base rates differ. Which one to give up is chosen at the decision rule, not inside the likelihood.

A one-unit feature increase adds its coefficient to the log-odds when other encoded features are held fixed, and exponentiating the coefficient gives an odds ratio for that unit change.

Odds are not probabilities. The same odds multiplier produces different absolute probability changes depending on the starting probability.

Both can produce linear decision boundaries, but their fitted outputs and objectives differ.

Logistic regression fits a Bernoulli likelihood through a logistic link: its outputs stay within zero and one, its coefficients add on the log-odds scale, it supports regularization, and the quality of its probabilities still needs checking. A linear probability model fits a numeric zero/one target with linear regression: it can predict below zero or above one, its coefficients give direct probability-point slopes, it is a simple diagnostic baseline, and its error variance is inherently nonconstant. A thresholded score model uses a ranking score without claiming calibrated probability: it may optimize margin or another objective, thresholds can still drive decisions, the score ordering may be useful, and probabilistic language about it would be misleading.

Consider a feature with coefficient log(2), so a one-unit increase doubles the modeled odds.

Keep probability modeling, threshold policy, and evaluation explicit. Define the event first: the positive class, the horizon, the censoring, the prediction time. Construct the pipeline, encoding, scaling and regularizing within the training folds. Evaluate discrimination with ranking and class-separation metrics appropriate to the prevalence. Evaluate calibration by comparing predicted probability with observed frequency across ranges and slices. Choose thresholds from error costs, capacity and abstention rules rather than from 0.5 by habit. Then monitor shift: prevalence, feature support, calibration, and downstream outcomes.

Step one is where the largest failures are built in, because the positive class is chosen rather than found. A commercial risk score used across US health systems had been trained to predict health-care cost rather than illness. Obermeyer and three colleagues took it apart in Science in 2019. Cost is well recorded, legible and apparently neutral. It was also not the event anyone cared about: at any given score, Black patients were sicker than White patients. “Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.” That figure is step five as much as step one. The threshold was not moved; the label under it was. The paper did not name the vendor. On the day it appeared, 25 October 2019, New York's Department of Financial Services and Department of Health wrote jointly to UnitedHealth Group about Optum's Impact Pro, the algorithm press reporting identified as the one studied.

Steps three and four are listed separately because they fail separately, and a deployed classifier can show both failing at once. The Epic Sepsis Model runs in hundreds of US hospitals. An external validation at Michigan Medicine covered 38,455 hospitalisations and appeared in JAMA Internal Medicine in 2021. The hospitalisation-level AUC was 0.63, 95% CI 0.62–0.64. The model missed 67% of sepsis cases while firing alerts on 18% of all hospitalisations. Two thirds of the events went unflagged, and nearly one hospitalisation in five carried an alert. Wong and colleagues concluded: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.” An independent validation published in 2024, across two county emergency departments and 145,885 encounters, reported sensitivity of 14.7%, specificity of 95.3% and a positive predictive value of 7.6% within a six-hour window. Both quantities were measurable before deployment. Both were measured after it.

Case-control sampling may deliberately oversample positive examples, and slopes can remain useful under certain designs, but raw fitted probabilities reflect the sampled class ratio unless corrected.

Document sampling weights, prevalence, and deployment base rates before publishing probabilities.

The design has a theorem attached to it. What survives sampling on the outcome was settled in Biometrika in 1979, by Prentice and Pyke. They start from a logistic model for “the probability of disease development in a defined time period”, then induce from it the model for the regression variable given disease status. Applied to case-control data, they show “that the odds ratio estimators and their asymptotic variance matrices may be obtained by applying the original logistic regression model to the case-control study as if the data had been obtained in a prospective study”. Read the guarantee for what it names. It names odds ratios and their variances, in studies “in which stratum sizes are reasonably large”. It says nothing about the level of the fitted probabilities. The sampling decided how many cases and how many controls entered the data at all. The slopes travel; the intercept stays behind with the sampling design.

The modern version of the same trap is resampling a training set for balance. Monte Carlo simulations with standard and ridge logistic regression, under random undersampling, random oversampling and SMOTE, produced strong overestimation of minority-class probability with no gain in AUROC at all. The same classifications could be obtained by shifting the threshold instead — the operation the corrections were standing in for. That was van den Goorbergh and colleagues, in JAMIA in 2022. An independent group replicated it across ten clinical datasets and 605,842 patients, reporting ROC-AUC changes of −0.002 to −0.01: “Models trained using imbalance correction exhibited higher Brier scores (0.029 to 0.080, p<0.05), reflecting poorer probabilistic accuracy, and marked deviations in calibration intercept and slope, indicating systematic distortions of predicted risk despite preserved rank-based performance.” Ranking preserved, probabilities moved, and a threshold change would have done the job for nothing.