Linear Regression Fundamentals

Linear regression predicts a numeric outcome by adding an intercept to weighted feature values. The apparent simplicity is what makes it useful. Every modeling choice is left in the open: representation, scale, interactions, loss, and residual error.

The relationship is linear in the coefficients. The raw inputs may be transformed first, through logs, splines, indicators, or interactions. That freedom is where the judgement lives. A column defined as 1000(Bk − 0.63)² is linear in its coefficient like any other. The model cannot tell you that someone chose the square, the centre and the scale before the estimator ever ran.

Ordinary least squares picks a single prediction vector. It is the vector in the column space of the design matrix that is closest to the observed target.

Four objects are worth naming. The observed target is the vector of numeric outcomes in the training sample. The feature span is every prediction vector obtainable from the chosen columns and transformations. The fitted values are the orthogonal projection of the target onto that span. The residual vector is what is left over — the component the representation cannot fit.

The estimator minimizes the sum of squared residuals between observed and predicted outcomes. Squaring makes large errors influential. It also produces a convenient convex objective when the design is fixed.

This objective is not automatically the business cost. A few extreme errors can dominate the fit even when operational loss grows differently.

The correct interpretation depends on design, preprocessing, and causal assumptions. The cheapest way to see that is to watch what happens to a coefficient when one column is added, and what happens to its sign when the data-generating process is not what the model assumes.

Start with conditioning. In autumn 1973, 4,526 people applied to the six largest graduate departments at the University of California, Berkeley. The aggregate figures sit on the manual page for R's UCBAdmissions data: “There were 2691 male applicants, of whom 1198 (44.5%) were admitted, compared with 1835 female applicants of whom 557 (30.4%) were admitted. This gives a sample odds ratio of 1.83, indicating that males were almost twice as likely to be admitted.” Now condition on the department applied to. The association disappears. Pooled with departmental autonomy taken into account, a 1975 paper in Science found instead “a small but statistically significant bias in favor of women”. No new data arrived. One column entered the design matrix, and every other coefficient came to mean something else.

Direction is not safe either. A pneumonia cohort of 14,199 patients — 9,847 for training, 4,352 for test — was reported in 1997, with “Has asthma” among the candidate predictors. Models fitted on it carried a negative weight on asthma. They had learned that a history of asthma lowers the risk of dying of pneumonia. The sign was an artefact of triage: asthmatic pneumonia patients were sent directly to intensive care. Caruana and five colleagues set it out in the first section of their 2015 paper: “The bad news is that because the prognosis for these patients is better than average, models trained on the data incorrectly learn that asthma lowers risk, when in fact asthmatics have much higher risk (if not hospitalized).” The coefficient predicted well and pointed the exact opposite way from the intervention it appeared to describe.

The most common misreading of a regression table has a name. It is the Table 2 fallacy, and Westreich and Greenland gave it that name in the American Journal of Epidemiology in 2013. The habit they describe is a familiar one: “a table might show odds ratios for one or more exposures and also for several confounders from a single logistic regression”. That, they argue, “can lead to mistaken interpretations of these estimates”. A single fitted model does not produce one kind of quantity. Listing the coefficients side by side invites “confusion of direct-effect estimates with total-effect estimates for covariates in the model”. The covariate coefficients “may also be confounded even though the effect estimate for the main exposure is not confounded”. Their remedy is not a better-annotated table. It is more models — “use of multiple models tailored to yield total-effect estimates for covariates”.

Three readings of the same number, then. A predictive slope is the expected change in prediction for one feature change while encoded peers stay fixed. It is conditional on the included columns, measured in target units per feature unit, sensitive to scaling and transformation, and not automatically causal. Berkeley shows how much that first clause carries. A standardized coefficient is that slope after features are placed on a common scale. It is useful for numerical comparison, it depends on the chosen standard deviation, it is still sensitive to correlation, and it is not a measure of feature importance. A causal effect is the change produced by an intervention under an identification strategy. It needs assumptions beyond regression, often randomization or adjustment, and it fails under confounding. The asthma weight is what claiming it from fit alone looks like.

For decades the first regression a student ever fitted was a small hedonic price model on the Boston housing data. Harrison and Rubinfeld published it in 1978: 506 rows, 14 columns, small enough to fit in a lecture and to print in a textbook appendix.

Now read one of the columns. It is defined as 1000(Bk − 0.63)², where Bk is the proportion of black residents by town. Nobody measured that quantity. Somebody chose the square, chose the centre, and wrote the result into the design matrix before least squares saw a single row. The scikit-learn project said so plainly in the deprecation notice on its loader page: “The Boston housing prices dataset has an ethical problem: as investigated in [1], the authors of this dataset engineered a non-invertible variable "B" assuming that racial self-segregation had a positive impact on house prices [2].” load_boston was deprecated in version 1.0 and removed in version 1.2.

This is what "representation is a choice" costs when the bill arrives. Work through what the column does to every interpretation the fitted model can support.

The algebra is easy. The contract around it requires care. Six steps, in order.

1. Define units — record target and feature units before scaling or transformation. 2. Build the design matrix — add indicators, transforms, interactions, and an intercept intentionally, and write down why each one is there; 1000(Bk − 0.63)² is what an unexamined step two leaves behind. 3. Inspect rank and correlation — find duplicate columns, near-collinearity, and unsupported contrasts. 4. Fit within a pipeline — keep imputation, encoding, and scaling inside the split. 5. Examine residuals — look by range, group, time, and influential observations; Anscombe's fourth set is eleven points whose whole slope rests on one of them. 6. Report conditional meaning — state what is held fixed, and avoid causal language without a design.

For point prediction, the most important question is whether the fitted relationship generalizes under the deployment distribution. Classical standard errors and hypothesis tests require additional assumptions. They concern sampling, dependence, variance, and model specification.

Do not import inferential confidence from software output without checking the data-generating design.

Google Flu Trends is what that failure looks like at scale. The service had fitted 50 million candidate search terms to 1,152 data points — a search wide enough to turn up terms that tracked the training window for reasons that had nothing to do with influenza. Four researchers reported that in Science on 14 March 2014. Under the heading "Big Data Hubris" they gave their verdict in one sentence: “In short, the initial version of GFT was part flu detector, part winter detector.” The fit was excellent and the deployment was not. From August 2011 the model missed high in 100 of the 108 weeks. A later independent reappraisal by Kandula and Shaman found GFT's mean squared error to be on average 2.5 times that of a simple lagged ILI estimate. The service was discontinued in 2015.

Nothing in the fitted output announced any of this. The training-window statistics stayed good while the thing the model was for stopped working.