Matrices and Linear Transformations

Linear transformations rotate, reflect, scale, shear, project, or combine these effects. Whatever geometric word fits the picture, the arithmetic underneath is always the same one. Each output coordinate is a weighted sum of the input coordinates, and the weights are the entries of the matrix. The geometry is a consequence of the weights, not a separate mechanism.

A matrix may store data, with rows as examples and columns as features. The same mathematical object may also act as a transformation that maps one vector space into another. These two roles meet in expressions such as Xw. The data matrix X collects examples. The vector w defines a linear rule. The product returns one score for each row. Matrix fluency comes from reading shapes and actions together: an expression should tell you what enters, what leaves, and which dimensions are being combined.

The word came before the algebra. In 1850 Sylvester coined “matrix” for an oblong arrangement of terms that gives rise to determinants. The operations arrived eight years later, in a memoir read to the Royal Society in January 1858. MacTutor's history of matrices records what that memoir did: “Cayley in 1858 published Memoir on the theory of matrices which is remarkable for containing the first abstract definition of a matrix”. The same entry lists what was in it — “Cayley gave a matrix algebra defining addition, multiplication, scalar multiplication and inverses”.

The founding paper does not begin from a grid either. Cayley starts from a set of linear equations, X = ax + by + cz, …, and treats the array of coefficients as an object in its own right. The double life this lesson is about — an array of numbers that is also an action — is the move that paper makes on its first page.

If B maps inputs to an intermediate representation and A maps that representation to outputs, then AB applies B first and A second. Order matters, because the spaces have to line up. Rotating and then stretching can produce a different result from stretching and then rotating.

That non-commutativity is not a caution teachers added later. It is in the founding document, flagged there as the surprise. Writing about the multiplication of matrices in his opening paragraph, Cayley notes that “there is the peculiarity that matrices are not in general convertible”. The first abstract definition of a matrix and the warning that AB need not equal BA sit on the same page of the same paper, read in January 1858.

Shape checking makes the composition visible. For A with shape m×n and B with shape n×p, the shared n is summed out, leaving an m×p transformation.

Consider y = Wx + b with x in ℝ³ and y in ℝ². The conventions below are not this lesson's house style. They are what the two dominant frameworks document. The PyTorch 2.9 documentation describes its dense layer this way: “Applies an affine linear transformation to the incoming data: y = xA^T + b”. Affine, not linear, in the library's own text. It stores the weight with shape (out_features, in_features) and the bias with shape (out_features). Keras documents the same layer as “output = activation(dot(input, kernel) + bias)”, taking input shape (batch_size, input_dim) to output shape (batch_size, units). A different organisation, the same batch × features convention, with the transpose absorbed into the kernel orientation.

These operations are often grouped together, but they answer different questions and carry different requirements. The pseudoinverse even carries two dates, because being defined and being computable were separate achievements ten years apart.

The transpose Aᵀ is always defined. It turns m×n into n×m, swaps rows and columns, reverses the direction of an inner-product map, and shows up in gradients and normal equations. It is not an inverse in general. The ordinary inverse A⁻¹ exists only under specific conditions — a square matrix of full rank — and stays numerically fragile when A is ill-conditioned.

The pseudoinverse A⁺ is the generalization, and its reach is the point of the paper that defined it. Penrose set it out in 1955 as the unique solution of a set of equations, and the abstract states the scope: “This generalized inverse exists for any (possibly rectangular) matrix whatsoever with complex elements.”

Existence is not an algorithm. Ten years later, in 1965, Golub and Kahan gave the numerically stable route: bidiagonalize, diagonalize, then form A⁺ = VΣ⁺U* by replacing each positive singular value with its reciprocal. Their abstract offers “the use of the pseudo-inverse … to solve least squares problems in a way which dampens spurious oscillation and cancellation”. That reciprocal is also where the fragility lives, because a positive singular value close to zero becomes a very large reciprocal. So A⁺ produces least-squares or minimum-norm solutions, is sensitive to small singular values, and still makes no lost information recoverable.

This routine prevents most dimension errors and many conceptual ones. Label every axis with what it represents. State the source space and the target space of each matrix. Read the multiplication right to left, so you know which transformation acts first and which dimension is summed. Ask what the map preserves — lengths, angles, rank, information. Only then choose the numerical operation that matches the mathematical goal: multiplication, factorization, or a solver.