Dense Layers: Many Neurons, One Matrix

Writing one weighted sum per output unit is conceptually clear but computationally awkward. Stack the weight vectors as rows or columns, follow a declared convention, and the layer becomes one matrix operation.

With batch-first notation, a common form is Y=XW+b. The orientation is not a matter of taste. The two most-used frameworks store the same layer transposed relative to each other. Keras 3's Dense(units) holds a kernel of shape (input_dim, units) — I×O — and its documentation states the operation outright: “Dense implements the operation: output = activation(dot(input, kernel) + bias)”. PyTorch's torch.nn.Linear(in_features, out_features) computes y = xA^T + b, and holds weight with shape (out_features, in_features) — O×I, the transpose of the Keras layout. NVIDIA's own performance guide records the same split. It adopts “the convention used by PyTorch and Caffe where A contains the weights and B the activations” and notes that “In TensorFlow, matrices take the opposite roles”. Both conventions implement the same affine map. Only the declared shapes tell you which one you are holding, which is why shape reasoning beats memorising a layout.

Much of the matrix is redundant, and that was shown before it was exploited. Denil and colleagues reported it at NIPS 2013: the parameters of a neural network are predictable enough from one another that “in the best case we are able to predict more than 95% of the weights of a network without any drop in accuracy”. Full connectivity is a statement that any coordinate may matter to any output. It is rarely a statement that all of them do.

Suppose a batch contains B examples. Each carries I input features, and the layer emits O features.

Those three integers are a hardware decision as well as a bookkeeping one. Forward propagation, activation-gradient computation and weight-gradient computation “are directly expressed as matrix-matrix multiplications”, NVIDIA's guide for these layers says, and its Quick Start Checklist gives the instruction plainly: “Choose the batch size and the number of inputs and outputs to be divisible by 4 (TF32) / 8 (FP16) / 16 (INT8) to run efficiently on Tensor Cores.” The worked example is an output width of 33,708 — a vocabulary. At that width “Tensor Cores cannot be applied and performance reduces drastically”; adding four padding tokens to reach 33,712 restores them. PyTorch's own performance tuning guide gives the same instruction: “set sizes to multiples of 8 (to map onto dimensions of Tensor Cores)”. Four columns out of thirty-three thousand decide whether the multiply runs on the fast path at all. B, I and O are chosen, not merely recorded.

Parameter counts reveal memory use and capacity before a model is trained, and they depend on exactly two of the three numbers above: the input width and the output width.

Dense layers make few assumptions about which coordinates should interact. That flexibility can become expensive.

The flexibility is not a design preference; there is a theorem underneath it. Finite linear combinations of a fixed sigmoidal function, composed with affine functionals, can uniformly approximate any continuous function of n real variables on the unit hypercube. Cybenko proved that in 1989, for the single-hidden-layer case. His abstract claims exactly that much and no more: “Our results settle an open question about representability in the class of single hidden layer neural networks.” The result is still stated in that form in the peer-reviewed literature. Kidger and Lyons open their 2020 paper with the sentence “The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth.” Approximation in principle, at whatever width it takes, is the guarantee. Efficient learning from finite data is not.

The charge is still being paid at the largest scale, on top of compact representations rather than raw pixels. Every encoder and decoder layer of the transformer carries a position-wise fully connected sublayer; Vaswani and colleagues specified it in 2017 with “dmodel = 512, and the inner-layer has dimensionality dff = 2048”. OpenAI's GPT-3 paper keeps the same ratio across all eight of its models: “we always have the feedforward layer four times the size of the bottleneck layer”. That includes the 175-billion-parameter model, 96 layers at dmodel = 12288.

Frameworks often add a length-O bias vector to every row of a B×O matrix, and broadcasting repeats the operation conceptually without storing B copies of the bias.

This is not a neural-network convenience. It is a general array rule with a published definition. NumPy's user guide compares shapes element-wise, starting from the trailing dimension and working left, and states the test in one line: “Two dimensions are compatible when they are equal, or one of them is 1.” Anything else raises a ValueError. The peer-reviewed description of the library, published in Nature in 2020, records the same mechanism. It adds that “one or both arrays are virtually duplicated (that is, without copying any data in memory), so that the shapes of the operands match”.

Read that rule again and notice what it never consults. It compares two integers for equality-or-1. It has no access to what either axis is supposed to mean. A tensor may run happily while adding an offset along an unintended axis.

The wider failure is measured, not hypothetical. One study collected 175 TensorFlow bugs — 87 from Stack Overflow and 88 from GitHub — and classified 24 of them, 15 and 9 respectively, as “Unaligned Tensor” faults. Zhang and colleagues gave the definition in 2018: “A bug spotted in computation graph construction phase when the shape of the input tensor does not match what it is expected is called an unaligned tensor bug.” A separate group built SFData in 2021, “a set of 146 buggy programs with crashing tensor shape faults”, and opens by stating that among DL library misuses “tensor shape faults are most prevalent”.

Perform this check before running code. It has a measured reason to exist, not a stylistic one. Of the 175 catalogued TensorFlow bugs above, 24 were unaligned tensors, and the broadcasting rule will not catch the ones that happen to be compatible.

Two of the five steps carry the specifics this lesson has already fixed. Step 4 counts parameters, and it is also where the widths are checked for divisibility by 4, 8 or 16 if the multiply is to reach Tensor Cores. Step 5 is the framework convention, and it has two documented answers: Keras 3 stores the kernel as (input_dim, units), PyTorch stores weight as (out_features, in_features).