Tensors, Axes, Shapes, and Batches

These terms describe different aspects of a tensor, and only some of them are written down anywhere.

An array, as the NumPy paper in Nature put it in 2020, is a pointer to memory plus metadata: data type, shape and strides. One sentence there fixes the first two rows of this hierarchy: “The shape of an array determines the number of elements along each axis, and the number of axes is the dimensionality of the array.”

A versioned standard draws the same line independently. The Python array API standard 2025.12 specifies array.ndim as “Number of array dimensions (axes).” alongside array.shape, and NumPy's own reference page for numpy.ndarray.shape reads “Tuple of array dimensions.” Rank is recorded. Shape is recorded. The third row of the hierarchy is recorded by neither: no attribute in the standard or in the implementation states which dimension holds the batch. Axis semantics live only in the head of whoever wrote the code, or in the ledger they keep.

Conventions differ by domain, and for the same four numbers they differ between shipped APIs. tf.nn.conv2d takes data_format 'NHWC' as its default. torch.nn.Conv2d specifies its input as (N, C_in, H_in, W_in). The ONNX operator specification fixes Conv's input X as “Input data tensor from previous layer; has size (N x C x H x W), where N is the batch size, C is the number of channels, and H and W are the height and width.”

Three specifications, one convolution, and no agreement on the order. ONNX alone adds an optional dimension denotation of [DATA_BATCH, DATA_CHANNEL, DATA_FEATURE, DATA_FEATURE …]. It is the only one of the three that names an axis rather than fixing its position. That is why the labels have to travel with the numbers. A four-element shape moved between two of these APIs is still a legal four-element shape.

Similar-looking tensor operations can have very different semantic effects. One of the four comes with a published algorithm that says precisely how it will go wrong.

Broadcasting aligns from the right. The Python array API standard 2025.12 states the rule: “If two arrays are of unequal rank, the array having a lower rank is promoted to a higher rank by (virtually) prepending singleton dimensions until the number of dimensions matches that of the array having a higher rank.” Both major implementations write down the same rule. NumPy's manual says it “starts with the trailing (i.e. rightmost) dimension and works its way left”, treating two dimensions as compatible when they are equal or one of them is 1. PyTorch's broadcasting semantics iterate “starting at the trailing dimension”.

Nothing in that algorithm consults what an axis means. A bias vector intended for the feature axis is accepted against whatever axis its trailing length happens to match, silently. What comes back is a well-formed tensor of exactly the expected shape. Reshape, transpose and reduce fail the same way for the same reason. Each is defined over lengths, and lengths are all the framework stores.

Variable-length sequences are padded to a common length. The mask that accompanies them is a real tensor with a shape of its own, not a matter of good practice. Hugging Face's Transformers glossary works the example: two BERT-tokenized sentences of length 8 and 19, padded with zeros to a common 19, with the attention_mask written out as eight 1s followed by eleven 0s. “The attention mask is a binary tensor indicating the position of the padded indices so that the model does not attend to them,” says the glossary entry for attention mask.

The object is specified in the frameworks too. PyTorch's nn.MultiheadAttention takes key_padding_mask of shape (N, S), “indicating which elements within key to ignore for the purpose of attention (i.e. treat as "padding")”. The mechanism goes back to Attention Is All You Need in 2017, which records the implementation in a single line: “We implement this inside of scaled dot-product attention by masking out (setting to −∞) all values in the input of the softmax which correspond to illegal connections.”

Drop the mask and the eleven padded positions of that worked example are ordinary evidence. They receive finite attention weight instead of none. They enter pooling, losses and normalization alongside the eight real tokens. The shape is still correct throughout.

Maintain a small ledger during design and debugging. It exists because of what the previous sections established. The framework already records rank and shape for you, in array.ndim and array.shape, and records nothing at all about what any axis means. The ledger is where the missing third row of the vocabulary is kept. It is what a mask check or a row-index check is checked against.

Assert expected rank, axis lengths, finite values, and allowed ranges at module boundaries. During early development, also inspect one example before and after every major transformation. Tests should include batch size one, empty optional fields, maximum sequence length, and non-contiguous tensor layouts when the framework permits them.

The category has been counted rather than merely complained about. A 2018 empirical study of TensorFlow program bugs classified 175 of them — 87 from Stack Overflow and 88 from GitHub — into seven root-cause categories. Twenty-four of the 175 were “Unaligned Tensor”: 15 from Stack Overflow, 9 from GitHub, 13.7% of the whole corpus. The definition is narrow and mechanical: “A bug spotted in computation graph construction phase when the shape of the input tensor does not match what it is expected is called an unaligned tensor bug.” Verma and Su re-used that same benchmark in 2020 for a tool called ShapeFlow. They call shape incompatibility “one of the most common bugs in deep learning code”, and on 52 such programs they report no false positives, a single false negative, and average speed-ups of 499x and 24x over running TensorFlow itself. Checking shapes is cheaper than running the graph, by roughly those factors.

Axis order is not only a readability question. It is measured in throughput. NVIDIA's Convolutional Layers User's Guide is explicit about the layout. It states that “convolutions implemented for Tensor Cores require NHWC layout and are fastest when input tensors are laid out in NHWC”, and adds that “NCHW layouts can still be operated on by Tensor Cores, but include some overhead due to automatic transpose operations”. PyTorch's own channels-last tutorial reports the other side of the same trade: over 22% faster training under automatic mixed precision, and gains of 8%–35% on Volta devices. That comes from re-ordering the same numbers in memory and changing nothing else.