Parameter Sharing and Inductive Bias

Finite data cannot determine behavior for every possible input. A model must prefer some extensions of the observed examples over others.

Neural architecture expresses such preferences through connectivity, shared parameters, normalization, positional information, and allowed information paths.

A preference of this kind can be written down and priced. A convolution normally shares one kernel across translations. In 2016 Cohen and Welling shared it across more: the p4 group of translations and 90-degree rotations. They tested on rotated MNIST, halving the filter count “so as to keep the number of parameters approximately fixed”. The same seven-layer network fell from 5.03% test error to 2.28%. The published state of the art was 3.98%. The assumption cost nothing in parameters and removed more than half the errors. It was true of the data.

Architecture is not the only place a preference can come from, and it is not always the largest. Hold ResNet-50 fixed, change only the data pipeline, and its shape bias moves — how often the network answers with an object's shape rather than its texture when the two are put into conflict in the same image. A standard random crop gave 19.5%. A center crop gave 28.4%. Adding colour distortion, Gaussian blur, Gaussian noise, a 64% minimum crop, stronger augmentation and 270 epochs took it to 62.2%: an actively shape-biased model, from the same architecture, at the cost of ImageNet top-1 falling from 76.6% to 71.1%. Hermann and colleagues ran that experiment in 2020, and their conclusion is one sentence: “While both architecture and training objective have an effect on the level of texture bias in a model, the statistics of the training dataset are the most important factor.” Geirhos and colleagues had already flipped the same architecture from 22% to 81% shape bias by retraining it on Stylized-ImageNet. The prior a deployed model actually carries is the joint product of what was built and what it was fed. Only an experiment that varies one at a time can tell you which did the work.

Parameter sharing applies one learned transformation. The same weights act across several positions or timesteps. One parameter set describes a reusable rule; that rule is applied across image patches, tokens or time steps; the resulting activations differ because the local inputs differ; later operations aggregate the responses across the structure.

The reuse does not have to be spatial, and its payoff can be measured. One weight matrix can hold two jobs at once — the input word embedding and the output softmax projection. Press and Wolf tied them together in 2017 and reported both halves of the trade. On Penn Treebank their small no-dropout language model fell from 4.65M parameters to 2.65M. Its test perplexity improved at the same time, from 114.5 to 112.4. Fewer parameters and a better model, from the same bet. Their weight-tied translation models carried about 28% fewer parameters, about 52% fewer with three-way tying, at similar BLEU. As their abstract puts it: “Finally, we show that weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance.”

A second group reached the same conclusion independently, also in 2017. Inan and colleagues reused the embedding and saw Penn Treebank test perplexity improve from 87.3 to 85.1 for a 200-unit model, and from 77.7 to 73.9 for a 650-unit one. The parameter saving is the less interesting half of the result. Sharing is a hypothesis about the world — here, that the vector representing a word on the way in should be the vector that scores it on the way out. When the hypothesis holds, the constraint is not a tax on the model. It is information the model no longer has to learn twice.

Model families differ partly in which interactions they make cheap and repeatable. A dense layer lets every output use every input with independent coefficients: a weak structural assumption, many parameters for wide inputs, global mixing in one step, useful for compact vectors. A convolution reuses one local kernel across positions: a locality prior, translation-related reuse, parameter-efficient spatial processing, and choices to be made about boundaries and scale. A recurrent layer reuses one state-update rule through sequence time: order-sensitive processing, variable-length reuse, a sequential dependency path, and a state that can lose distant information.

Each is a different bet about where structure lives.

An equivariant representation changes predictably when the input is transformed, so a translated image may produce a correspondingly translated feature map.

An invariant output stays the same under a transformation, such as an object label that should not change after a small translation; architectures often build equivariance first and aggregate toward invariance later.

The distinction is load-bearing in shipped systems. AlphaFold is one. Its structure module updates its activations with a block called invariant point attention, described in Nature in 2021: “The IPA augments each of the usual attention queries, keys and values with 3D points that are produced in the local frame of each residue such that the final value is invariant to global rotations and translations”. The block's output does not move when the whole protein is rotated or translated. The update it produces is then applied inside each residue's own local frame, so the module as a whole is equivariant on the residue gas. Invariance where the answer must not change; equivariance where the structure must change with the input.

An outside body measured what that construction bought. At CASP14 the predictions reached a median backbone accuracy of 0.96 Angstrom r.m.s.d.95, against 2.8 Angstrom for the next best method. The CASP organisers' own final ranking by combined z-scores places group 427 (AlphaFold2) at 244.0217 over 92 domains, against 90.8241 for the second-placed BAKER group.

Architectural priors are useful only when they resemble the real task — and a prior can quietly deliver a different capability from the one its designers assumed.

Put texture and shape into direct conflict inside the same image, then ask what answers come back. Nine psychophysical experiments, 48,560 trials, 97 observers. People answer with the object: “Human observers show a striking bias towards responding with the shape category (95.9% of correct decisions).” The ImageNet-trained networks answered with the texture. ResNet-50 gave the shape answer 22.1% of the time against 77.9% texture; VGG-16 17.2% against 82.8%; GoogLeNet 31.2% against 68.8%; AlexNet 42.9% against 57.1%. Geirhos and colleagues published that in 2019. No one specified a texture detector. A locality prior plus a dataset produced one, and it took an experiment designed against the assumption to find out. Hermann and colleagues later measured a standard random-crop ResNet-50 on the same cue-conflict stimuli and obtained a 19.5% shape bias — the same surprise, from an independent group.

Translate domain knowledge into explicit architectural questions. Name the transformations that should preserve or predictably alter the target. Locate the interactions: are the dependencies local, global, sequential, relational, or unordered? Decide what can share, identifying computations repeated across positions, entities, or tasks. Preserve the information sharing would erase, adding position, time, identity, or boundary signals. Then test the broken assumptions, evaluating slices where locality, stationarity, or symmetry does not hold.

The fourth step has a canonical instance. The transformer dropped recurrence and convolution, and with them the architecture's only handle on word order. The 2017 paper that introduced it, Attention Is All You Need, gives the reason for handing that order back: “Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence.” Sinusoidal positional encodings are that injection: an assumption removed from the architecture and handed back as an input feature. Its Table 1 prices what each family makes cheap — the maximum path length between two positions is O(1) for self-attention, O(n) for a recurrent layer and O(log_k(n)) for a convolutional one.

The fifth step is what stops the fourth from becoming folklore. Haviv and colleagues removed the injection in 2022 and measured. A 1.3B-parameter causal language model trained with no positional encoding at all reaches 13.10 validation perplexity on the Pile, against 13.05 with learned positional embeddings — a gap of 0.05. Bidirectional masked language models, whose attention carries no causal mask, fail to converge without positional information. The order the fourth step supplies by hand was already reaching the causal model through its mask. The same fix was near-redundant in one setting and indispensable in the other. Only the slice test tells them apart.