Skip to content
AI.info

Deep architectures

Locality, Weight Sharing, and Equivariance

Explain the architectural value and limitations of local connectivity, shared parameters, receptive fields, invariance, and equivariance.

By the end you can

Example

A shifted pattern exposes the hidden assumption

A convolutional layer is supposed to ask the same question everywhere in the image. It does not. Kayhan and van Gemert tested it in 2020 and opened their abstract with the answer: “We show that CNNs can and will exploit the absolute spatial location by learning filters that respond exclusively to particular absolute locations by exploiting image boundary effects.”

The effect is not confined to a rim of pixels. Modern receptive fields are large, so filters working far from the border still feel the boundary. They rank the variants. Valid convolution, with no padding at all, is the most location-revealing case; zero padding is also sensitive. Their fix is full convolution.

A second group reached the same diagnosis independently that year, and pinned the leak on zero padding specifically. A position probe read out of VGG16 scored .742/.814/.405 Spearman correlation with absolute position. Strip the zero padding out and the same probe fell to .381/.359/.011. Run the experiment forwards and it agrees. Their padding-free PosENet probe scored .012/-.001/-.001. Allow padding=1 and it climbed to .274/.205/.148. At padding=2 it reached .397/.380/.214. Absolute position is not sneaking past the architecture. The architecture is handing it over.

  • A fully connected network can learn separate weights for the same pattern at every location, spending data and parameters on repeated copies of one detector.
  • A convolution reuses one detector across positions, making translated evidence easier to recognize.
  • Pooling can increase tolerance to small shifts, but may erase the exact location a downstream consumer — a repair robot, a segmentation head — still needs.
  • Padding and boundaries break perfect translation symmetry, and the break is measurable: removing zero padding dropped a VGG16 position probe from .742/.814/.405 to .381/.359/.011.
  • Camera perspective, rotation, and scale require additional design or augmentation beyond translation sharing.

Four ideas that should not be collapsed into one

Locality restricts immediate interactions. Weight sharing reuses the same transformation at many positions. Equivariance means the representation changes predictably when the input transforms, while invariance means the output remains unchanged.

A convolution is translation equivariant under idealized conditions. A classification head may seek approximate translation invariance. A segmentation output must move with the object, and therefore stay equivariant.

Collapsing the last two carries a price. Cohen and Welling put a number on it in 2016. They built a variant, P4CNNRotationPooling, that follows every p4-convolution with a coset max-pooling over rotations. Its intermediate features are already rotation-invariant rather than rotation-equivariant. The motivation was “the hypothesis that premature invariance is undesirable in a deep architecture”. On rotated MNIST it scored 3.21% +/- 0.0012 test error. The P4CNN that stays equivariant until the final layer scored 2.28% +/- 0.0004. Their verdict: “This network (P4CNNRotationPooling) outperforms the baseline and the previous state of the art, but performs significantly worse than the P4CNN which does not pool over rotations in intermediate layers.”

Nothing was traded for it. An independent tabulation lists both rows at 3.21 and 2.28, each at 25k parameters. Discarding the transformation early rather than carrying it along costs a 41% relative increase in error, and saves nothing.

Equivariance preserves how a transformation acts; invariance deliberately removes sensitivity to it — and removing it too early took test error from 2.28% to 3.21% at equal parameter count.

Visual

How receptive fields grow through depth

A unit may depend on a larger input region after several local layers. How fast that region grows is a design choice, and there is an exact formula behind it. Yu and Koltun wrote it down in 2016. Their abstract states the principle: “The architecture is based on the fact that dilated convolutions support exponential expansion of the receptive field without loss of resolution or coverage.”

The derivation gives the law. Stack 3x3 convolutions with dilation 1, 2, 4, 8, and so on. The receptive field of an element in layer i+1 is then exactly (2^(i+2) - 1) x (2^(i+2) - 1). That is exponential in depth, while the parameter count per layer stays fixed. It is the third step of the diagram made precise. Coverage expands faster because the sampling pattern, not the kernel size, is doing the work.

The growth curve came attached to a benchmark. Their Dilation10 model scored 67.1% mean class IoU and 86.5% mean category IoU on the Cityscapes test set. An unrelated group, the PSPNet authors, later tabulated the same two figures as the dilated-convolution reference point.

FigureProcess · 4 steps
  1. 1

    Single local operation

    A unit sees only a small neighborhood.

  2. 2

    Stacked local operations

    Each layer combines neighborhoods from the previous layer.

  3. 3

    Downsampling or dilation

    Coverage expands faster, but resolution or sampling pattern changes.

  4. 4

    Global aggregation

    Pooling or attention can connect the final decision to the whole input.

Comparison

Architecture choices for transformed inputs

Different tasks demand different responses to the same physical transformation, and the choice is worth real accuracy in both directions.

Take rotation as a symmetry worth encoding. Cohen and Welling extended weight sharing from translations to a larger group: “G-CNNs use G-convolutions, a new type of layer that enjoys a substantially higher degree of weight sharing than regular convolution layers.” On rotated MNIST — 10,000 training, 2,000 validation and 50,000 test images — their P4CNN reached 2.28% +/- 0.0004 test error. The matched plain CNN scored 5.03% +/- 0.0020. The previous state of the art was 3.98%. The parameter budget barely moved: an independent tabulation puts the plain CNN at 22k parameters and the P4CNN at 25k. Where the transformation preserves the label, encoding it more than halved the error at the same cost.

Now take a transformation that changes what the image means. Medical imaging does not treat left and right as a nuisance symmetry. It writes laterality into the file format. DICOM defines Image Laterality (0020,0062) in the General Image Module as the “Laterality of (possibly paired) body part (as described in Anatomic Region Sequence (0008,2218)) examined.” Its Enumerated Values are R for right, L for left, U for unpaired and B for both left and right, and it must stay consistent with Laterality (0020,0060) where that is present.

A separate standards body makes it operational. The IHE International Mammography Image Profile requires the acquisition modality to supply Patient Orientation (0020,0020), Image Laterality (0020,0062) and the View Code Sequence (0054,0220). Review workstations from different vendors then hang left and right breasts in the same places. In that domain a horizontal-flip augmentation is not free data. It contradicts the metadata the whole workflow is built on.

FigureComparison · 3 columns

Invariant output

The prediction should remain stable after a permitted transformation.

  • Object category after small translation
  • Speaker identity across time offset
  • Document topic after paragraph reordering only when order is irrelevant
  • Requires careful definition of permitted transformations

Equivariant output

The output should transform in correspondence with the input.

  • Segmentation mask follows an object
  • Keypoints move with the image
  • Forecast shifts with the time origin
  • Useful when location or order must be preserved

Sensitive output

The transformation itself changes the meaning.

  • Left and right medical anatomy
  • Temporal order in causal events
  • Absolute position on a board
  • Sharing can be harmful when coordinates carry semantics

Analogy

A stencil reused across a surface

An inspector checks a large sheet with one small transparent stencil. The same pattern slides over every region, and no new stencil is manufactured for each coordinate.

A stencil carries one fixed pattern, while learned kernels produce many continuous channels and interact across depth. Parameter sharing still reduces the number of separately learned detectors.

That efficiency has an exchange rate, and the Vision Transformer paper measured it by throwing the stencil away. Its introduction states the handicap: “Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data.” Trained on ImageNet alone, ViT lands a few points below comparably sized ResNets. Pre-trained on the in-house JFT-300M set — 303M images, 18k classes — ViT-H/14 reaches 88.55% top-1 on ImageNet. The same paragraph closes: “We find that large scale training trumps inductive bias.”

What sharing buys, then, is sample efficiency rather than a ceiling. Even the sample-efficiency gap is closable. Touvron and colleagues reported 83.1% top-1 single-crop from ImageNet-only training, with an 86M-parameter model, on one computer, in under three days. They called it DeiT.

Shared local filters convert repeated structure into statistical efficiency — an advantage denominated in training images, not in accuracy ceilings.

Key idea

The effective receptive field may be smaller than the theoretical one

A theoretical receptive field lists every input that can influence a unit through the graph. Some of those inputs move the unit far more than others.

Optimization, nonlinearities, and parameter values often concentrate sensitivity near a subset of positions. Declaring a large theoretical field does not prove that the model uses distant evidence effectively.

Connectivity permits influence; trained sensitivity determines whether that influence matters.

Case

Effective receptive fields grow as the square root of depth

That gap was measured in 2016, and the measurement gave it a name. The effective receptive field, Luo and colleagues report, “both has a Gaussian distribution and only occupies a fraction of the full theoretical receptive field size”. A Gaussian decays quickly from its centre. So does the influence.

Worse, the two fields do not grow together. Stacking n convolutional layers expands the effective field as O(√n), while “the theoretical receptive field grows linearly, therefore relative to the theoretical receptive field, the ERF actually shrinks at a rate of O(1/√n)”. Their measured best-fit slopes in the log domain were 0.56 for effective size and −0.43 for the ratio, against the 0.5 and −0.5 that the derivation alone would give.

Carry those slopes out to twenty-five stacked layers. The theoretical field is 25 times the size of one layer's. The effective field is 6.07 times. The ratio is 25.1 per cent. That last number is the share of the nominal field the network actually uses.

Figure

Stacking layers widens what a unit could see much faster than what it does see. The slopes are the best fits reported by Luo, Li, Urtasun and Zemel; theory alone would give 0.5 and −0.5.

Steps

Audit a symmetry assumption before encoding it

Architectural bias is useful only when the transformation preserves the task meaning. Both ends of that audit are already on this page. The same rotation is worth the difference between 5.03% and 2.28% test error on rotated MNIST, where orientation carries no label information. Laterality in DICOM is the opposite case. The standard requires it to be recorded and kept consistent, because flipping the image changes which body part is being described. One transformation, two verdicts, decided by the task rather than by the architecture. Name the transformation, decide what the output owes it, check the boundaries, and test it rather than assuming augmentation settled the question.

FigureProcess · 4 steps

  1. 1. Define the transformation

    Name translation, rotation, permutation, scaling, or another candidate symmetry.

  2. 2. Check target behavior

    Decide whether the output should stay fixed, move predictably, or change meaning.

  3. 3. Inspect boundaries

    Account for padding, missing context, coordinate systems, and finite domains.

  4. 4. Test transformed slices

    Evaluate controlled transformations rather than assuming augmentation solved them.

Coordinates can restore information that sharing removes

Shared operations intentionally ignore some absolute-position distinctions. When the task depends on position, designers may add coordinate channels, positional embeddings, boundary indicators, or geometry-aware features.

This is not a failure of convolution. It is an explicit correction to an inductive bias that was useful but incomplete.

Part of the correction also arrives uninvited. Kayhan and van Gemert showed filters learning to respond exclusively to particular absolute locations through image boundary effects. The second group showed the channel it travels down. Their padding-free PosENet position probe scored .012/-.001/-.001, rose to .274/.205/.148 at padding=1, and reached .397/.380/.214 at padding=2. A designer who wants position should add it deliberately, and know how much is already leaking in.

Good architecture design states both the symmetry it encodes and the exceptions it must recover.

Key idea

Do not demand universal invariance

A system that is invariant to every transformation cannot distinguish cases that differ only by those transformations. Excess invariance is information loss.

Choose the smallest symmetry group justified by the task, then measure performance on transformations inside and outside that group.

That measuring has been done. Azulay and Weiss reported in the Journal of Machine Learning Research in 2019 that “small translations or rescalings of the input image can drastically change the network’s prediction”. Their diagnosis has two halves: “the convolutional architecture does not give invariance since architectures ignore the classical sampling theorem, and data augmentation does not give invariance because the CNNs learn to be invariant to transformations only for images that are very similar to typical images from the training set”. They test two repairs, antialiasing the intermediate representations and increasing data augmentation. Both “provide only a partial solution at best”.

Partial has a size. “Modern convolutional networks are not shift-invariant, as small input shifts or translations can cause drastic changes in the output.” That is Richard Zhang, opening his 2019 abstract. Standard downsampling destroys shift-equivariance inside the network. Inserting a low-pass filter before subsampling buys some of it back.

His paper gives the ledger. Progressively stronger Rect-2, Tri-3 and Bin-5 filters raised ResNet50's ImageNet shift-consistency by +0.8%, +1.7% and +2.1%, and top-1 accuracy by +0.7% to +0.9%. Doubling the depth to ResNet101 bought only +0.6% stability and +1.2% accuracy. An independent group re-ran the method under their own identical training schedule and reproduced the effect: a ResNet-101 baseline at 77.7% top-1 and 90.6 consistency, against 78.4% (+0.7) and 91.6 (+1.0) with a 3x3 low-pass filter. Partial means roughly two points of shift-consistency. It does not mean shift-invariant.

Invariance is a contract about which distinctions the model is allowed to discard.

Key takeaways