Vision Transformers and Hierarchical Visual Attention

How tokens are built determines what detail is available before attention begins.

Two papers fix the numbers behind the first two bullets. Sixteen pixels a side was the original choice: cut the image into 16x16 patches, flatten them, feed the sequence to a plain transformer. Dosovitskiy and eleven co-authors showed in 2021 that this is enough. Their finding was that “a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks”, provided it is “pre-trained on large amounts of data”. Swin Transformer went four times finer, the same year. It uses “a patch size of 4 × 4 and thus the feature dimension of each patch is 4 × 4 × 3 = 48”. Attention is then confined to windows whose size “is set to M = 7 by default”. Sixteen pixels a side or four is not a detail for the engineer to settle later. It is the scale at which the model performs its first irreversible averaging.

The third bullet has a measured price. Replace ViT's default stride-16 16x16 patchify stem with a small stack of stride-two 3x3 convolutions. Drop one transformer block so that flops and runtime stay constant. The model then gets better on both axes at once: “Using a convolutional stem in ViT dramatically increases optimization stability and also improves peak performance (by ∼1-2% top-1 accuracy on ImageNet-1k), while maintaining flops and runtime.” Xiao and five co-authors reported that in 2021.

The stability half carries the larger number. For the patchify-stem baseline the AdamW-versus-SGD accuracy gap reaches up to 10 points, and training sometimes diverges outright. For the convolutional-stem models at large sizes and long schedules it falls below 0.2 points, and sensitivity to learning rate and weight decay moves 'closer to that of modern CNNs'. The gain is not a special case. It holds from 1G to 36G flops and from ImageNet-1k to ImageNet-21k. Note what it is not. The fragility is sharply reduced, not eliminated.

A Vision Transformer splits an image into patches, projects each patch to an embedding, adds positional information, and processes the sequence with Transformer encoder blocks.

A classification token or pooled representation feeds the output head. The architecture offers global interaction early. What it lacks is the strong locality and translation bias built into convolution.

The choice affects feature resolution. It also affects detection interfaces and computational shape.

Two of these four options have a published exchange rate rather than a hedge. Stack depthwise convolution stages under attention stages and you get CoAtNet: “Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy; When pre-trained with 13M images from ImageNet-21K, our CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data”. Dai and three co-authors reported that in 2021. Scaled further with JFT-3B, the same family reaches 90.88%. So the hybrid column is not a vague suitability for moderate data. Thirteen million images bought what 300 million bought a pure ViT. That is a factor of 23.

Token merging has its own rate. Combine similar tokens gradually inside an already-trained ViT, with no retraining at all, and the throughput doubles for almost nothing: “Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.2x the throughput of ViT-L on video with only a 0.2-0.3% accuracy drop in each case.” Bolya and five co-authors reported that in 2023. Two times the throughput for two or three tenths of a point, on a named model at a named resolution. That is the sentence the trade-off column should be read against.

Resolution falls. Channel capacity and semantic abstraction usually rise.

The last stage is where the pyramid is paid for. Put a convolution-free hierarchical backbone with a progressive shrinking pyramid under an off-the-shelf detector and the score moves: “For example, with a comparable number of parameters, PVT+RetinaNet achieves 40.4 AP on the COCO dataset, surpassing ResNet50+RetinNet (36.3 AP) by 4.1 absolute AP.” That is the Pyramid Vision Transformer, from Wang and eight co-authors in 2021. The spelling of RetinNet is the published abstract's own.

The detector and the parameter budget were held comparable. Only the backbone changed. Exposing several scales to a dense-prediction head was worth 4.1 absolute AP. That is a number, not an argument that hierarchy ought to help detection.

The decisive feature may be a tiny lesion, a whole-scene relation, or both.

Swin Transformer is the worked example of steps two through four. Its shifted-window scheme “brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection”. That gives the architecture “linear computational complexity with respect to image size”. Its stagewise merging exposes the several scales that dense prediction heads consume. The reported numbers cover all three head types. ImageNet-1K top-1 accuracy is 87.3. COCO test-dev reaches 58.7 box AP and 51.1 mask AP. ADE20K val reaches 53.5 mIoU. That surpasses the previous state of the art “by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K”.

Step five is where the earlier numbers return. A stem choice worth 1-2 top-1 points. A pyramid worth 4.1 absolute AP on COCO. A merging schedule worth 2x throughput for 0.2-0.3 points. Each is small enough to be lost in an unmatched comparison. Each is large enough to decide the system.

A high attention weight is one routing coefficient inside a multilayer computation. Residual paths, value vectors, later blocks, and the task head all affect the final output.

Raw attention is the weakest of the attention-based explanation methods, and there is a benchmark that says so. Scored against the ImageNet-Segmentation ground truth, it reaches 67.84% pixel accuracy, 80.24 mAP and 46.37 mIoU. Attention rollout reaches 73.54 / 84.76 / 55.42. The relevance-propagation method proposed by Chefer and two co-authors in 2021 reaches 79.70 / 86.03 / 61.95. Twelve points of pixel accuracy separate the prettiest heatmap from the method that follows the value paths and skip connections it ignores.

The sharper fact is procedural. Two of the methods could not be entered in the target-class perturbation test at all: “Notice that because rollout and raw attention produce constant visualization given an input image, we omit their scores in the target-class test.” A map that comes out the same whichever class you query it about cannot be an explanation of any particular class.

Use counterfactual masking, feature interventions, localization metrics, and failure slices alongside attention visualization. A compelling heatmap is not sufficient evidence of causal reliance.