Skip to content
AI.info

Computer vision

Vision Transformers in Practice

Study Vision Transformer components, data requirements, hierarchical variants, dense prediction, and evidence-based architecture comparison.

By the end you can

Analogy

A planning committee for image regions

A city map is divided into districts, and each district exchanges summaries with every other district. Long-range coordination becomes direct. The meeting also grows expensive as districts multiply.

Districts deliberate; token interactions are learned numerical operations. What the meeting captures is global communication and quadratic pair growth. The price of the meeting is not a metaphor in this lesson. The same 86-million-parameter model costs 8.6 GFLOPs when it is cut into large districts and 156 GFLOPs when it is cut into small ones. The next section gives the accuracy that buys.

Token count, not merely image width, drives the cost of global attention.

Visual

From pixels to class token

A basic Vision Transformer converts a grid into a sequence. Then it mixes information, over and over.

Google Research announced the design in a post written by two of the paper's authors, and that post puts the whole pipeline in one sentence: “ViT represents an input image as a sequence of image patches, similar to the sequence of word embeddings used when applying Transformers to text, and directly predicts class labels for the image”. The same post records the efficiency claim that made the design interesting — “outperforming a comparable state-of-the-art CNN with four times fewer computational resources”. Every step below is one clause of that sentence made mechanical.

FigureProcess · 5 steps

  1. 1. Patchify the image

    Split the image into fixed regions or produce patch-like features.

  2. 2. Project patches

    Map each region into a shared embedding dimension.

  3. 3. Add position

    Encode where tokens originated so permutation is not mistaken for geometry.

  4. 4. Apply transformer blocks

    Alternate attention, feed-forward transformations, normalization, and residual paths.

  5. 5. Produce task outputs

    Use a class token, pooled features, or dense decoder connections.

Patch size is an information and compute decision: 79.1% at 8.6 GFLOPs, 85.6% at 156

Larger patches shorten the sequence and cut the cost of attention. But a small object, or a boundary, can disappear inside a single token. Smaller patches keep that detail and cost more memory and more compute.

The trade has been priced. Google Research's FlexiViT paper opens its introduction with the comparison: “For example, a ViT-B/8 model achieves 85.6% top-1 accuracy on ImageNet1k with 156 GFLOPs and 85 M parameters, while a ViT-B/32 model achieves only 79.1% accuracy with 8.6 GFLOPs and 87 M parameters. Despite the major difference in performance and compute, these models have essentially the same parametrization.”

Read the three numbers together. The parameter count barely moves — 85 M against 87 M — so a model card that lists only parameters cannot tell these two apart. Accuracy moves 6.5 points. Compute moves by roughly 18x. The entire difference is where the crop lines were drawn, before the first projection.

So choose patch scale against the smallest evidence the task must resolve. Report the resizing and crop policy, because they change the effective object scale. And report the patch size next to the parameter count. On its own, the parameter count hides an 18x decision.

85 M parameters and 87 M parameters, 6.5 points of ImageNet top-1 apart: a 16-pixel patch is not a neutral preprocessing choice.

Comparison

Global, windowed, and hierarchical attention

Visual transformers trade communication range, inductive bias, and efficiency differently. Each end of that range has been measured rather than asserted.

The global end came first, at ICLR 2021. Dosovitskiy and colleagues reported that “a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks”, and that “when pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train”.

What that global access looks like inside the stack was measured later. Raghu and colleagues computed attention-weighted pixel distance, averaged over 5,000 datapoints, across ResNet50x1, ResNet152x2, ViT-B/32, ViT-B/16, ViT-L/16 and ViT-H/14, all of them pretrained on JFT-300M. They found this: “In agreement with Dosovitskiy et al. [14], we observe that even in the lowest layers of ViT, self-attention layers have a mix of local heads (small distances) and global heads (large distances). This is in contrast to CNNs, which are hardcoded to attend only locally in the lower layers. At higher layers, all self-attention heads are global.” So the first column below is not merely long-range. It is mixed local and global at layer one, and entirely global at the top. That is a receptive-field profile the convolution cannot express and the transformer has to learn.

And learning it is conditional. The same authors found that when pretraining is restricted to ImageNet rather than large-scale data, the early layers fail to learn the local attention that remains vital. Mixed early heads are something the pretraining budget buys. They are not a property the block owns.

The windowed line answers a different question. Swin Transformer computes its representation “with Shifted windows”, “limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection”, so that it “has linear computational complexity with respect to image size”. It reports “87.3 top-1 accuracy on ImageNet-1K”, “58.7 box AP and 51.1 mask AP on COCO test-dev” and “53.5 mIoU on ADE20K val”. Hold on to the 87.3. Two sections from here, a pure convolutional network passes it.

FigureComparison · 3 columns

Global attention

Every token can directly attend to every other token.

  • Strong long-range access
  • Quadratic token interaction
  • Simple sequence design
  • Example: image classification

Windowed attention

Tokens interact primarily inside local windows.

  • Lower cost
  • Needs cross-window communication
  • Restores locality bias
  • Example: high-resolution vision

Hierarchical transformer

Spatial resolution decreases while feature depth grows.

  • Multi-scale features
  • Compatible with dense heads
  • More architectural choices
  • Example: detection and segmentation

Key idea

Architecture comparisons inherit pretraining budgets: 2.7 points from the recipe alone

Transformers often benefit strongly from large-scale pretraining, augmentation, distillation, and regularization. A result may reflect the training recipe as much as the block type. That sentence has a number attached to it.

Take a ResNet-50, change nothing about the architecture, apply a modern training recipe, and it gains 2.7 points. The ConvNeXt authors at Facebook AI Research and UC Berkeley did exactly that, in a section called “Training Techniques”: “By itself, this enhanced training recipe increased the performance of the ResNet-50 model from 76.1% [1] to 78.8% (+2.7%), implying that a significant portion of the performance difference between traditional ConvNets and vision Transformers may be due to the training techniques.” Any CNN-versus-transformer gap narrower than 2.7 points, measured against a ResNet-50 trained the old way, is a gap about recipes.

The other side of the budget has a ceiling worth knowing. Four researchers at Google Brain scaled a ViT as far as it would go: “As a result, we successfully train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90.45% top-1 accuracy. The model also performs well for few-shot transfer, for example, reaching 84.86% top-1 accuracy on ImageNet with only 10 examples per class.” Two billion parameters is what the top of the transformer curve costs. And 84.86% from ten labelled examples per class is what that budget carries into transfer. A comparison in which only one side has been anywhere near that budget is a comparison of budgets.

Compare matched compute, data, resolution, and tuning effort. Include a strong convolutional baseline, rather than setting a modern recipe against an outdated one.

2.7 points moved with the ResNet-50 architecture untouched: a fair experiment changes one design choice at a time.

Case

The recipe moved, not the block: 83.1% on ImageNet alone

The pretraining-budget argument has a clean example, and it predates the ConvNeXt result above by a year. DeiT set out to train “a competitive convolution-free transformer by training on Imagenet only”, and did it “on a single computer in less than 3 days”.

The headline figure is the one to hold: “Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data”. The models it was competing with had been pre-trained on hundreds of millions of images. Then a distillation token, learning from a convolutional teacher through attention, carried the same architecture to “85.2%”.

Nothing in the block changed between the transformers that needed hundreds of millions of external images and the one that needed a single computer for three days. The data budget changed. The recipe changed. Note the parameter count too: 86M, the same ViT-B scale whose patch size alone moves accuracy by 6.5 points. Two levers outside the architecture, and both of them larger than most architecture claims.

Example

What dense tasks demand beyond a classification token

Detection and segmentation need spatially resolved features, and the payoff for supplying them has been measured. A team at Intel Labs assembled tokens from several transformer stages into image-like representations at multiple resolutions, and reported: “For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network.” On segmentation the same design reached 49.02% mIoU on ADE20K. The bullets below are not housekeeping. They are where that 28% comes from.

  • Feature hierarchy: multiple resolutions help represent objects of different sizes — the Intel Labs design built its own by assembling tokens from several transformer stages into image-like representations at multiple resolutions, and reached 49.02% mIoU on ADE20K.
  • Decoder or neck: tokens must become boxes, masks, or dense maps, and that conversion is where the measured gain sits — up to 28% relative on monocular depth, against a state-of-the-art fully-convolutional network.
  • Positional interpolation: new resolutions require careful handling of learned positions.
  • Boundary detail: coarse patching blurs thin structures, and the patch decision costs 6.5 points of top-1 before any decoder is attached.
  • Memory planning: high-resolution sequences can dominate accelerator memory; 156 GFLOPs against 8.6 for the same parameter count is the shape of that risk.
  • Augmentation alignment: geometric transforms must stay synchronized with labels.

Steps

Run a defensible CNN-versus-transformer study

The study should isolate architecture from recipe and infrastructure. The strong convolutional baseline that the second step demands already exists in the literature.

A ConvNet for the 2020s closes its abstract with the result that ends most architecture arguments: “Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.” The official code and model release gives the configuration behind that number: ConvNeXt-XL at 384x384, 350M parameters, 179.0 GFLOPs, ImageNet-22K pretrained. That is 87.8% against the 87.3% Swin reports two sections above — the pure convolutional family, in front, on the transformer's own benchmark.

So when the steps below say match the budget, they mean a specific pair of numbers: 350M parameters and 179.0 GFLOPs at 384x384, with the same ImageNet-22K pretraining on both sides. A transformer that beats an untuned ResNet-50 at 76.1% has beaten a training recipe from another decade, not a convolution.

FigureProcess · 5 steps

  1. 1. Match the evidence

    Use identical train, validation, and test data with the same leakage controls.

  2. 2. Match the budget

    Constrain parameters, compute, training time, and tuning trials.

  3. 3. Tune each fairly

    Use architecture-appropriate learning rates and regularization without unlimited search.

  4. 4. Test slices

    Compare small objects, textures, occlusion, shift, and limited-data regimes.

  5. 5. Measure deployment

    Record latency, memory, batch behavior, and target-hardware support.

Key idea

Attention maps are useful probes, not complete explanations

An attention visualization shows one internal interaction pattern under a chosen aggregation. Residual paths, feed-forward blocks, heads, layers, and downstream decoders also shape the output. The measurement in the comparison section makes that concrete. Heads at the same depth disagree — some local, some global. A single averaged map is already a summary of disagreeing evidence.

Use attention maps to form hypotheses. Then test them with ablation, occlusion, counterfactuals, or task-specific perturbations.

The field has built alternatives. Chefer and colleagues observe that “existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph”. Their method instead “assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers”, a propagation that “involves attention layers and skip connections, which challenge existing methods” — the two components a raw attention map silently omits. Benchmarked on recent visual transformers, they report “a clear advantage over the existing explainability methods”. A visualization is a hypothesis.

Visibility of an attention weight does not establish causal importance.

Example

Practice: select a backbone for pathology tiles

A team must classify gigapixel-slide tiles while detecting rare, small cellular patterns. This exercise has a published precedent to argue against, rather than a hypothetical to imagine.

UNI is a ViT-Large/16 pretrained with DINOv2, and its abstract in Nature Medicine states the budget plainly: “We introduce UNI, a general-purpose self-supervised model for pathology, pretrained using more than 100 million images from over 100,000 diagnostic H&E-stained WSIs (>77 TB of data) across 20 major tissue types.” It was then evaluated on 34 computational-pathology tasks.

An independent group describes the same model from the outside: “a ViT-large model trained on 100,000 proprietary slides using the DINOv2 SSL algorithm”, with a pretraining set of “100 million tiles from 20 major tissue types”. They benchmarked it against other public pathology encoders on clinical slides from two medical centres.

Answer each bullet against that precedent. Say what UNI chose. Then say what you would choose differently, and what evidence would justify it.

  • Choose a patch scale and explain the smallest signal it can preserve — UNI used /16 patches over more than 100 million pathology tiles, and the FlexiViT figures say that moving from /32 to /8 buys 6.5 points of top-1 for roughly 18x the compute.
  • Decide whether global or hierarchical attention is feasible at tile resolution, given that 156 GFLOPs against 8.6 is the range a single ViT-B spans on patch size alone.
  • Specify a convolutional baseline with a comparable training budget. ConvNeXt-XL at 384x384 — 350M parameters, 179.0 GFLOPs, 87.8% ImageNet top-1 — has already beaten Swin Transformers on COCO detection and ADE20K segmentation.
  • Add a slice for scanner and staining variation. That is why an independent group re-benchmarked UNI on clinical slides from two medical centres, rather than trusting the original evaluation surface.
  • Measure both slide-level utility and tile-level localization behavior, across a task set wide enough to expose failures. UNI reports 34 computational-pathology tasks, and a two-task evaluation is not evidence of a general backbone.

Key takeaways