Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression

AI summary

Overview

Research area: Efficient computer vision — specifically, training-free token reduction for Vision Transformers (ViTs) and Data-efficient Image Transformers (DeiT), plus robustness evaluation of compressed models.

Technical level: Intermediate. Readers need basic familiarity with ViT patch tokenization, self-attention, the [CLS] token, and prior methods such as EViT and ToMe, but the paper is written in a readable, method-then-experiment structure.

Scope: The paper proposes a dual-stage, plug-and-play framework that combines class-attention pruning with similarity-based merging for off-the-shelf pretrained ViT/DeiT models, and evaluates it on ImageNet-100 under semantics-preserving transformations using standard efficiency metrics plus two new robustness metrics (RAD and PFR).

Venue and provenance (as reported): arXiv:2610.18039v2 [cs.CV], 14 Oct 2026, submitted to the GlobalSouthAI @ NeurIPS 2026 workshop; licensed CC BY 4.0. Authors are affiliated with the Department of Mathematics, Indian Institute of Technology Roorkee.

What This Paper Is About

Vision Transformers split an image into a fixed number of patch tokens and let every token attend to every other token, so compute grows quadratically with token count (stated as O(N²)) and inference stays expensive. Existing training-free fixes either discard unimportant tokens (pruning) or fuse similar ones (merging), but each is usually applied alone, and the paper argues this causes semantic loss or "hallucination-like" prediction instability when inputs are transformed. The goal is a single training-free framework that applies both operations in sequence to pretrained models, together with metrics that measure how stable the compressed model's predictions actually remain.

Key Contributions

1. Prune-and-Merge ViT, a training-free hybrid token reduction framework that stacks class-attention-based pruning (Stage 1) on top of similarity-driven bipartite merging (Stage 2) for off-the-shelf ViT and DeiT backbones.
2. A dual-stage reduction schedule applied at layers l ∈ {3, 6, 9}, with a composite keep rate K = k_p · k_m that lets the balance between pruning and merging be shifted dynamically — an idea the paper labels "Controlled Aggression."
3. Two robustness metrics: Relative Accuracy Drop (RAD), measuring relative accuracy degradation under semantics-preserving transformations, and Prediction Flip Rate (PFR), measuring prediction consistency between original and transformed inputs, with ΔPFR used as a proxy for hallucination.
4. An empirical robustness analysis on ImageNet-100 showing the hybrid keeps RAD near zero or slightly positive against the raw backbone (Table 5) and that ΔPFR shrinks sharply as keep rate rises (Table 4), for ViT-Tiny and ViT-Base under Rotation, Color Jitter and Horizontal Flip.

Main Findings

• Accuracy advantage concentrates in aggressive compression. At K = 0.25, the hybrid reaches 67.62 (DeiT-Tiny), 79.58 (DeiT-Small), 60.12 (ViT-Tiny), 55.58 (ViT-Small) and 71.16 (ViT-Base), compared with EViT at 52.54, 70.16, 44.06, 35.48 and 61.00 respectively, and Top-K at 52.02, 68.78, 42.90, 34.44 and 58.72 (Table 1).
• The advantage is not uniform across the table. For example, at K = 0.25 on DeiT-Base the hybrid scores 47.88 versus EViT 51.32 and Top-K 53.04, and at high keep rates such as K = 0.9 on DeiT-Tiny the hybrid scores 88.06 against EViT 88.44, Top-K 88.48 and ToMe 88.44. The paper's stated claim of "higher top-1 accuracy across all architectural scales" is therefore not reflected in every cell of Table 1.
• Merging-heavy beats pruning-heavy at low keep rates. At K = 0.25 on ViT-Base, the merging-heavy configuration scores 83.56 versus 74.08 for pruning-heavy and 71.16 for balanced; on DeiT-Small the same pattern gives 84.84 versus 79.76 and 79.58 (Table 2). The paper attributes this to critical tokens being fused rather than discarded.
• ToMe cannot reach the most aggressive setting. Dashes in Tables 1, 3, 5, 6 and 7 mark configurations standalone merging cannot reach, because bipartite matching cannot exceed 50% reduction while keeping a complete bipartite graph over the tokens.
• Compute is competitive but slightly higher in GFLOPs terms. On ViT-Base, the baseline is 16.867 GFLOPs; at K = 0.25 the hybrid uses 6.309 versus EViT 6.349, Top-K 6.287 and ToMe not applicable (Table 3). At K = 0.9 the hybrid reports 0.992 for DeiT-Tiny/ViT-Tiny against EViT 0.989 and Top-K 0.981. The GFLOPs tables list identical values under the DeiT and ViT blocks.
• Latency parity at aggressive settings. On ViT-Base at K = 0.25 the hybrid reports the lowest latency of 4.222 ms/img versus EViT 5.061 ms/img and Top-K 4.417 ms/img, against a ViT-Base baseline of 12.721 ms/img. On ViT-Tiny at K = 0.9 it reports 0.971 ms/img (Table 6).
• Throughput gains scale with model size. At K = 0.25 on ViT-Base the hybrid reaches 236.85 imgs/sec against a 78.61 imgs/sec baseline, described as a 3.01× speedup, versus EViT at 197.60 imgs/sec. At K = 0.9 the hybrid reports 101.98 imgs/sec. ViT-Tiny peaks at 1073.18 imgs/sec at K = 0.7 (Table 7).
• Larger backbones are more stable under compression. Average ΔPFR on ViT-Base is 0.3611 at K = 0.25, 0.0422 at 0.50, 0.0099 at 0.70 and 0.0022 at 0.90; ViT-Tiny gives 0.4105, 0.0943, 0.0272 and 0.0059 (Table 4). The paper reads this as ViT-Base's greater parameter density providing more stable "anchor" tokens.
• RAD stays near zero against the raw backbone. Against raw ViT, Tiny's RAD values are 0.073, 0.023, 0.003 and −0.004 across K = 0.25/0.50/0.70/0.90, and Base's are 0.045, 0.009, 0.001 and −0.001. Negative values appear only against other reduction frameworks, indicating lower hallucination rates than those baselines (Table 5).
• Per-augmentation PFR detail. For ViT-Base, raw PFR is 0.0472 (Rotation), 0.0128 (Color Jitter) and 0.0112 (Horizontal Flip); hybrid PFR at K = 0.25 rises to 0.4528, 0.3552 and 0.3464, then falls back to 0.0458, 0.0162 and 0.0130 at K = 0.90 (Table 8).

Methodology in Plain English

The framework treats token reduction as two steps acting in sequence at three points inside the network (layers 3, 6 and 9), positioned post-attention and pre-MLP.

Step 1 — Pruning. The [CLS] token's attention over the patch tokens is averaged across all H heads to score how important each patch is. The top-k patches are kept, where k = max(1, ⌈k_p · P⌉) and P = N − n_s counts non-special tokens. Rather than throwing away the losing patches, all the inattentive tokens are combined into a single representative token by attention-weighted averaging, so global background context is not lost outright.

Step 2 — Merging. The remaining sequence of N_p = k + 1 tokens is split into two roughly equal sets. Each token in one set is matched to its most similar counterpart in the other using cosine similarity between key features, and the r = ⌊(1 − k_m) · N_p⌋ most similar pairs are fused. Merged tokens are combined by a size-weighted average, where the size vector s counts how many original patches a token represents. To keep softmax behavior correct in later layers, proportional attention adds log s to the attention logits, so a merged token exerts the same influence as if its patches were still separate and identical.

Composite budget. The overall keep rate is multiplicative, K = k_p · k_m, so the user can dial the pruning ratio and merging threshold independently. The paper argues this permits shifting reduction burden from pruning to merging as K falls.

Evaluation setup. ViT and DeiT at Tiny, Small and Base scales, 16×16 patches at 224×224 resolution, FP32 inference, official pretrained weights, run with PyTorch and timm on a single NVIDIA T4 GPU (16 GB VRAM) via Google Colab. Evaluation uses the ImageNet-100 validation set, built as a 100-class subset of the ImageNet-1K label set with the corresponding linear projection weights retained. Baselines are the raw backbone, Random Drop, Top-K, EViT and ToMe, all at K ∈ {0.25, 0.5, 0.7, 0.9}, scored by top-1 accuracy, GFLOPs, latency (ms/img) and throughput (imgs/sec). Robustness uses three semantics-preserving transformations — Rotation, Color Jitter and Horizontal Flip.

Why This Matters

Impact on research. Most token-reduction papers report accuracy and speed only; this one adds stability under input transformation as a first-class measurement. RAD and PFR give the field a way to ask whether a compressed ViT is merely smaller or actually less trustworthy, and the paper's finding that merging-heavy schedules dominate at K = 0.25 provides a concrete design rule for hybrid pipelines.

Real-world applications (implied by the paper's framing of resource-constrained and edge deployment):

• On-device or edge vision inference, where ViT-Tiny-class models must fit tight compute budgets — the paper reports peak throughput of 1073.18 imgs/sec for ViT-Tiny at K = 0.7.
• Real-time video or camera analytics, where per-frame latency matters and a 4.222 ms/img result on ViT-Base at K = 0.25 is directly relevant.
• Deployments served from a single modest accelerator, since all throughput and latency figures were measured on one NVIDIA T4 GPU (16 GB VRAM), a widely available and relatively affordable device class.
• Safety- or reliability-sensitive pipelines where prediction flips under harmless input changes (rotation, color shifts, mirroring) are a real failure mode, and where the paper's ΔPFR analysis would guide acceptable compression levels.

Industry relevance. Because the method is training-free and plug-and-play on off-the-shelf checkpoints, it can be retrofitted to already-deployed models without retraining or access to the original training pipeline — the main practical barrier to adopting compression in production. The cost is modest: GFLOPs are essentially on par with EViT and ToMe, while latency and throughput at aggressive keep rates are competitive or better.

Future Directions

• Close the accuracy gap at high keep rates. The hybrid is only competitive-to-best at aggressive K; at K = 0.9 on several backbones EViT, Top-K or ToMe score higher. A scheduling rule that adapts the pruning/merging split as a function of K is the natural next step, and Table 2 already hints at such a rule.
• Extend beyond ImageNet-100. The evaluation is a 100-class subset with the corresponding linear head retained, and the paper itself cites prior work noting that transferability to narrow subsets requires careful alignment. Larger, more diverse, and non-classification benchmarks (detection, segmentation, video, vision-language) remain untested.
• Break the 50% merging barrier. The dashes throughout the tables mark exactly where standalone bipartite merging cannot go. The hybrid sidesteps this by pruning first, but the paper notes that unequal bipartite graphs are an active direction — combining that with the hybrid could push compression further.
• Link measured stability to hardware-aware deployment. The paper references VisionTrim-style frameworks that optimize for specific edge-device latencies but prioritize throughput over prediction stability. Combining hardware-aware scheduling with RAD/PFR constraints is an open and practical question.

Target Audience

Graduate students and researchers working on efficient transformer architectures, model compression or inference acceleration will find the algorithmic content directly usable. Practitioners deploying pretrained ViTs on edge or single-GPU systems will benefit from the latency and throughput tables and from the observation that merging-heavy schedules are preferable at low keep rates. Evaluators and reliability researchers will find RAD and PFR useful even if they never adopt the compression method itself, since these metrics are model-agnostic. The paper assumes comfort with attention mechanics, so it is less suited to readers with no prior exposure to Vision Transformers or token-merging methods like ToMe.

Authors’ abstract

Vision Transformers achieve strong performance but incur high computational costs from quadratic token interactions. Training-free token reduction methods improve efficiency, but their robustness remains underexplored. We propose Prune-and Merge ViT, a training-free framework for off-the-shelf models. It combines class attention-based pruning with similarity-based merging to reduce redundancy while preserving semantic information. To evaluate robustness we introduce two metrics. Relative Accuracy Drop (RAD) measures relative accuracy degradation under transformations, and Prediction Flip Rate (PFR) quantifies prediction consistency. On pretrained ViT and DeiT, Prune-and-Merge reduces computation competitively while maintaining robustness and better accuracy than existing token reduction methods. Our analysis suggests that hybrid token reduction does not introduce hallucination-like behavior and preserves prediction stability for common image transformations.