SPOT: Sparsification with Attention Dynamics via Token Relevance in Vision Transformers

AI summary

arXiv: 2511.10488v1 [cs.CV] — 13 Nov 2025 Authors: Oded Schlesinger, Amirhossein Farzam, J. Matias Di Martino, Guillermo Sapiro Affiliations listed: Duke University, Princeton University, Universidad Católica del Uruguay, Apple

Overview

Research area: Computer vision / efficient deep learning — specifically token sparsification inside Vision Transformers (ViTs).

Technical level: Advanced. The method builds directly on self-attention mechanics, attention-map statistics, and fine-tuning objectives, though the core idea is expressible simply.

Scope: The paper proposes SPOT (Sparsification with attention dynamics via token relevance), a lightweight, plug-in module that predicts which image tokens are redundant by combining token embeddings with attention statistics gathered across transformer layers, enabling token pruning at inference with up to 40% fewer GFLOPS while maintaining or improving accuracy.

What This Paper Is About

Vision Transformers process every image patch as a token, and the cost of self-attention grows quadratically with the number of tokens, so much of that computation is spent on patches that carry little task-relevant information. Existing methods for deciding which tokens to remove typically look at only one source of evidence at a single layer — either token embeddings or class-token attention scores — which the authors argue leads to suboptimal and noisy decisions. SPOT addresses this by predicting token importance from a compact summary of both token features and attention-map statistics, including how those attention distributions evolve across layers, then dropping the lowest-ranked tokens.

Key Contributions

1. A framework for cross-layer token importance detection. SPOT models attention dynamics across layers, not just the current layer, using token-level information and attention-derived information to judge which tokens matter for the task.

2. A modular, computationally inexpensive predictor. The token-prioritization module can be plugged into existing ViT self-attention layers without structural modification to the backbone, and is designed to add negligible overhead.

3. Efficiency gains without accuracy loss. The authors report up to a 40% reduction in GFLOPS relative to standard ViTs while maintaining or even improving accuracy, across DeiT and LV-ViT backbones in tiny and small sizes.

4. Broad empirical validation and versatility. Evaluations span ImageNet-1K, cross-dataset transfer to CIFAR-100, Food-101, DTD and EuroSAT, integration with soft sparsification methods, and qualitative visualizations showing retained tokens align with semantically meaningful image regions.

Main Findings

• Accuracy and efficiency on ImageNet-1K hard sparsification (DeiT-T): baseline DeiT-T reaches 72.2% at 1.3 GFLOPS. SPOT reaches 72.3% (+0.1) at 0.8 GFLOPS (-0.5), while DynamicViT* gets 71.4 (-0.8) at 0.8 (-0.5), A-ViT 71.0 (-1.2) at 0.8 (-0.5), EViT* 71.9 (-0.3) at 0.8 (-0.5), and Zero-TPrune 70.4 (-1.8) at 0.9 (-0.4).

• DeiT-S results: baseline 79.8% at 4.6 GFLOPS. SPOT is reported at 79.5 (-0.3) at 2.8 GFLOPS (-1.8) and at 79.7 (-0.1) at 3.0 GFLOPS (-1.6). For comparison, DynamicViT 79.3 (-0.5) at 2.9 (-1.7), A-ViT 78.6 (-1.2) at 3.6 (-1.0), EViT 79.4 (-0.4) at 3.0 (-1.6), IA-RED² 79.1 (-0.7) at 3.2 (-1.4), RL4EViT 79.2 (-0.6) at 3.0 (-1.6), Zero-TPrune 79.4 (-0.4) at 3.1 (-1.5), and METR 79.5 (-0.3) at 3.0 (-1.6).

• LV-ViT results: LV-ViT-T baseline is 79.1% at 2.9 GFLOPS; DynamicViT* is 77.1 (-2.0) at 2.0 (-0.9) and SPOT is 77.6 (-1.5) at 2.0 (-0.9). LV-ViT-S baseline is 83.3% at 6.6 GFLOPS; DynamicViT is 83.0 (-0.3) at 4.6 (-2.0), EViT is 83.0 (-0.3) at 4.7 (-1.9), and SPOT is 83.1 (-0.2) at 4.5 (-2.1).

• Headline efficiency claim: SPOT attains up to a 40% reduction in GFLOPS, illustrated by the DeiT-S case going from 4.6 to 2.8 GFLOPS.

• Low predictor overhead: the predictor adds only 0.032, 0.051, 0.129, and 0.129 GFLOPS for DeiT-T, LV-ViT-T, DeiT-S, and LV-ViT-S respectively — approximately 4% of the models' GFLOPS and parameter counts.

• Attention statistics carry most of the signal (ablation): when mean-based statistics are removed from the feature set, accuracy is 79.4 (-0.4) with predictor cost 0.120 GFLOPS and overall 3.0 (-1.6); when variance-based statistics are removed, 79.5 (-0.3), 0.120, and 3.0 (-1.6); when cross-layer terms D^M and D^V are removed, 79.5 (-0.3), 0.111, and 3.0 (-1.6). The authors note the accuracy decline across ablations supports the value of the holistic within-layer and cross-layer approach.

• Token information is not strictly necessary: decreasing d_remap reduces the projected token-embedding dimension. GFLOPS when d_remap = 0 is 2.9, so the savings from d_remap = d_k down to d_remap = 0 amount to 0.1 GFLOPS. The authors report that compact attention-map representations still enable effective sparsification with limited or no reliance on token embedding information.

• Integration with soft sparsification: with SPViT* on DeiT-T, SPOT raises accuracy from 72.0 to 72.1 (+0.1) at 0.8 GFLOPS; with dTPS on DeiT-T, from 72.9 to 73.0 (+0.1) at 0.8 GFLOPS. On DeiT-S, SPViT* goes from 79.5 to 79.7 (+0.2) at 2.9 (-1.5) GFLOPS, and dTPS from 80.1 to 80.2 (+0.1) at 3.0 (-1.6) GFLOPS. The authors note these gains occur where existing methods already saturate performance.

• Cross-dataset transfer with DeiT-T: baseline (1.3 GFLOPS) scores 66.2% on CIFAR-100, 56.7% on Food-101, 58.7% on DTD, and 87.8% on EuroSAT. SPOT at 0.8 GFLOPS (-0.5) scores 66.9 (+0.7), 60.7 (+4.0), 62.0 (+3.3), and 90.1 (+2.3) respectively.

• Interpretability: visualizations on ImageNet-1K validation samples (DeiT-S) show progressively pruned tokens correspond to regions of lower semantic saliency, with retained tokens aligned to semantic image objects.

• Not reported in the provided content: the appendix tables and figures referenced for perturbation robustness (Table 5), dataset properties (Table 6), attention aggregation strategy comparison (Table 7), information granularity analysis (Table 8), smaller-scale visualizations (Figure 6), throughput analysis (Figure 7), the third loss coefficient value (λ₃), and the full details of how λ values were set, are cited in the paper but their contents are not included in the truncated text.

Methodology in Plain English

A ViT splits an image into patches, turns each into a token, and lets each token attend to all others through an attention map. SPOT exploits the fact that these maps are already computed and normally thrown away after each layer.

For each layer and each attention head, SPOT summarizes the attention map in three ways:

• the row from the class token to all patch tokens, and the column from all patch tokens back to the class token (the class token's outgoing and incoming attention);
• the mean and variance of each patch token's outgoing attention (its row in the non-class submatrix);
• the mean and variance of each patch token's incoming attention (its column in that submatrix).

The same statistics are accumulated from the input layer up to the current layer, giving the predictor a picture of how attention has evolved. These summaries are concatenated with token embeddings that have been projected into a smaller d_remap-dimensional space and split into "global" and "local" halves, following the approach of DynamicViT. The combined vector — per head, per layer — is fed to a small multilayer perceptron that outputs an importance score.

Tokens are then filtered in stages. A retention rate ρ (set to 0.7 in the main experiments) determines how many tokens survive each decision, and at iteration k the effective retention rate is ρ^k = ρ^k. Masks are drawn with Gumbel-Softmax sampling to keep training differentiable, and masking is hierarchically compatible: a token dropped at iteration k stays dropped in all later iterations T > k. The class token is always retained.

Training fine-tunes both the predictor modules and the ViT backbone jointly. The total loss combines four terms: a cross-entropy classification loss, a sparsity-rate deviation loss that pushes the empirical retention rate toward the target rates, a KL-divergence term matching the original pre-trained model's predictions, and a cosine-similarity term matching last-layer token representations for retained tokens. Models were optimized with AdamW on eight NVIDIA GeForce RTX 4090 GPUs, using a learning rate of 0.001 for the network and 0.01 for the prediction modules, fine-tuned for 80 epochs with batch size 512, patch size 16×16, and 224×224 ImageNet inputs. Loss coefficients λ₁ and λ₂ were set to 2 and 0.5 respectively across all experiments.

Why This Matters

Impact on research. The paper argues that redundant-token detection — not just token removal — is the crux of efficient ViT sparsification, and that using attention dynamics across layers plus token features gives a more context-aware signal than single-source, single-layer heuristics. It also positions the module as a drop-in component that can improve both hard and soft sparsification pipelines, and it flags a known problem in the literature: attention maps contain spikes with seemingly spurious high-attention values that do not correlate with semantically relevant features.

Real-world applications:

• On-device and edge vision, where 40% reductions in GFLOPS and a predictor costing roughly 4% of model GFLOPS make large ViTs more deployable under fixed compute or power budgets.
• Large-scale image classification services, such as content tagging and search, where inference throughput per image directly affects serving cost.
• Domain-specific recognition with limited data, such as remote sensing (EuroSAT), texture recognition (DTD), and food recognition (Food-101), where SPOT improved accuracy over the baseline in the reported transfer experiments.
• Robust perception under noisy inputs, since the paper reports maintained performance on noisy and corrupted inputs (details deferred to the appendix).

Industry relevance. The approach requires no structural changes to the ViT backbone and reuses already-computed attention values, which lowers the barrier to retrofitting existing trained models. It is also reported to generalize across four different backbone configurations (DeiT-T, DeiT-S, LV-ViT-T, LV-ViT-S), and to compose with existing soft-sparsification methods rather than replacing them. Code and models are released at https://github.com/odedsc/SPOT.

Future Directions

• How far can attention-only prediction go? The ablation shows GFLOPS drop by only 0.1 when token-derived information is removed entirely (from 2.9 at d_remap = 0). The open question is whether attention statistics alone can carry the full task, or whether token features remain necessary at other scales and datasets.

• Explaining the DeiT-T gains. SPOT simultaneously improved accuracy and efficiency on DeiT-T, which the authors attribute to a possible implicit regularization effect from removing noisy tokens in a low-capacity model. This is stated as a likely explanation rather than a proven mechanism, leaving it open for further investigation.

• Reconciling with attention spikes. The paper motivates its aggregation strategy by the presence of seemingly spurious high-attention values, and defers a theoretical perspective to the appendix. Whether the cross-layer aggregation fully resolves those spikes, or merely dampens them, is not settled in the main text.

• Extending beyond the evaluated settings. The current experiments use retention rate ρ = 0.7 and K = 3 sparsification iterations, with classification on ImageNet-1K and four smaller transfer datasets. How the framework behaves at more aggressive retention rates, more iterations, or on tasks such as detection and segmentation — where the paper notes transformers have shown promise — is not addressed in the experiments reported here.

Target Audience

This paper is most useful to researchers and engineers working on efficient transformer inference, model compression, and token pruning or merging. It also suits practitioners deploying ViTs on resource-constrained hardware who need a module they can attach to an existing fine-tuned backbone, and computer-vision researchers interested in interpretable, attention-driven token selection. Readers should be comfortable with self-attention formulations and standard ViT training pipelines; the mathematical content is presented in full in the methods section.

Authors’ abstract

While Vision Transformers (ViT) have demonstrated remarkable performance across diverse tasks, their computational demands are substantial, scaling quadratically with the number of processed tokens. Compact attention representations, reflecting token interaction distributions, can guide early detection and reduction of less salient tokens prior to attention computation. Motivated by this, we present SParsification with attentiOn dynamics via Token relevance (SPOT), a framework for early detection of redundant tokens within ViTs that leverages token embeddings, interactions, and attention dynamics across layers to infer token importance, resulting in a more context-aware and interpretable relevance detection process. SPOT informs token sparsification and facilitates the elimination of such tokens, improving computational efficiency without sacrificing performance. SPOT employs computationally lightweight predictors that can be plugged into various ViT architectures and learn to derive effective input-specific token prioritization across layers. Its versatile design supports a range of performance levels adaptable to varying resource constraints. Empirical evaluations demonstrate significant efficiency gains of up to 40% compared to standard ViTs, while maintaining or even improving accuracy. Code and models are available at https://github.com/odedsc/SPOT .