AnyUp: Universal Feature Upsampling

AI summary

Overview

Research area: Computer Vision — learned feature upsampling for pre-trained vision encoders (DINOv2, CLIP, SigLIP, MAE, DINOv3).

Technical level: Intermediate. The paper assumes familiarity with vision transformers, attention mechanisms, linear probing, and dense prediction tasks, though its central idea can be grasped without deep architectural knowledge.

Scope: The paper introduces AnyUp, a single learned feature upsampler trained once that can upsample features from any encoder, at any resolution, without encoder-specific retraining, and evaluates it on semantic segmentation, monocular depth estimation, and surface normal estimation.

What This Paper Is About

Pre-trained vision encoders such as DINO, CLIP, SigLIP, and MAE output feature maps whose resolution is limited to the number of transformer tokens, which prevents pixel-level prediction. Learned upsamplers (FeatUp, LoftUp, JAFAR) improve on simple bilinear upsampling but must be retrained for every feature extractor they are paired with. AnyUp's goal is a single upsampling model that generalizes to unseen feature types and dimensionalities at inference time while still setting state-of-the-art upsampling quality.

Key Contributions

1. AnyUp, an inference-time feature-agnostic upsampler. The authors state it is the first learnable upsampling method that generalizes to any input feature at inference time, can upsample from any to any resolution, is task-agnostic, and is trainable, as laid out in Table 1 of the paper.

2. A feature-agnostic layer. A convolution layer that represents feature maps of any dimensionality as maps with canonical dimensionality by convolving each input channel independently with a learned kernel basis, applying a softmax over the basis dimension, and averaging the resulting contributions over all input channels. This makes the output invariant to the number of input channels.

3. A window attention-based upsampling architecture. Attention is restricted to local windows around each query point rather than global attention, which avoids attention "outliers" where distant, unrelated image regions serve as references, and which simplifies the optimization objective while improving efficiency.

4. An improved training pipeline. An image-part (local crop) based supervision strategy plus self-consistency and input-consistency regularization, which the authors report is faster and more memory-efficient than JAFAR's low-resolution training and more light-weight than LoftUp's second-stage strategy. Code and pre-trained weights are released publicly.

Main Findings

• Semantic segmentation: With inputs at 448x448 (features 28x28 or 32x32, upsampled 14x or 16x back to input resolution), AnyUp reaches 62.16 mIoU / 81.37 accuracy on COCO, 84.00 mIoU / 96.19 accuracy on PASCAL VOC, and 42.43 mIoU / 75.85 accuracy on ADE20k. On PASCAL VOC it is slightly below JAFAR (84.36 mIoU / 96.22 accuracy); on COCO it edges out LoftUp (62.15 / 81.32) and on ADE20k it leads FeatUp (42.19 / 75.57).

• Depth and surface normals (NYUv2, 224x224): AnyUp achieves the lowest surface normal RMSE (31.17), ahead of JAFAR (31.54), FeatUp (32.69), Bilinear (32.70) and LoftUp (33.94), and the highest angular accuracies at 11.25 degrees (0.29), 22.5 degrees (0.57) and 30 degrees (0.69). For absolute depth it reaches RMSE 0.4755 and delta-1 0.8216, and for relative depth RMSE 0.3378 and delta-1 0.9233, both best among the compared methods.

• Any-to-any resolution: Across the tested resolution changes (16 to 112, 32 to 224, 32 to 112 pixels), AnyUp leads on most settings, e.g. 62.25 mIoU on COCO at 32 to 224 versus FeatUp's 61.92 and JAFAR's 61.91. The paper notes AnyUp is slightly worse than the best competing method only when upsampling from 16 to 112 pixels (59.63 vs JAFAR's 59.79 mIoU).

• Feature space preservation: Using linear probes pre-trained on original low-resolution DINOv2 ViT-S features, AnyUp scores 40.83 mIoU / 74.94 accuracy on ADE20k and RMSE 0.498 / delta-1 0.822 on NYUv2 depth, outperforming Bilinear, Guided Filter, FeatUp and JAFAR. LoftUp collapses in this setting (4.27 mIoU / 46.58 accuracy, RMSE 0.765), which the authors attribute to its affinity matrix loss aligning intra-image feature similarities rather than directly supervising predictions with target features.

• Cross-encoder generalization: A model trained only on DINOv2 ViT-S, tested on SigLIP 2 LoftUp features, reaches 51.68 mIoU (73.35 accuracy) and 0.59 / 0.48 RMSE, while a LoftUp model trained and tested on SigLIP 2 LoftUp reaches only 40.73 (64.87) and 0.72 / 0.60. The same DINOv2-trained AnyUp tested on SigLIP 2 JAFAR reaches 58.51 (78.36) and 0.91 / 0.58, compared to 60.10 (79.40) and 0.93 / 0.60 for a JAFAR model trained and tested on SigLIP 2 JAFAR. On DINOv3 (ViT-S+), the DINOv2-trained model reaches 62.96 (81.82) and 0.51 / 0.37 versus 62.99 (81.84) and 0.51 / 0.37 when trained on DINOv3 itself.

• Multi-backbone training: Training on DINOv2 (ViT-S), CLIP (ViT-B), SigLIP (ViT-B), DINOv2 with registers (ViT-S) and an ImageNet-trained ViT-B improves average performance across test backbones (58.80 mIoU / 78.98 accuracy, 0.6758 / 0.4685 RMSE) compared to DINOv2-only training (58.20 / 78.54, 0.6778 / 0.4709), and improves unseen DeiT (ViT-B) results (54.83 / 76.56 vs 53.93 / 75.90), at the cost of a small drop on DINOv2 (62.04 / 81.30 vs 62.16 / 81.37).

• Architecture generalization: Varying the DINOv2 size used for upsampler training and for linear probing, the paper reports the trend ViT-L >= ViT-B >= ViT-S holds regardless of the training model, and that training on a smaller, less powerful ViT does not significantly degrade upsampling quality.

• Ablations: The full model outperforms all ablated variants. Removing window attention gives 62.12 (81.34) and 0.4854 / 0.3449; removing the proposed data sampling in favor of JAFAR's simpler approach gives 62.03 (81.28) and 0.4773 / 0.3387; removing all regularization gives 61.90 (81.23) and 0.4786 / 0.3401; removing the feature path for key computation gives 61.97 (81.23) and 0.4791 / 0.3441. Notably, the last ablation still matches prior upsampling methods, indicating that position and color matching alone provide a relatively strong prior when trained with their pipeline.

• Filter basis size: Performance increases with larger basis size, from 0 filters (61.97 mIoU, 0.4791 / 0.3441) up to 256 (62.19, 0.4765 / 0.3371). The implementation uses M = 128 (62.16, 0.4755 / 0.3378). The comparison reported as SOTA is 62.15, 0.4816 / 0.3413.

• Efficiency: AnyUp has 0.8M parameters (FeatUp 0.2M, LoftUp 4.3M, JAFAR 0.7M). The authors report that window attention reduces runtime and memory requirements by more than 50% compared to JAFAR and LoftUp, while being slightly less efficient than FeatUp. Training uses ImageNet for 100,000 steps, batch size 4 with 4 random local crops per image, AdamW at learning rate 2e-4, taking around 5 hours on a single NVIDIA-H100 GPU.

Methodology in Plain English

AnyUp builds on JAFAR's formulation of upsampling as attention from high-resolution image pixels (queries) to low-resolution feature patches (values), which is naturally resolution-agnostic. The authors address two limitations of that design.

First, JAFAR's initial feature-processing layer only accepts features of a fixed dimensionality and must be relearned per backbone. AnyUp replaces it with a feature-agnostic layer: each input channel is convolved separately with a set of learned small kernels, a softmax is applied across those kernel responses, and the results are averaged across all input channels. Because channels are pooled rather than mixed, the output always has the same canonical number of channels

Authors’ abstract

We introduce AnyUp, a method for feature upsampling that can be applied to any vision feature at any resolution, without encoder-specific training. Existing learning-based upsamplers for features like DINO or CLIP need to be re-trained for every feature extractor and thus do not generalize to different feature types at inference time. In this work, we propose an inference-time feature-agnostic upsampling architecture to alleviate this limitation and improve upsampling quality. In our experiments, AnyUp sets a new state of the art for upsampled features, generalizes to different feature types, and preserves feature semantics while being efficient and easy to apply to a wide range of downstream tasks.