UPLiFT: Efficient Pixel-Dense Feature Upsampling with Local Attenders

AI summary

Overview

Research area: Computer vision — task-agnostic feature upsampling (also called feature super-resolution) for pre-trained visual backbones, with extensions into generative tasks.

Technical level: Advanced. The paper proposes a new attention-style operator and assumes familiarity with Vision Transformers, cross-attention (query-key-value), convolutional encoder/decoder architectures, VAE latent spaces, and diffusion models.

Scope: The paper introduces UPLiFT, an iterative convolutional feature upsampler built around a new "Local Attender" operator, and evaluates it on semantic segmentation, monocular depth estimation, text-to-image upscaling, and image super-resolution.

What This Paper Is About

Pre-trained visual backbones such as DINOv2 must downsample images into coarse tokens, so their output features are less spatially dense than many dense prediction and generative tasks need. Rather than paying the quadratic cost of running a backbone at higher token density, feature upsamplers learn to map low-resolution features to high-resolution ones as a task-agnostic add-on. This paper asks whether iterative upsampling (which is cheap but prone to "semantic drift") can match the recent cross-attention-based upsamplers (which are accurate but scale quadratically), and answers yes by replacing cross-attention with a locally-defined attentional pooling operator.

Key Contributions

1. UPLiFT architecture: An efficient latent feature upsampler that produces high-quality, pixel-dense features from pre-trained backbone features, using a single compact decoder applied iteratively rather than a set of separately sized modules.

2. The Local Attender operator: A reformulation of attentional pooling into a fully local operation defined by fixed directional offsets around each token. It removes the standard query-key-value formulation and the need for positional embeddings, while preserving the feature-regularization benefit that cross-attention provides.

3. State-of-the-art predictive results at lower cost: UPLiFT surpasses prior feature upsamplers on semantic segmentation across four datasets and is competitive on monocular depth estimation, while running faster than the recent state-of-the-art methods.

4. An extended study of feature upsampling for generative tasks: UPLiFT upsamples VAE latent features for text-to-image upscaling and image super-resolution, achieving competitive results against Coupled Flow Matching (CFM) with far fewer parameters, training data, and inference cost.

Main Findings

• Linear versus quadratic scaling: UPLiFT's inference time and memory scale linearly with the number of visual tokens, whereas LoftUp, JAFAR, and AnyUp-v1 show quadratic scaling. Under a 24 GB memory limit on an A5000 GPU with patch size 16, LoftUp, JAFAR, and AnyUp-v1 ran out of memory at approximately 1500 tokens, AnyUp-v2 (which uses Neighborhood Attention / NATTEN) reached approximately 5000 tokens, and UPLiFT-FAST exceeded 14000 tokens. The paper notes a time jump for UPLiFT-FAST at approximately 13500 tokens, attributed to likely GPU memory allocation issues.

• Semantic segmentation results: UPLiFT achieved the highest mIoU and Accuracy on all four datasets tested. mIoU/Accuracy were 62.55/81.57 on COCO, 85.21/96.51 on VOC, 42.97/76.00 on ADE20K, and 65.38/94.41 on Cityscapes. UPLiFT uses 0.8M parameters and 79.4 ms per image; the UPLiFT-FAST variant uses 0.8M parameters and 41.6 ms, with near-identical scores.

• Depth estimation results: UPLiFT tied with AnyUp-v1 for the lowest RMSE (0.63) on COCO-Stuff depth estimation and reported δ1 of 61.16, placing second behind AnyUp-v1 (61.32) and AnyUp-v2 (61.26). The authors note this is notable because depth requires broader image understanding, yet UPLiFT's features are built only from local information.

• Iterative LiFT degrades at pixel density: The results confirm prior observations that running LiFT iteratively to pixel-dense features hurts performance; the single-step LiFT-2× variant scored better than full iterative LiFT on segmentation (for example, 58.28 versus 57.42 mIoU on COCO).

• Text-to-image upscaling on COCO: With 53M parameters and 5.15 s/image, UPLiFT achieved FID 24.23 for 512→1024 upscaling, compared with CFM-20 at 306M parameters, 8.79 s/image, and FID 28.81 — a latency reduction of 41%. The SD1.5 baseline at 512×512 scored FID 24.88 at 3.96 s/image, and SDXL at 1024×1024 scored FID 24.07 at 39.84 s/image.

• Text-to-image upscaling on reLAION-400M: UPLiFT's 40-step configuration reached CLIP 31.17, FID 20.73, and patch-FID 15.49 at 2.27 s/image, versus CFM-4 (CLIP 26.16, FID 21.61, p-FID 15.83) and CFM-40 (CLIP 26.14, FID 21.67, p-FID 15.96). The paper states UPLiFT surpassed CFM on all three metrics with faster inference.

• Image super-resolution (4× via latent upsampling): With 2 iterations and 53M parameters, UPLiFT scored SSIM 0.84, PSNR 29.93, FID 5.20, and p-FID 6.62 on FacesHQ, and SSIM 0.73, PSNR 26.70, FID 5.29, and p-FID 7.23 on LHQ. CFM, using dataset-specific models with 50 iterations and 113M (FacesHQ) or 306M (LHQ) parameters, scored SSIM 0.82 / PSNR 30.40 / FID 1.36 / p-FID 1.62 on FacesHQ and SSIM 0.69 / PSNR 25.69 / FID 2.27 / p-FID 2.38 on LHQ. UPLiFT's inference time was 271 ms per image, only 13% slower than bilinear upsampling in latent space.

• Training cost advantage for generative use: UPLiFT was trained for 5 epochs on Unsplash-Lite, which contains 25k images, while CFM's general-purpose model was trained on the Unsplash dataset with over 5 Million images. The paper reports UPLiFT uses 1/6th the network parameters and 1/200th the training data of CFM, and that its larger generative model still has one half to one sixth the parameters of comparable CFM models.

• Ablation on training depth: UPLiFT performs best when multiple training depths are used; the primary results use d ∈ {1, 2, 3} per training step, with additional configurations in the appendix.

Methodology in Plain English

The upsampler has two convolutional parts. A shallow but dense encoder looks at the input image once and produces pixel-density guidance features; these are then nearest-neighbor downsampled to whatever size a given upsampling step needs. A small decoder performs 2× upsampling, and the same decoder weights are applied repeatedly until features reach pixel density. LiFT previously had to re-run its image encoder at every step on upsampled images; UPLiFT's encoder runs only once.

The repeated application of a simple decoder is what caused semantic drift in earlier iterative methods. To fix that, the authors put a Local Attender at the end of each decoder step. Instead of computing queries, keys, and values, the Local Attender takes a high-resolution guide map G (the decoder's output) and a value map V (the original backbone feature). A single 1×1 convolution turns G into an attention map with one weight per fixed directional offset in a predefined neighborhood; a softmax normalizes these weights. Offset copies of V are then weighted and summed, so the output is a linear combination of backbone features over a local neighborhood. Because the only learnable element is that 1×1 convolution and the neighborhood size is constant, cost scales linearly — written as O(nT) in the number of spatial tokens T. All operations are relative to each token's own position, so no positional embeddings are needed. The operator also extends to upsampling by grouping the higher-resolution guide tokens into c×c cells, where each cell attends to the neighborhood around the corresponding value token.

Training is self-supervised and mirrors LiFT: a high-resolution image I and a downsampled version I′ are both passed through the frozen backbone to get F and F′. The decoder is then run d times, with intermediate upsampled features compared against intermediate backbone features via an L2 loss summed across steps. For predictive tasks the backbone was DINOv2-S/14, trained for one epoch on ImageNet-1K with maximum ground-truth image size 448 and maximum input size 224, using 3 depth levels. For generative tasks a larger UPLiFT was trained for 5 epochs on Unsplash-Lite with a maximum ground-truth size of 1024 and 4 depth levels, using the Stable Diffusion 1.5 VAE for encoding and decoding. A variant called UPLiFT-FAST applies inference-time optimizations that change neither the architecture nor the training.

Why This Matters

Impact on research. The paper challenges the field's recent shift toward cross-attention-based feature upsampling by showing an iterative convolutional approach can be both more accurate and cheaper. If local, offset-defined attentional pooling can replace query-key-value cross-attention for this task, it suggests the same substitution may be viable in other settings where dense token-to-token attention is expensive but local structure dominates.

Real-world applications:

• Semantic segmentation and scene parsing for autonomous systems and image analysis, where dense per-pixel labels are needed from efficient backbone features.
• Monocular depth estimation, which the paper evaluates on COCO-Stuff.
• Image super-resolution, where latent-space upsampling with a decoder recovers high-resolution images from 4× coarser latents.
• Efficient text-to-image generation, upscaling Stable Diffusion 1.5 outputs from 512×512 to 1024×1024 as a cheaper alternative to native high-resolution generation.

Industry relevance. The core selling point is cost: UPLiFT uses 0.8M parameters and 79.4 ms per upsampling step in the predictive setting, and 53M parameters for the generative setting versus CFM's 306M. Training data is also dramatically smaller (25k images versus over 5 Million). Linear memory scaling matters operationally because cross-attention-based methods ran out of memory at roughly 1500 tokens on a 24 GB GPU in this study, while UPLiFT-FAST exceeded 14000 tokens — meaning high-resolution feature extraction can stay on commodity hardware.

Future Directions

• Closing the fidelity gap on super-resolution: CFM still achieves much lower FID and p-FID on FacesHQ and LHQ, so improving generative fidelity without giving up UPLiFT's low parameter, data, and inference budget is an open problem.
• Recovering the flexibility of arbitrary output sizes: The authors acknowledge that UPLiFT's fixed-step upsamplers make it slightly less flexible than cross-attention methods that can upsample directly to any target size.
• Applying the Local Attender beyond upsampling: Since the operator was motivated by evidence that ViT attention heads learn fixed-directional local offsets, testing it as a general replacement for other attention operations is a natural extension.
• Better use of global context: UPLiFT is slightly behind AnyUp-v1/v2 on depth estimation δ1 despite tying on RMSE, and the authors attribute this to its purely local information use — suggesting room for improving global reasoning while keeping linear scaling. Neighborhood design choices are also deferred to the appendix, implying further tuning may be possible.

Target Audience

Researchers and engineers working on efficient computer vision, feature super-resolution, and dense prediction models; practitioners deploying segmentation, depth, super-resolution, or diffusion upscaling pipelines under compute and memory constraints; and readers interested in alternatives to query-key-value cross-attention that preserve attention's regularizing benefits at linear cost. Readers should be comfortable with Vision Transformers, convolutional encoder-decoders, and latent diffusion terminology.

Authors’ abstract

The space of task-agnostic feature upsampling has emerged as a promising area of research to efficiently create denser features from pre-trained visual backbones. These methods act as a shortcut to achieve dense features for a fraction of the cost by learning to map low-resolution features to high-resolution versions. While early works in this space used iterative upsampling approaches, more recent works have switched to cross-attention-based methods, which risk falling into the same efficiency scaling problems of the backbones they are upsampling. In this work, we demonstrate that iterative upsampling methods can still compete with cross-attention-based methods; moreover, they can achieve state-of-the-art performance with lower inference costs. We propose UPLiFT, an architecture for Universal Pixel-dense Lightweight Feature Transforms. We also propose an efficient Local Attender operator to overcome the limitations of prior iterative feature upsampling methods. This operator uses an alternative attentional pooling formulation defined fully locally. We show that our Local Attender allows UPLiFT to maintain stable features throughout upsampling, enabling state-of-the-art performance with lower inference costs than existing pixel-dense feature upsamplers. In addition, we apply UPLiFT to generative downstream tasks and show that it achieves competitive performance with state-of-the-art Coupled Flow Matching models for VAE feature upsampling. Altogether, UPLiFT offers a versatile and efficient approach to creating denser features.