DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction

AI summary

Overview

• Research area: Computer vision, specifically multi-task dense prediction (joint semantic segmentation and monocular depth estimation) built on top of a frozen Vision Foundation Model (VFM) encoder.
• Technical level: Advanced. The paper assumes familiarity with Vision Transformers, DPT-style multi-scale reassembly, depthwise separable convolutions, inverted bottlenecks, and multi-task loss balancing.
• Scope in one sentence: The paper proposes DPNeXt, a pure-convolutional, multi-scale feature-fusion decoder plus a training-only boundary supervision strategy (MTBG), as a lightweight replacement for the standard Dense Prediction Transformer (DPT) decoder in ViT-based multi-task dense prediction.

What This Paper Is About

Multi-task dense prediction uses one network to produce pixel-level semantics and depth, which is useful for robotic scene understanding, but the decoding stage that converts frozen ViT features into task predictions is a bottleneck: naive decoders underperform, while transformer decoders and CNN-style channel-expansion fusion are computationally heavy. The paper's goal is to redesign that decoding stage so a frozen DINOv2-Reg backbone can be exploited with a very small number of trainable parameters, and to add a task-balancing strategy that improves geometric consistency without adding any inference cost. Experiments target Cityscapes and NYUv2.

Key Contributions

1. DPNeXt decoder. A streamlined alternative to standard DPT decoding, built from two components: the Isotropic Projection Adapter (IPA), which projects features into a unified fusion dimension d_fusion = 256 without artificial channel expansion, and the Dual Depthwise Separable Inverted Fusion (DDSIF) block, which replaces dense convolutions with two depthwise separable stages and an expansion ratio r = 2. The authors report that DPNeXt-S reduces trainable parameters by 78.6% relative to standard DPT and completely modularizes the task heads.
2. Multi-Task Boundary Guidance (MTBG). A symmetric boundary-focused supervision strategy to mitigate negative inductive transfer between segmentation and depth. Boundary masks come from provided boundary labels or are derived from semantic segmentation labels using Canny edge detection followed by dilation, following the PIDNet protocol. The auxiliary boundary head is discarded at inference, so there is no extra annotation requirement and no inference overhead.
3. Benchmark evaluation. Cityscapes and NYUv2 experiments report the best semantic segmentation and depth estimation performance among the compared methods, summarized with a combined Joint Performance Score (JPS).
4. Efficiency analysis. Parameter counts, GFLOPs, and measured FPS on resource-constrained laptop hardware (NVIDIA RTX 2080 Laptop GPU), showing that measured inference speed can diverge from theoretical FLOPs.

Main Findings

• Cityscapes accuracy. DPNeXt-B achieves the best overall result among compared methods with a JPS of 0.867 (mIoU 79.64, RMSE 4.95 m) at 93.4M total / 6.8M trainable parameters. DPNeXt-S reaches JPS 0.858 (mIoU 78.32, RMSE 5.45 m) with 28.5M total / 6.5M trainable parameters, outperforming prior baselines while using fewer than 7M trainable parameters. For reference, the reproduced DPT-DINOv2-S baseline scores 75.40 mIoU / 5.52 m / JPS 0.843 with 52.2M total and 30.2M trainable parameters, and SwinMTL scores 76.41 mIoU / 6.32 m / JPS 0.843 with 87.4M trainable parameters. M2H is listed at 77.60 mIoU / 6.10 m / JPS 0.850 but with no parameter count reported.
• NYUv2 accuracy. DPNeXt-B achieves the best semantic segmentation and depth estimation results among the compared methods with mIoU 62.01, RMSE 0.4168 m, and JPS 0.789, using 93.8M total / 7.2M trainable parameters. DPNeXt-S scores 57.15 mIoU, 0.4783 m, JPS 0.762 with 28.9M total / 6.8M trainable. For comparison, M2H is reported at 61.54 mIoU / 0.4196 m / 0.787 with 81.54M total / 59.48M trainable parameters, and large-scale baselines include InvPT (402.1M trainable), TaskPrompter (359.5M), MTMamba++ (258.6M), and MLoRE (550.8M).
• Inference speed. At a fixed 224×224 input, DPNeXt-S reaches 51.02 FPS, the fastest among the compared models, despite having 98.24 GFLOPs — higher than SwinMTL (59.37 GFLOPs, 19.74 FPS) and M2H-Small (51.6 GFLOPs, 20.72 FPS). DPNeXt-B reaches 36.49 FPS at 131.83 GFLOPs. The authors attribute this to pure-CNN operations without custom layers, and state that theoretical GFLOPs do not fully capture practical hardware efficiency.
• Parameter efficiency comes from trainable weights. DPNeXt-B has a larger total parameter count than DPNeXt-S (93.4M vs 28.5M on Cityscapes) but a similar trainable count (6.8M vs 6.5M), since the frozen backbone accounts for most of the total.
• Ablation: decoder design. Moving from standard DPT (52.236M total / 30.177M trainable, 68.261 mIoU, 7.220 m RMSE, JPS 0.796) to +IPA (37.412M / 15.353M, 68.229 mIoU, 7.080 m, 0.797), then to +DDSIF Dual DSConv (27.062M / 5.004M, 68.952 mIoU, 6.839 m, 0.802), then to +DDSIF Inverted Bottleneck (28.131M / 6.073M, 69.660 mIoU, 6.803 m, 0.806). IPA removes channel expansion and cuts trainable parameters while maintaining performance; the inverted bottleneck improves aggregation at a small parameter cost.
• Ablation: MTBG components. With OHEM only, the model scores 70.658 mIoU / 6.998 m RMSE / JPS 0.810. Adding BAD alone improves depth RMSE to 6.805 m (JPS 0.812). Adding BAS alone improves segmentation to 71.066 mIoU but degrades depth to 7.083 m RMSE (JPS 0.811), which the authors describe as a task trade-off. The full MTBG (BAS + BAD) gives 71.391 mIoU / 6.864 m / JPS 0.814, the best aggregate JPS, at only a marginal increase of 0.38M trainable parameters and zero inference cost.
• Qualitative behavior. DPNeXt-S is reported to produce sharper depth maps and better boundary delineation than reproduced SwinMTL and DPT-DINOv2-S, and to avoid the window-shaped artifacts the authors observe in SwinMTL's depth predictions. The figure also marks failure cases of the proposed model.

Methodology in Plain English

The system keeps a pretrained Vision Foundation Model — register-augmented DINOv2 — frozen and never fine-tunes it. Features are taken from four backbone layers (3, 6, 9, and 12); because the ViT patch grid has resolution H/14 × W/14, each level is resized by a scale from {4, 2, 1, 1/2} to form a pyramid. The paper's motivation for skipping CNN-style channel widening is an attention-map visualization of the [CLS] token at layers 3, 6, 9, and 12, which shows attention going from sparse and localized to object-level, to broad contextual, to global scene-level — the authors read this as evidence that semantic grouping emerges early and global context is preserved by self-attention, making extra channel expansion redundant.

The decoding then proceeds in two steps. IPA projects each level to a shared dimension of 256 and resizes it. DDSIF blocks then aggregate features recursively from deep to shallow: the deepest level is passed through a DDSIF block alone, and each subsequent shallower level is combined with the previous output. Every DDSIF block applies 2× bilinear upsampling before its output projection, so the final shared feature P1 has 8× the patch resolution in each spatial dimension; task heads produce predictions that are bilinearly upsampled back to the input resolution. DDSIF itself does two depthwise separable stages: depthwise spatial filtering at the fusion dimension, expansion by a pointwise convolution with ratio 2, a second depthwise filter in the expanded space, and a pointwise projection back to the fusion dimension.

Segmentation uses a pixel-wise OHEM variant of cross-entropy (threshold τ = 0.9, minimum kept-pixel ratio 1/8). Depth uses Scale-Invariant Logarithmic (SiLog) loss with λ_silog = 0.5, with metric depth clipped to a dataset range and log-normalized to [0, 1] following SwinMTL — but integrated into the primary generator network rather than only an auxiliary critic. On NYUv2 only, a surface normal head adds an L1 auxiliary loss that is discarded at inference. MTBG adds an auxiliary boundary loss with per-batch class weights computed from positive and negative pixel counts, plus boundary-aware variants of the segmentation and depth losses (BAS, BAD) that restrict the base loss to boundary pixels; the boundary head is training-only. Total loss weights were initialized from homoscedastic uncertainty weighting and then fine-tuned.

Training used 400 epochs with batch size 8, AdamW with (β1, β2) = (0.9, 0.999), weight decay 10⁻⁴, base learning rate 10⁻³, linear warm-up for the first 5% of updates from a start factor of 10⁻³, then polynomial decay with power 1.0, plus color augmentation, horizontal flipping, and random scaling. Main models trained on an NVIDIA RTX 3090; ablations used four NVIDIA Titan RTX GPUs with 160 epochs and batch size 32; accuracy and speed measurement used an NVIDIA RTX 2080 Laptop GPU. Cityscapes (2,975 training / 500 validation images, 19 semantic categories, depth range [0.001, 80.0] m with CREStereo depth maps) was trained on 518×518 crops and evaluated at full 1024×2048 resolution with 518×518 single-scale sliding-window inference, overlap 0.5, and Gaussian blending for stitching only. NYUv2 (795 training / 654 testing RGB-D pairs, 40-class semantics, [0.001, 10.0] m depth range) was evaluated by resizing to 448×576 in a single forward pass. No test-time augmentation such as flipping or multi-scale inference was used.

Why This Matters

The paper argues that for robotics perception, the practical constraint is not the backbone but the decoder: a widely shared frozen VFM plus a cheap trainable decoder could give multi-task dense outputs at a small trainable-parameter budget. It is a counterexample to the assumption that lower GFLOPs implies faster real-world inference, since DPNeXt-S has higher GFLOPs than some baselines yet the fastest measured FPS on laptop hardware. It also shows a task-balancing mechanism can be added at training time and removed at inference, avoiding the latency costs the authors attribute to auxiliary-branch task-balancing strategies.

Real-world applications suggested by the paper:

• Robotics perception, where dense semantic and geometric cues support downstream modules; the authors note the fused latent feature can also serve as a compact latent representation.
• Autonomous driving perception, with Cityscapes as the driving benchmark and the paper funded under a program on synthetic data collection and simulation technology for autonomous driving.
• 3D reconstruction from dense predictions, illustrated in the framework overview figure.
• Resource-constrained deployment, such as laptop-class hardware, where the fastest measured inference was measured on an RTX 2080 Laptop GPU.

Industry relevance: the paper targets settings where 3D LiDAR and high-maintenance HD maps are considered too expensive, using monocular cameras and frozen foundation models to generate pixel-level semantics and depth for scalable pseudo-labeling and downstream perception, with checkpoints and a demo video promised at a public GitHub repository.

Future Directions

• Extending the single-frame dense prediction framework toward mobile deployment for streaming and open-vocabulary perception, with feature-space analysis.
• Downstream embodied validation for dynamic tasks such as end-to-end autonomous driving and racing.
• Investigating the segmentation-versus-depth trade-off the ablation exposed when BAS is applied alone (segmentation improves to 71.066 mIoU while depth RMSE worsens to 7.083 m), and whether other weightings of the symmetric MTBG supervision resolve it more cleanly.
• Broader validation of the GFLOPs-versus-measured-FPS gap with additional backbones, tasks, and hardware, since the efficiency claims rest on measurements from a single laptop GPU at 224×224 input.

Target Audience

Researchers and engineers working on multi-task dense prediction, efficient decoder design, and ViT/VFM-based perception for robotics or autonomous driving. It is also relevant to practitioners who care about deployment on resource-constrained hardware and about the difference between theoretical FLOPs and measured throughput. Readers without background in DPT-style reassembly, depthwise separable convolutions, or multi-task loss weighting will find parts of the method section dense, which is why the paper sits at an advanced technical level.

Authors’ abstract

Multi-Task Learning (MTL) in robotics perception systems supports comprehensive 3D spatial scene understanding by integrating semantic segmentation and depth estimation. While Vision Foundation Models (VFMs) are increasingly adopted as robust feature encoders, existing decoding strategies present a critical bottleneck. To address this, we propose DPNeXt, a streamlined multi-scale feature fusion decoder and efficient alternative to the standard Dense Prediction Transformer (DPT). DPNeXt uses dual depthwise separable inverted bottlenecks to improve frozen VFM utilization through fusion-centric decoding and independent task modularization. To further mitigate negative inductive transfer between tasks, we introduce the Multi-Task Boundary Guidance (MTBG) strategy. Unlike prior boundary-aware methods that add fusion modules or gating, MTBG applies symmetric boundary-focused supervision to encourage geometric consistency without extra annotation or inference cost. Experiments on Cityscapes show that DPNeXt-S outperforms prior state-of-the-art (SOTA) MTL models, while DPNeXt-B further improves the overall performance and achieves the best results among the compared methods. On NYUv2, DPNeXt-B also achieves the best semantic segmentation and depth estimation results among the compared methods while requiring substantially fewer trainable parameters than prior large-scale MTL models. Compared with the standard DPT, DPNeXt-S reduces trainable parameters by 78.6% and achieves the fastest inference speed among the compared models on resource-constrained laptop hardware. The source code, model checkpoints, and a demo video will be made available at https://github.com/kangjehun/DPNeXt.