Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing

AI summary

Overview

Research area: Parameter-efficient fine-tuning (PEFT) of pre-trained vision backbones, with a focus on dense prediction tasks (semantic segmentation, object detection, instance segmentation, panoptic segmentation) rather than image classification alone.

Technical level: Advanced. The paper assumes familiarity with adapter-based PEFT (LoRA, AdaptFormer), mixture-of-experts (MoE) routing, low-rank adaptation, and hierarchical vision backbones such as Swin and ConvNeXt.

Scope: The paper introduces ParaX, an adapter-style PEFT method that uses a shared "expert center" of trainable parameter matrices and a dynamic parameter routing mechanism to generate input-dependent low-rank weights, and evaluates it across dense prediction, pose estimation, remote sensing segmentation, and image classification.

What This Paper Is About

Existing adapter-based PEFT methods use input-agnostic, low-rank transformations, meaning the same learned adapter weights apply to every input, and adapters in different layers do not interact. The authors argue this causes weaker spatial modeling on dense prediction tasks (smaller effective receptive fields) and redundant, highly similar representations across layers. ParaX addresses both issues by letting each module dynamically compose its own weight matrices from a large pool of trainable parameter matrices shared across the layers of a network stage.

Key Contributions

1. A shared expert center design. Each stage of a hierarchical backbone holds a collection of trainable parameter matrices that act as "experts." For channel transformations these come in pairs {E_A ∈ R^(M×C×Ĉ), E_B ∈ R^(M×Ĉ×C)}, where M is the expert center capacity; for spatial transformations a second set {S_A, S_B, S_C} implements multi-kernel depthwise convolutions with kernel sizes K1², K2², K3².

2. A dynamic parameter routing mechanism. A lightweight router (global average pooling, a linear layer reducing to 16 channels, then parallel linear layers with softmax) produces gating vectors {G1, G2} ∈ R^M that are used to aggregate expert matrices into dynamic down-projection (W1 ∈ R^(C×Ĉ)) and up-projection (W2 ∈ R^(Ĉ×C)) weight matrices, making the low-rank adaptation input-dependent.

3. Dynamic multi-scale spatial mixing. Three additional gating vectors {G_A, G_B, G_C} produce dynamic depthwise convolution kernels (D2Convs) applied sequentially with residual connections, with a Spatially-varying Aggregation (SA) module (1×1 convolution plus softmax) combining the multi-scale outputs using spatial attention maps.

4. Cross-layer feature interaction through sharing. Because multiple layers within a stage query the same expert center, the design creates implicit cross-layer feature interaction that the authors show improves feature diversity relative to prior PEFT methods.

Main Findings

• Semantic segmentation on ADE20K (Swin-L, UperNet): ParaX reaches 52.0 mIoU with 7.3M trainable parameters, versus 51.2 mIoU for full fine-tuning with 195.0M parameters, and 51.6 mIoU for Mona with 7.5M parameters. The abstract states this is a 0.8% improvement over full fine-tuning using less than 4% of the trainable parameters.

• Semantic segmentation on ADE20K (Swin-B): ParaX reaches 50.3 mIoU with 5.2M parameters, slightly above full fine-tuning (50.2 mIoU, 86.8M parameters) while saving approximately 95% of trainable parameters, and 0.5% above Mona (49.8 mIoU, 5.2M parameters). With ConvNeXt-B and ConvNeXt-L, ParaX reaches 51.1 and 52.0 mIoU versus 51.4 and 52.4 for full fine-tuning.

• Object detection and instance segmentation on COCO2017 (Mask R-CNN): With Swin-B, ParaX achieves 47.3 AP^b / 42.7 AP^m versus Mona's 46.6 / 42.4 (a stated improvement of 0.7%/0.3%) and full fine-tuning's 47.5 / 42.8. With Swin-L, ParaX reaches 48.6 AP^b / 44.0 AP^m, matching full fine-tuning's 48.6 AP^b and exceeding its 43.8 AP^m, and the paper reports a 3.7%/2.8% improvement over LoRand. With ConvNeXt-L, ParaX reaches 49.5 AP^b / 44.8 AP^m, stated as 0.6%/0.4% above Mona and 1.4%/1.6% above full fine-tuning.

• Panoptic segmentation on COCO2017 (Panoptic FPN): ParaX reaches 48.8 PQ with Swin-B (5.2M parameters), 50.2 PQ with Swin-L (7.3M), 48.9 PQ with ConvNeXt-B (6.5M), and 50.4 PQ with ConvNeXt-L (9.2M). The paper reports improvements of 1.7% and 1.1% in PQ over AdaptFormer with Swin-B and Swin-L, and 1.3% and 1.1% in PQ over CoLoRA with ConvNeXt variants. The authors note a moderate gap with full fine-tuning (50.3/51.4/50.2/51.0 PQ) remains, attributing it to the fundamental limits of PEFT when using less than 8% of full fine-tuning's trainable parameters.

• Image classification with ViT-B/16 (MAE pre-trained): ParaX (3.9M parameters) achieves 87.7% on CIFAR-100, 97.7% on SVHN, 89.7% on Food-101, and 74.3% on ImageNet-R, for a mean accuracy of 87.4%. This is 0.7%, 0.4%, and 0.1% above Mona on CIFAR-100, SVHN, and Food-101, and 1.6% above Mona on ImageNet-R.

• Ablation on latent dimension and expert center capacity (Swin-B, COCO2017): Setting M according to the number of layers L, the best result is [M = L, Ĉ = 128] with 5.2M parameters, 47.3 AP^b and 42.7 AP^m, outperforming [M = 4L, Ĉ = 40] (46.1/42.0), [M = 2L, Ĉ = 72] (46.9/42.4), and [M = L/2, Ĉ = 192] (47.2/42.6).

• Ablation on shared expert center scope (Swin-B stage 3, 18 layers): Splitting the expert center into smaller groups degrades performance: 18 layers/group

Authors’ abstract

Adapting pre-trained vision models using parameter-efficient fine-tuning (PEFT) remains challenging, as it aims to achieve performance comparable to full fine-tuning using a minimal number of trainable parameters. When applied to complex dense prediction tasks, existing methods exhibit limitations, including input-agnostic modeling and redundant cross-layer representations. To this end, we propose ParaX, a new adapter-style method featuring a simple mixture-of-experts (MoE) architecture. Specifically, we introduce shared expert centers, where each expert is a trainable parameter matrix. During a feedforward pass, each ParaX module in the network dynamically generates weight matrices tailored for the current module via a simple dynamic parameter routing mechanism, which selectively aggregates parameter matrices in the corresponding expert center. Dynamic weight matrices in ParaX modules facilitate low-rank adaptation in an input-dependent manner, thus generating more customized and powerful feature representations. Moreover, since ParaX modules across multiple network layers share the same expert center, they improve feature diversity by promoting implicit cross-layer feature interaction. Extensive experimental results demonstrate the superiority of ParaX across diverse visual recognition tasks. Code is publicly released at: https://github.com/LMMMEng/ParaX.