DiscoVL: Unveiling Disentangled C ross-Modal Representation Learning via Orthogonal Adversarial Regularization for V ision-Language Models

AI summary

Overview

• Research area: Parameter-efficient adaptation and generalization of vision-language models.
• Technical level: Intermediate.
• Scope: The paper introduces DiscoVL, a way to adapt CLIP while separating task-specific information from transferable visual-language knowledge.

What This Paper Is About

Prompt-learning methods can make a vision-language model perform better on a target dataset, but the adaptation can mix dataset-specific clues with knowledge that helps the model generalize to unfamiliar classes and domains. DiscoVL aims to keep those kinds of information distinct while improving how visual and textual features interact.

Key Contributions

1. A disentangled representation approach: DiscoVL divides adapted features into a shared part intended to retain general semantics and a task-specific part intended to capture distinctions useful for the target dataset.
2. A cross-modal aligner (DCRA): It uses compact, multi-branch transformations and feedback between image and text features to align information across modalities and model layers.
3. Orthogonal adversarial representation learning (OARL): It combines triplet-based constraints with orthogonal regularization to discourage features from collapsing around class centers and to reduce redundant information.
4. Evaluations across multiple transfer settings: The paper reports results for base-to-novel generalization, cross-dataset evaluation, domain generalization, and few-shot learning, along with component and loss ablations.

Main Findings

• Base-to-novel generalization: Across the reported average results on 11 datasets, DiscoVL scores 85.98% on base classes, 77.61% on novel classes, and 81.58% on their harmonic mean. Compared with MMRL, the paper reports gains of 0.30, 0.45, and 0.38 percentage points, respectively.
• Results across individual datasets: DiscoVL has the highest harmonic mean among the compared methods on 7 out of 11 benchmarks. On EuroSAT, its harmonic mean is 41.60%, an improvement of 1.11 percentage points over the prior best result reported in the paper.
• Cross-dataset evaluation: DiscoVL reports an average accuracy of 67.48%, compared with 67.25% for MMRL, and leads on 6 out of 10 target datasets. It does not lead on every target dataset.
• Domain generalization: The reported average across the ImageNet variants is 60.83%. That is 0.18 percentage points above PromptSRC’s reported average; DiscoVL’s largest reported gain over MMRL among the variants is 0.47 percentage points on ImageNet-R.
• Few-shot learning: The paper reports average results of 73.48% and 76.63% in the 1-shot and 2-shot settings, and 84.58% in the 16-shot setting. It says these results exceed the previous best methods in the settings discussed.
• Ablations: The component study reports harmonic mean rising from 81.20% to 81.32% with DCRA, and to 81.58% after adding OARL. The loss ablation also reports incremental gains from the adversarial triplet terms and orthogonal regularization.

Methodology in Plain English

DiscoVL starts with a pre-trained CLIP model and adapts it using trainable representation tokens rather than changing the whole backbone. The design treats the model’s features as having two roles: retaining broad knowledge useful beyond the training classes, and learning details that distinguish classes in the current dataset.

Its DCRA component transforms visual and textual features through several compact branches. A learned gate chooses how much to use each branch, while feedback passes information between the image and text streams. The goal is to make their representations more consistent without treating all layers and modalities as if they contained the same information.

OARL applies two kinds of constraints. Its triplet objective encourages an image feature to match a representation feature from the same class and stay distinct from one from a different class. Its orthogonal regularization discourages redundant or overly aligned features. The model also uses a cosine-similarity loss to keep adapted class features close to frozen CLIP features.

For predictions on base classes, DiscoVL combines the class-token and task-specific representation predictions. For novel classes, it uses the class-token prediction, relying on that feature to retain more of CLIP’s general knowledge.

The experiments use ViT-B/16 CLIP as the visual backbone. The paper reports evaluations on 11 datasets for the main classification tasks and four ImageNet variants for domain generalization.

Why This Matters

Research impact: DiscoVL focuses on a different problem from simply learning better prompts: it explicitly tries to keep transferable knowledge from being overwritten by task-specific adaptation. Its components provide a way to investigate whether that separation helps generalization.

Potential applications:

• Classification with unfamiliar categories: The base-to-novel setup is relevant when a model must recognize classes it did not see during adaptation.
• Changing data sources or conditions: Cross-dataset and domain-generalization results relate to deploying image classifiers on data that differs from the training set.
• Recognition with limited labels: The few-shot experiments address settings where only a small number of labeled examples are available.
• Specialized image recognition: The evaluated tasks include fine-grained categories, scenes, actions, textures, and satellite imagery.

Industry relevance: The approach builds on a frozen pre-trained model and learns a comparatively small adaptation module, matching the practical interest in adapting capable models without the cost and potential forgetting associated with full fine-tuning. The paper does not report runtime or parameter-count comparisons, so the size of those practical savings is not established by the supplied results.

Future Directions

• Test whether the approach transfers to other vision-language backbones and architectures, rather than only the reported CLIP setup.
• Measure parameter count, training and inference cost, and memory use to clarify the practical efficiency of the added modules.
• Evaluate the stability of the reported gains across more data splits and independent runs, and report uncertainty for comparisons.
• Probe the learned shared and task-specific subspaces directly to test how well they separate transferable semantics from dataset-specific cues.

Target Audience

This paper is most useful to researchers working on vision-language models, prompt learning, transfer learning, and generalization to new classes or domains. It may also interest practitioners adapting image classifiers when labeled data are limited or deployment data differ from training data.

Authors’ abstract

Pre-trained vision-language models excel across varied perception tasks, but adapting them to novel downstream settings without sacrificing generalization remains non-trivial. Existing parameter-efficient prompt learning method often yields inconsistent representations and fails to account for semantic distribution shifts. In this work, we present DiscoVL, a disentangled cross-modal representation learning framework that couples orthogonal adversarial regularization with structured cross-modal alignment for vision-language models. To address the insufficient cross-modal interaction, our DiscoVL designs a multi-branch low-rank residual aligner that decomposes representations into subspaces and enables bidirectional cross-modal feedback between visual and textual streams at each layer. Furthermore, while conventional triplet constraints overfit features to class centroids, we design an orthogonal regularization for adversarial triplet loss, which prevents centroid collapse and substantially boosts generalization. Evaluations on 15 benchmarks demonstrate that DiscoVL delivers consistent improvements over state-of-the-art methods for base-to-novel generalization, cross-dataset evaluation, and few-shot learning