V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs

AI summary

Overview

Research area: Adversarial machine learning and AI safety, specifically transfer-based black-box adversarial attacks against Large Vision-Language Models (LVLMs).

Technical level: Advanced. The paper hinges on the internal mechanics of transformer attention blocks, channel-wise activation distributions, and entropy analyses of vision encoder features, and assumes familiarity with CLIP/ViT encoders, multi-head self-attention, PGD-style optimization, and black-box transfer attacks.

Scope: The paper diagnoses why prior LVLM adversarial attacks cannot precisely control which image concept is manipulated, and proposes an attack that targets value (V) features instead of patch token features (X).

What This Paper Is About

Existing adversarial attacks on LVLMs can shift an image's overall semantics toward a target, but they fail at fine-grained control over individual concepts in the image — even modifying three concepts at once yields a success rate below 10%. The authors argue this is because attacks operate on patch token features, which are semantically entangled by global context aggregated through attention. Their goal is an attack that reliably and precisely changes one specified object (for example "dog") into another specified object ("cat") while leaving the rest of the image's meaning intact.

Key Contributions

1. Identification of value features as a better attack handle. The authors show that value features (V), computed inside the attention block, suppress the dominant global-context channels present in patch features (X), retaining higher-entropy, disentangled local semantic information — supported by channel distribution, entropy, and text-alignment analyses on CLIP-L/14@336.

2. The V-Attack method. A transfer-based black-box attack targeting V features across an ensemble of surrogate models, composed of a Self-Value Enhancement module (self-attention over V to refine intrinsic semantic richness) and a Text-Guided Value Manipulation module (locating features aligned with a source concept and optimizing them toward a target concept).

3. Large-scale empirical validation. Experiments on MS-COCO and ILSVRC2012 across LLaVA, InternVL, DeepseekVL, GPT-4o, and additional commercial and reasoning models (GPT-5, Gemini-2.5, Claude-4, GPT-o3, Gemini-2.5-thinking, Gemini-2.5-pro), reporting an average 36% improvement in attack success rate over state-of-the-art methods.

4. Ablations isolating the source of the gain. Custom baselines — Patch-Attack (all patch features) and X-Attack (patch features aligned with the source concept) — isolate the effect of targeting V rather than X under otherwise identical settings (ensemble, augmentation, location).

Main Findings

• Value features are more disentangled than patch features. On CLIP-L/14, attention maps for randomly selected patches are highly diffused across unrelated regions, while V's channel distribution is markedly more uniform and suppresses the channels dominant in X.

• V retains richer local information over depth. Entropy of X plummets in the middle layers, while the entropy of V remains consistently high.

• V aligns with text better than X. Peak cosine similarities between feature maps and text prompts are 0.28 vs. 0.22 for "dog" and 0.33 vs. 0.25 for "horse" (V vs. X respectively), with V producing distinct spatial regions and X producing chaotic maps.

• State-of-the-art attack success. On MS-COCO, V-Attack (ensemble) reaches average ASRs of 0.567 on the Image Captioning (CAP) task and 0.560 on VQA, versus the prior best method M-Attack at 0.450 (CAP) and 0.411 (VQA). The single-surrogate variant averages 0.354 (CAP) and 0.355 (VQA).

• The method transfers to the newest commercial systems. Reported ASRs on the same MS-COCO subset: GPT-5 0.534 (CAP) / 0.405 (VQA); Gemini-2.5 0.453 / 0.440; Claude-4 0.263 / 0.217; GPT-o3 0.589 / 0.378; Gemini-2.5-thinking 0.472 / 0.461; Gemini-2.5-pro 0.441 / 0.443.

• Value location is the dominant driver. Removing the location module drops DeepseekVL VQA ASR from 0.636 to 0.463, reverting to near the vanilla baseline.

• Self-Value Enhancement is task-specific. On DeepseekVL it is marginal for CAP (0.560 without enhancement to 0.563 with), but pronounced for VQA (0.521 to 0.636).

• V beats X under matched conditions. X-Attack, identical to V-Attack except for targeting patch features, is consistently outperformed on both CAP and VQA across all evaluated models.

• Ensembles help transferability. Ensemble V-Attack outperforms the single-model attack (the ablation section reports ensemble averages of 0.549 CAP and 0.560 VQA versus single-model averages of 0.354 and 0.355). The single-model attack performs exceptionally well on LLaVA because CLIP-L/14@336 is its vision encoder and serves as the single surrogate.

• Larger perturbation budgets help. On ILSVRC2012 the attack is stronger at epsilon = 16/255 than at 8/255 across all models and methods, including the Patch-Attack baseline.

• Imperceptibility is competitive. V-Attack (ensemble) reports L1/L2 norms of 0.074/0.079 versus 0.079/0.085 for M-Attack, 0.085/0.091 for SSA-CWA, 0.088/0.101 for AnyAttack, and 0.106/0.113 for AdvDiff. AnyAttack and SSA-CWA produce visible artifacts, such as giraffe-like textures on elephant skin in Figure 6, which the authors note may risk detection as AI-generated content.

• Reasoning models are not immune. In an example on GPT-o3, the model failed after 12 seconds of analysis and classified the object as a cat.

Methodology in Plain English

The authors first build an intuition for why past attacks are imprecise. They inspect the internal features of a CLIP vision encoder and find that the usual attack target — the patch token representations — are polluted by global context: each patch "sees" far beyond its own region through attention, so its representation is a blur of the whole image. They then examine a different quantity inside the same attention block, the value features (V), and find these are far cleaner, more uniform across channels, and better matched to specific words. This diagnostic step is the heart of the paper.

The attack itself works in three stages. First, V features are extracted from the last attention block of each surrogate vision encoder (an ensemble of CLIP variants, or a single CLIP-L/14@336). Second, a Self-Value Enhancement step runs self-attention with queries, keys, and values all drawn from V, letting the features sharpen their own internal correlations. Third, the Text-Guided Value Manipulation step uses the model's own image and text projection layers to compute cosine similarity between each enhanced value feature and the source text prompt (for example "dog"); features above a per-model threshold — set as the midpoint between the maximum and minimum similarity — are selected as the ones representing that concept. Only those selected features are optimized, using an ensemble loss that pushes them away from the source text and toward the target text (for example "cat"). The perturbation is updated with a PGD-style sign update under an L-infinity constraint, with cropping and resizing used as data augmentation.

Evaluation uses CAP and VQA tasks on 300 MS-COCO 2017 validation images and 1000 ILSVRC2012 validation images, with GPT-4o acting both to define source/target object pairs and to score outputs on a ternary scale (1 successful substitution, 0.5 partial, 0 unaltered), averaged into an Attack Success Rate. Default settings are epsilon = 16/255, 200 optimization steps, and a crop range of [0.75, 1].

Why This Matters

Impact on research. The paper reframes LVLM adversarial attacks as a feature-selection problem: the failure of prior work is attributed not to optimization strategy but to the entanglement of the representation being attacked. It introduces the Local Semantic Attack as a simplified, measurable probe of controllability and provides an internal-mechanism explanation (channel suppression, entropy, text alignment) that other researchers can test, extend, or try to defend against.

Real-world applications

• Red-teaming and safety auditing of vision-language systems before deployment, particularly proprietary models that cannot be inspected directly.
• Content moderation and misinformation defense, where an attacker could otherwise make an image's subject be described as something it is not.
• Accessibility and image-captioning pipelines, which rely on LVLMs to describe images accurately and could be silently manipulated.
• Robotics and autonomous agent perception built on LVLM backbones, where misread object identity can propagate into downstream decisions.

Industry relevance. The attack is transfer-based and black-box, requiring only surrogate encoders rather than access to the target model, and it remains effective against commercial systems including GPT-4o, GPT-5, Gemini-2.5 variants, and Claude-4. The authors also observe that visibly distorted perturbations from some baselines risk triggering AI-generated-content detection, whereas V-Attack's perturbations are comparatively imperceptible — a practical consideration for anyone deploying detection or provenance systems.

Future Directions

• Defenses targeting value features. If V is the vulnerability surface, defenses might regularize, perturb, or filter value representations rather than patch tokens; the paper explicitly frames its findings as informing future defense strategies, though it does not propose one.
• Understanding why Value Location dominates. The ablation shows location matters far more than enhancement, but the paper does not explain the mechanism behind this asymmetry in depth.
• Extending beyond object substitution. The work focuses on single-concept local attacks; whether the approach generalizes to attributes, relations, or scenes — and to the multi-concept case where prior methods fall below 10% — is not established.
• Robustness under richer prompting and defenses. The evaluation uses fixed task templates (a single caption prompt and three GPT-4o-generated VQA questions per image), leaving open how the attack behaves against adversarial prompt variation, preprocessing defenses, or model updates.

Target Audience

Researchers and practitioners working on adversarial robustness, multimodal model safety, and vision-language model evaluation. It is most useful to those already comfortable with transformer internals and transfer-based attack methodology, and to industrial safety teams responsible for red-teaming or hardening deployed LVLM products. Readers seeking a purely applied or defense-oriented treatment will find the paper's emphasis is on attack construction rather than mitigation, and several implementation details (scoring rubric reliability, dataset construction specifics, model configuration) are referenced to appendices that are only partially included in the material provided.

Authors’ abstract

Adversarial attacks have evolved from simply disrupting predictions on conventional task-specific models to the more complex goal of manipulating image semantics on Large Vision-Language Models (LVLMs). However, existing methods struggle with controllability and fail to precisely manipulate the semantics of specific concepts in the image. We attribute this limitation to semantic entanglement in the patch-token representations on which adversarial attacks typically operate: global context aggregated by self-attention in the vision encoder dominates individual patch features, making them unreliable handles for precise local semantic manipulation. Our systematic investigation reveals a key insight: value features (V) computed within the transformer attention block serve as much more precise handles for manipulation. We show that V suppresses global-context channels, allowing it to retain high-entropy, disentangled local semantic information. Building on this discovery, we propose V-Attack, a novel method designed for precise local semantic attacks. V-Attack targets the value features and introduces two core components: (1) a Self-Value Enhancement module to refine V's intrinsic semantic richness, and (2) a Text-Guided Value Manipulation module that leverages text prompts to locate source concept and optimize it toward a target concept. By bypassing the entangled patch features, V-Attack achieves highly effective semantic control. Extensive experiments across diverse LVLMs, including LLaVA, InternVL, DeepseekVL and GPT-4o, show that V-Attack improves the attack success rate by an average of 36% over state-of-the-art methods, exposing critical vulnerabilities in modern visual-language understanding. Our code and data are available https://github.com/Summu77/V-Attack.