Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers

AI summary

Overview

• Research area: Adversarial robustness and security of vision-language models (VLMs) used as product search rerankers; multimodal information retrieval; generative engine optimization.
• Technical level: Advanced. The method assumes white-box access to model gradients and requires familiarity with adversarial optimization, soft prompt tuning, and Projected Gradient Descent.
• Scope: The paper formalizes and demonstrates a targeted multimodal ranking attack (MGEO) that jointly perturbs a product's image and text to push it to rank 1 in a VLM reranker, evaluated on Qwen2.5-VL-7B over an Amazon-derived product benchmark.

What This Paper Is About

Existing attacks on LLM-based ranking systems manipulate text only, ignoring the visual signals that VLMs depend on, while existing VLM attacks target classification or jailbreaking rather than ranking. The authors define a realistic threat model in which a malicious seller controls one product listing, must keep the image imperceptibly altered and the text fluent and on-topic, and wants that product promoted to rank 1. Their goal is to show that jointly optimizing both modalities exploits the VLM's cross-modal coupling and outperforms text-only attacks, image-only attacks, and a baseline that uses strong commercial generative models.

Key Contributions

1. Problem formulation. Introduces the novel task of multimodal ranking attacks against VLM-based rerankers under a realistic threat model where an adversary controls a single product and must satisfy strict plausibility constraints on both image and text.
2. Method (MGEO). Proposes Multimodal Generative Engine Optimization, described as the first framework that jointly perturbs image and text to manipulate VLM ranking outcomes.
3. Unified adversarial machinery. Combines gradient-based soft embedding optimization for generating fluent adversarial text suffixes (adapted from StealthRank) with a constrained PGD image attack using target, smoothness, and magnitude loss terms, including a spatial weighting scheme based on foreground/background detection.
4. Alternating optimization algorithm. Alternates text and image updates across multiple rounds to exploit cross-modal interactions, and empirically shows this joint attack beats text-only, image-only, and a strong-commercial-model heuristic baseline.

Main Findings

• MGEO produces the largest rank promotion. On Qwen2.5-VL-7B, average rank change was −2.25 for MGEO, versus −1.30 for the image-only attack, −0.73 for the text-only attack, and −0.30 for the HSCM baseline (negative values indicate upward promotion). The paper notes that for a list of ten products, the expected rank change under random promotion is −4.5.
• The joint attack is synergistic, not additive. MGEO's effect exceeds the sum of the text-only and image-only improvements, indicating the two modalities reinforce each other rather than contributing independently.
• Surface quality does not equal rank manipulation. The HSCM baseline, built with gpt-4o-mini for text suffixes and gpt-image-1-mini for image editing, performed worse than even the optimization-based text-only attack. Its outputs were often more fluent and aesthetically edited but weakly correlated with the VLM's scoring mechanism.
• Heuristic refinement optimizes for humans, MGEO optimizes for the model. Qualitative comparison (Figure 3) shows a case where MGEO moved a product from 3/10 to 1/10 while the baseline produced no rank change at all.
• Regularization trades stealth for strength. With smoothness and magnitude weights at (10, 10), average rank change was −1.53; at (5, 5) it was −2.25; at (0, 5) it was −2.31; at (5, 0) it was −2.72; and with no regularization (0, 0) it was −2.29. The fully unregularized setting was therefore not the strongest, which the authors attribute to the model overfitting the target and failing to output the ranking directly at test time.
• Magnitude regularization is the dominant stealth factor. Removing smoothly regularization (s=0, m=5) gave only a marginal improvement, whereas removing magnitude regularization (s=5, m=0) gave a much larger rank improvement but visually conspicuous distortions. The authors adopt the moderate (5, 5) setting for all main experiments.
• The vulnerability generalizes beyond e-commerce. On the OpenEvents V1 news dataset, multimodal attacks achieved a −2.71 rank shift versus −1.84 for text-only baselines.
• Failure cases exist. For some products, substantial promotion required perturbations with clearly noticeable visual artifacts. Figure 5 shows a target promoted from 10/10 to 1/10 but only with conspicuous artifacts, violating the stealth assumption.

Methodology in Plain English

The setup is a reranking scenario: a VLM receives a user query plus a list of candidate products, each with an image and a description, and outputs an ordering. The adversary picks one target product and rewrites only that listing.

For the text side, the researchers treat the product images as a fixed visual context and run gradient-based optimization in the model's continuous embedding space to build an appended suffix. The suffix is optimized against a multi-objective loss with three parts: a ranking loss that maximizes the probability of the desired ranked sequence, a fluency regularizer that keeps the suffix coherent with the original description, and an n-gram penalty that discourages obvious ranking keywords such as "top" or "recommend". After a number of updates, the suffix embedding is greedily decoded into actual tokens and appended to the target's description.

For the image side, they use Projected Gradient Descent on the target image with a loss combining a cross-entropy target term over the desired ranking tokens, a smoothness term penalizing abrupt changes between neighboring pixels, and a magnitude term weighting the total perturbation, where foreground pixels receive a larger weight so changes stay minimal on the main product region. Perturbations are applied after resizing to the model's input resolution but before normalization.

The two branches are then coupled: across multiple rounds, the text suffix is optimized while the image is frozen, then the image is optimized while the suffix is frozen, and so on. Each modality adapts to the other's perturbed state, letting the attack reach joint minima that unimodal attacks cannot.

Evaluation used Amazon product pages across 10 categories, each with 10 to 15 products, with lists of 10 candidates per query to simulate the reranking stage of a two-stage pipeline (typically 10–50 items). All experiments ran on Qwen2.5-VL-7B. Rank change was measured leave-one-target-out, treating each product as the attack target in turn, and the reported metric is the average post-attack rank minus pre-attack rank across all products.

Why This Matters

• Research impact: The work moves ranking-integrity attacks beyond the text-only setting and shows that a previously untested cross-modal channel is exploitable, arguing that the same cross-modal coupling that makes VLMs effective can undermine their ranking faithfulness.
• E-commerce search integrity: Malicious sellers could covertly boost their own listings without obvious marketing language or visible image edits.
• Platform auditing and moderation: Detection pipelines need to inspect imperceptible pixel-level perturbations and unusual suffixes, not just surface-level content quality or brand guidelines.
• Recommendation and content feeds: The OpenEvents V1 result indicates the same class of manipulation applies to news or content ranking, not only products.
• Industry relevance: The result that a commercial generative-model workflow (gpt-4o-mini plus gpt-image-1-mini) produced only −0.30 average rank change suggests that defenses cannot simply rely on the assumption that high-quality generated content is the main threat; gradient-aligned optimization is the more serious risk. The authors state they do not release attack-ready code intended for misuse and frame the work as a step toward defenses and auditing.

Future Directions

• Broader model coverage. The paper's stated limitation is that experiments only cover Qwen2.5-VL; testing additional VLMs may reveal model-specific behavior or new challenges.
• Dynamic marketplaces. The current framework assumes a static listing with fixed competitors and fixed metadata; real platforms have dynamic content updates and platform-controlled preprocessing that would require adapting the attack procedure.
• Defenses. The authors explicitly leave systematic defenses unexplored and call for input perturbation detection, ranking consistency checks, and adversarial training of the ranker.
• Stealth–strength tradeoff. The failure analysis shows some products can only be promoted at the cost of visible artifacts; understanding what makes a target visually and semantically flexible is an open question.

Target Audience

Researchers and practitioners in adversarial machine learning, multimodal retrieval, and recommender-system security; platform trust-and-safety and search-integrity engineers; and anyone studying the robustness of VLM-based ranking pipelines. Readers need comfort with gradient-based optimization and white-box threat models to follow the methodology section in detail.

Authors’ abstract

Vision-Language Models (VLMs) integrate visual and textual knowledge into unified representations that increasingly underpin modern retrieval and recommendation systems. However, it remains unclear how reliably these models utilize their cross-modal knowledge when ranking multimodal items, and whether their knowledge grounding can be subverted. In this paper, we expose a fundamental vulnerability in how VLMs apply multimodal knowledge for product ranking: through Multimodal Generative Engine Optimization (MGEO), we show that an adversary can manipulate a VLM's ranking decisions by jointly crafting imperceptible image perturbations and fluent textual suffixes that exploit the model's internal cross-modal knowledge coupling. Using an alternating optimization strategy, MGEO targets the deep interactions between visual and linguistic representations within the VLM, achieving rank manipulations that substantially exceed those of unimodal attacks and heuristic baselines powered by strong commercial models. Our findings reveal that surface-level content quality is insufficient for rank promotion; instead, direct alignment with the model's internal knowledge utilization mechanism is required. These results raise important questions on the faithfulness and robustness of knowledge grounding in multimodal foundation models, and motivate future work on defense mechanisms for multimodal retrieval systems. Code is available at: https://github.com/glad-lab/MGEO