VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models

AI summary

Overview

Research area: Multimodal large language models (MLLMs) — specifically the evaluation of "visual prompts" (VPs), the graphical marks such as bounding boxes, arrows, and masks that users draw on an image to point a model at a region of interest.

Technical level: Intermediate. Readers should be comfortable with vision-language benchmarks and the idea of prompting a model with an annotated image, but the paper's framing of the problem is accessible.

Scope: The paper introduces VP-Bench, a two-stage benchmark that measures (1) whether 28 MLLMs can even perceive different styles of visual prompt, and (2) whether those prompts actually help on six real downstream tasks.

What This Paper Is About

When a person wants a model to look at a specific part of an image, they naturally draw a box, an arrow, or a scribble on it. No existing benchmark systematically tested whether MLLMs can correctly interpret these marks, or whether the choice of mark changes how well the model solves a real problem. VP-Bench fills that gap by testing VP perception in isolation first, then measuring how much VPs help on grounded, task-oriented questions.

Key Contributions

1. A two-stage evaluation framework. Stage 1 measures VP perception in natural scenes; Stage 2 measures whether VP understanding transfers to practical problem-solving. The authors state their evaluation includes over 100 times as many images as prior VP benchmarks, with 34,267 images and 38,932 questions in total (versus 303 images in ViP-Bench and 119 in SoV).
2. A large-scale taxonomy of VP perception. Stage 1 covers eight VP shapes across 355 attribute combinations, exceeding the scale of previous studies by more than 40 times. Table 2 reports 78 attributes and 5 colors.
3. Six VP-enabled downstream tasks in Stage 2, drawn from datasets including SZ-CXR, Gleason2019, SD-100, Emotic, MapillaryVistas, SeeClick, and PSG.
4. An empirical account of what drives VP performance, showing that VP shape matters more than most other factors, that regular shapes beat irregular ones, and that stating the VP shape in the text prompt helps.

Main Findings

• Models lag humans on VP perception. The human baseline averages 90.03% overall, while the best model (InternVL3-78B) reaches 87.97%. The three top models — InternVL3-78B, InternVL3-38B, and Molmo-72B-0924 — cluster around 87%. The paper reports roughly a 10% deficit for MLLMs relative to human annotators.

• Shape matters enormously. Bounding boxes average 87.49% accuracy across models versus 67.22% for points (Table 4). Regular shapes (tag, arrow, bounding box, oval) sit around 80% average accuracy, while irregular ones (mask, point, scribble) fall below 70%. Models appear biased toward regular geometric forms present in training data.

• Referring-expression questions are the weak point. Mean accuracy for VP-referring questions hovers around 70%, and the "community favored" Qwen2.5-VL-72B reaches only 75.79%. Average model accuracy by question type is 85.62% for existence, 79.92% for enumeration, 84.03% for rough localization, and 70.21% for referring (Table 4). The paper notes that most models exceed 90% on existence queries, around 85% on enumeration, and over 92% on rough-location queries.

• Contrast color and thin-to-medium strokes work best. A contrast-colored bounding box with medium thickness is optimal for more than half the models, and a contrast-colored thin oval is also frequently best. Contrast color means a distinctive red, green, or blue relative to the background. Thin-to-medium prompts preserve more context.

• Describing the VP in the prompt helps, sometimes dramatically. On InternVL3-78B, adding a VP description raised the average from 81.30% to 87.97%; on Qwen2.5-VL-72B, from 76.89% to 82.77%. The gains are concentrated in hard shapes: for Mask, InternVL3-78B improved by 29.24% and Qwen2.5-VL-72B by 27.3%. For Arrow and Contour, changes were marginal (InternVL3-78B: 0.32% and 1.37%; Qwen2.5-VL-72B: 0.61% and 0.81%). On Point, InternVL3-78B gained 12.38% while Qwen2.5-VL-72B declined 3.05%.

• A model's favorite VP is not always the best VP for a task. Comparing "Best VP" (BVP, the model's own best-perceived combination) against "Random Best VP" (R-BVP, randomly chosen from top-performing combinations) shows mixed results. InternVL3-78B does better with BVP in five tasks (+5.85% on MIA, +1.00% on SD-100, but −0.26% on SeeClick). DeepSeek-VL2 is worse with BVP by 11.71% on MapillaryVistas and 19.43% on SeeClick, yet better by 1.68% on MIA. GPT-4o gains 6.35% on MIA with BVP but loses 10.89% on MapillaryVistas.

• Just training on VP data does not clearly help. ViP-LLaVA-7B reached 73.01% in Stage 1 versus 64.33% for LLaVA-1.5-7B, but on Stage 2 it underperformed LLaVA-1.5-7B by 2.88% on MapillaryVistas and 3.25% on SeeClick. ViP-LLaVA-13B's Stage 1 accuracy dropped 4.13% relative to ViP-LLaVA-7B and 6.72% relative to LLaVA-1.5-13B.

• Debiasing is built in. Table 2 reports that 4.5% of questions lack the visualized VP mentioned in the question, and 8% contain incorrect model VP instructions, designed to catch hallucinated VP recognition.

• InternVL3-78B exceeds humans on contours. It reaches 91.56% on contour VPs versus the human 87.68%, while still trailing humans by 9.79% on Point and by about 2% on Mask and Scribble.

Methodology in Plain English

The authors built the benchmark in two parts.

Stage 1 — can models see the mark? They started from MS-COCO, which has segmentation and bounding box annotations that can be turned into visual prompts. They defined eight VP shapes (tag, bounding box, arrow, mask, contour, oval, point, scribble) and organized attributes hierarchically: shared ones like line width and color, plus shape-specific ones such as vertex shape for boxes, pointer shape for arrows, font size for tags, and point size for points. For each VP they generated multiple-choice questions about presence, spatial location, counting, and referring to the marked object. Question templates were written by hand with GPT-4o's help; distractors came either from hand-crafted rules or from Qwen2-VL-72B with tailored prompts. One debiased sample was added for every six entries.

Stage 2 — does the mark help? From Google, Papers with Code, and Kaggle they selected datasets for six tasks: Medical Image Analysis (SZ-CXR and Gleason2019), 3D object recognition (SD-100), facial emotion recognition (Emotic), street view recognition (MapillaryVistas), GUI element recognition (SeeClick), and scene graph generation (PSG). Each task was capped at 200 randomly selected images unless the dataset was smaller. Questions and distractors followed the same generation recipe. Each model was evaluated using its own best-performing VP from Stage 1, compared against a randomly chosen top VP.

Evaluation. All questions are multiple choice with up to four options, and accuracy is the primary metric, following the VLMEvalKit procedure. The authors also introduce a text "VP description" scheme that verbalizes the prompt's spatial meaning (for example, "the red box outlining the target region") so the same cue appears in both image and text.

Why This Matters

Impact on research. Prior VP benchmarks were small — 303 images in ViP-Bench, 119 in SoV — and largely qualitative. VP-Bench supplies a controlled, large-scale measurement separating "can the model see the mark" from "does the mark help solve the task," which lets researchers attribute failures correctly instead of confounding perception with reasoning.

Real-world applications:

• Medical imaging: the MIA task uses chest X-rays (SZ-CXR) and prostate histopathology (Gleason2019), where a clinician pointing at a suspected lesion is a natural interaction.
• GUI automation: the SeeClick task tests recognizing interface elements, relevant to agents that act on screens.
• Assistive and autonomous driving: MapillaryVistas street view recognition tests prompts on outdoor scenes where a user or system marks a sign, vehicle, or pedestrian.
• Scene understanding and 3D perception: PSG (scene graph generation) and SD-100 (3D object recognition) cover relation reasoning and depth-rich object identification.
• Emotion and face analysis: Emotic tests facial emotion recognition with a marked region.

Industry relevance. Any product that lets users circle or box a region — photo editors, diagram tools, medical viewers, robotics interfaces,

Authors’ abstract

Multimodal large language models (MLLMs) have enabled a wide range of advanced vision-language applications, including fine-grained object recognition and contextual understanding. When querying specific regions or objects in an image, human users naturally use "visual prompts" (VPs), such as bounding boxes, to provide reference. However, no existing benchmark systematically evaluates the ability of MLLMs to interpret such VPs. This gap leaves it unclear whether current MLLMs can effectively recognize VPs, an intuitive prompting method for humans, and use them to solve problems. To address this limitation, we introduce VP-Bench, a benchmark for assessing MLLMs' capability in VP perception and utilization. VP-Bench employs a two-stage evaluation framework: Stage 1 examines models' ability to perceive VPs in natural scenes, using 30k visualized prompts spanning eight shapes and 355 attribute combinations. Stage 2 investigates the impact of VPs on downstream tasks, measuring their effectiveness in real-world problem-solving scenarios. Using VP-Bench, we evaluate 28 MLLMs, including proprietary systems (e.g., GPT-4o) and open-source models (e.g., InternVL3 and Qwen2.5-VL), and provide a comprehensive analysis of factors that affect VP understanding, such as variations in VP attributes, question arrangement, and model scale. VP-Bench establishes a new reference framework for studying how MLLMs comprehend and resolve grounded referring questions.