What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging

AI summary

Overview

Research area: Computer Vision, specifically vision-language models (VLMs), negation understanding, and described object detection (DOD).
Technical level: Advanced.
Scope: This paper introduces a negation-focused dataset pipeline and a lightweight token-merging adaptation method to reduce affirmative bias in VLMs for object detection.

What This Paper Is About

State-of-the-art VLMs often ignore negation in text, a failure called affirmative bias. This is especially damaging in described object detection, where a model must detect objects based on descriptions that may specify what is not present or what attributes an object does not have. The paper aims to make VLMs reliably understand negation by generating better negation data and fixing how negation cues are represented during tokenization.

Key Contributions

1. CoVAND dataset pipeline: A systematic chain-of-thought (CoT) and VQA-based pipeline that generates high-quality, instance-grounded negation data for training and evaluation.
2. NegToMe token merging module: A lightweight text token merging method that groups negation cues with attributes into coherent semantic phrases, preserving negation polarity at the input level.
3. Parameter-efficient adaptation: Integration of NegToMe with strategic LoRA fine-tuning, enabling robust negation understanding with limited data and low adaptation cost.
4. Strong empirical gains: Significant improvements on challenging negation benchmarks, including a lower false positive rate and up to +10.8 NMS-AP on OVDEval, with generalization to state-of-the-art VLMs.

Main Findings

• Affirmative bias is architectural as well as data-driven: VLMs lose negation cues when tokenization fragments terms like "not" and object or attribute words into separate tokens.
• Token merging preserves meaning: NegToMe binds fragmented negation tokens into a single semantic token, so "not" plus "girl" is not treated as simply "girl" and keeps a distinct negative meaning.
• CoVAND improves training signal quality: The CoT and VQA-based pipeline produces instance-grounded negation examples that are more useful than ordinary captions or generic VQA data.
• Detection performance rises sharply: The method boosts NMS-AP by up to +10.8 points on OVDEval and reduces false positives on negation-heavy detection tasks.
• The approach generalizes and is efficient: NegToMe works with parameter-efficient LoRA fine-tuning and transfers to state-of-the-art VLMs even with limited data.

Methodology in Plain English

The researchers first built a data generation pipeline called CoVAND. It uses chain-of-thought reasoning and visual question answering to create descriptions where negation is tied to specific objects in an image, rather than being vague or disconnected from the visual scene.

Next, they introduced NegToMe, a module that merges text tokens before the VLM processes them. In standard tokenization, a phrase like "not girl" may be split into separate pieces, and the model can effectively ignore "not." NegToMe groups the negation word with the relevant attribute or object into one coherent token, so the model sees a meaning that is correctly different from the positive phrase alone.

They then fine-tuned VLMs using LoRA, which updates only a small number of parameters. This makes the adaptation lightweight and practical, while still improving negation understanding. They evaluated the method on negation-focused detection benchmarks, especially OVDEval.

Why This Matters

Impact on research: This work identifies negation as a fundamental weakness in VLMs and offers both a dataset and an architectural fix. It shifts attention from simply scaling models to improving tokenization, compositional reasoning, and evaluation for negation.

Real-world applications:

• Security and surveillance: Detect instructions such as "person not wearing a helmet" or "vehicle without a license plate" more reliably and with fewer false alarms.
• E-commerce and visual search: Support queries like "dress without stripes" or "shoe not red" so users find items matching negative constraints.
• Robotics and autonomous systems: Enable robots to follow negative instructions, such as avoiding objects that lack a required safety marking.
• Accessibility and image understanding: Improve descriptions like "room with no people" or "table without a vase," which matter for assistive technologies.

Industry relevance: Companies deploying VLMs for visual search, content moderation, retail, security, and robotics can benefit from lower false positive rates and more reliable detection. The parameter-efficient LoRA integration also lowers the cost of adapting large models, making the approach attractive for production systems with limited compute or data.

Future Directions

1. Scale and diversify CoVAND across more languages, domains, and complex negation forms, including double negation and conditional negation.
2. Extend NegToMe beyond current VLMs to generative vision-language models, multimodal LLMs, and other tokenization-heavy architectures.
3. Evaluate on broader detection and reasoning benchmarks, measuring the trade-off between false positives and false negatives under negation.
4. Combine NegToMe with other reasoning or tokenization strategies, and explore automatic detection of negation cues in free-form text.

Target Audience

VLM and computer vision researchers, NLP researchers working on compositional reasoning, object detection practitioners, dataset builders, and engineers deploying visual search, security, robotics, or accessibility systems. Graduate students and advanced undergraduates with background in deep learning and vision-language models will benefit most.

Authors’ abstract

State-of-the-art vision-language models (VLMs) suffer from a critical failure in understanding negation, often referred to as affirmative bias. This limitation is particularly severe in described object detection (DOD) tasks. To address this, we propose two primary contributions: (1) a new dataset pipeline and (2) a novel, lightweight adaptation recipe. First, we introduce CoVAND, a dataset constructed with a systematic chain-of-thought (CoT) and VQA-based pipeline to generate high-quality, instance-grounded negation data. Second, we propose NegToMe, a novel text token merging module that directly tackles the architectural cause of affirmative bias. NegToMe fundamentally addresses the structural loss of negation cues in tokenization, grouping them with attributes into coherent semantic phrases. It maintains correct polarity at the input level, enabling robust negation understanding even with limited data. For instance, to prevent a model from treating the fragmented tokens "not" and "girl" as simply "girl", NegToMe binds them into a single token whose meaning is correctly distinguished from that of "girl" alone. This module is integrated with a parameter-efficient and strategic LoRA fine-tuning approach. Our method significantly improves performance on challenging negation benchmarks with a lowered false positive rate, boosting NMS-AP by up to +10.8 points on OVDEval and demonstrating generalization to SoTA VLMs. This work marks a crucial step forward in addressing negation understanding for real-world detection applications.