ObjEmbed: Towards Universal Multimodal Object Embeddings

AI summary

Overview

Research area: Computer vision and multimodal representation learning — specifically vision-language embedding models that align images, image regions, and text in a shared semantic space.

Technical level: Advanced. The paper assumes familiarity with contrastive image-text pre-training (CLIP-style), large multimodal language models (MLLMs), RoIAlign, sigmoid focal loss, and open-vocabulary object detection benchmarks.

Scope: The paper introduces ObjEmbed, an MLLM-based embedding model that produces one embedding per detected object (plus a global image embedding) in a single forward pass, unifying object detection, referring expression comprehension, local image retrieval, and global image retrieval in one framework.

What This Paper Is About

Existing multimodal embedding models are good at aligning a whole image with a whole caption, but they struggle to align a specific image region with a specific phrase. FG-CLIP improves regional alignment but its object embeddings do not explicitly model bounding-box quality, and open-vocabulary detectors localize well but generalize poorly due to limited training data.

ObjEmbed's goal is to build a single model whose embeddings capture both the semantics of an object and the localization quality of its bounding box, so that the model can be used for object-level tasks (detection, referring expression comprehension) and image-level tasks (local and global retrieval) without separate specialized architectures.

Key Contributions

1. An object-centric MLLM embedding model. ObjEmbed finetunes Qwen3-VL-Instruct and introduces five special tokens — ⟨object⟩, ⟨iou⟩, ⟨global⟩, ⟨local_text⟩, and ⟨global_text⟩ — whose final-layer hidden states serve as embeddings. Each object in an image, plus the full image, is encoded in a single forward pass.

2. A dual-token object representation. Each object is represented by two complementary tokens: an object token for semantic matching and a dedicated IoU token that regresses the box's IoU with ground truth. The final matching score is the product of the classification score (from object embeddings and local text embeddings) and the predicted IoU score.

3. A unified scoring scheme across tasks. Object detection and referring expression comprehension match text embeddings against object embeddings; local image retrieval uses the maximum object matching score across the image; global image retrieval uses the global image embedding directly.

4. A 1.3M-image training set with 8.1M bounding boxes, assembled from open-source detection and referring datasets plus 300k images curated from SA-1B and 200k self-crawled images annotated with WeDetect-Uni and captioned/described by Qwen3-VL-235B.

Main Findings

• Object detection is competitive with specialists. ObjEmbed-4B reaches 53.0% mAP on COCO (AP_s 35.6, AP_m 59.6, AP_l 72.2), 58.0 AP on COCO-O, 50.8 AP on ODinW13, and 40.2 / 42.2 / 34.4 AP on D3 (Full / Pres / Abs). ObjEmbed-2B scores 52.9 mAP on COCO. The paper notes that the untuned Qwen3-VL-2B baseline scores only 16.9 mAP on COCO, and Qwen3-VL-4B 29.1.

• Referring expression comprehension improves over larger models. ObjEmbed-4B reaches an average of 89.5 accuracy on RefCOCO/+/g (RefCOCO testA 94.3, RefCOCO+ testB 80.6, RefCOCOg test 90.7). ObjEmbed-2B averages 87.6. This exceeds InternVL3.5-38B (89.1 average) and ChatRex 7B (89.1 average) among the compared methods.

• Local image retrieval is where the gain is largest. ObjEmbed-4B scores 71.7 Recall@1 on SORCE-1K, 39.3 on REIRCOCO, 77.6 mAP@50 (T2I) and 85.3 (I2I) on ILIAS, for an average of 68.5. The paper states this surpasses existing global image embedding models by around 20 points. The strongest baseline listed, FG-CLIP2 ViT-So/16, averages 49.1.

• Global image retrieval is competitive despite modest training data. ObjEmbed-4B averages 81.7 points across long-caption (ShareGPT4V, DCI), short-caption (COCO, Flickr30K), and multilingual-caption (COCO-CN, Flickr30K-CN) benchmarks, compared with 81.5 for Qwen3-VL-Embedding-2B and 80.8 for FG-CLIP2 ViT-So/16.

• Decoupling semantics from localization helps. Ablation (Table 6) shows that switching the classification label to box IoU raises COCO mAP by 5.2%, and splitting into two tokens (one for classification, one for IoU regression) adds a further 3.2%. Placing the classification token before the IoU token is slightly better (45.5 vs 45.1 mAP).

• Task and object instructions matter. Adding an object separator instruction increases COCO mAP by 3.4% (Table 7); adding task instructions on top further raises COCO mAP from 45.5 to 47.1.

• Joint objectives are mutually beneficial. Combining object-level and image-level training raises local image retrieval by 2.7 points while keeping detection and REC performance stable, and improves global image retrieval by 0.4 points versus image-level training alone (Table 8).

• Two global image tokens with both caption types work best. In Table 9, the best configuration uses separate global tokens for short and long captions (Exp6: COCO 52.8, RefCOCO 87.4, local retrieval 68.6, global retrieval 81.6). Using only long captions drops global retrieval to 72.7.

• The model is robust to the proposal network. Performance is stable across 50/100/150 proposals and across WeDetect-Uni and UPN (Table 10). Global retrieval is unaffected by proposal quality.

• Oracle proposals reveal an upper bound. Mixing in ground-truth boxes raises AP by 12.2% on COCO and 17.6% on LVIS, indicating that overall performance is limited by proposal recall (WeDetect-Uni AR 66.7 on COCO, 50.8 on LVISv1 val) rather than by the embedding learning itself.

• Efficiency. Each object consumes only 8 tokens; with a full image encoded into 1000 tokens, the total sequence stays under 2000, enabling FlashAttention-2 acceleration. The paper reports that Table 12 covers box regression support but its contents are truncated in the provided text and are not reported here.

Methodology in Plain English

The team starts from a pretrained multimodal language model (Qwen3-VL-Instruct) and adds a small object projector plus five special tokens. Given an image, an off-the-shelf proposal generator (WeDetect-Uni) produces the top 100 region proposals. Each region's RoI feature is compressed into a single token and placed into the text sequence as Object i: ⟨object⟩ ⟨iou⟩, so all objects and the full image are processed together in one pass.

Rather than asking one token to do two jobs, the model separates them: the ⟨object⟩ token handles "what is this?", while the ⟨iou⟩ token handles "how good is this box?". A linear head turns the IoU token into a predicted quality score, and this score is multiplied by the classification similarity to produce the final match. This means a confident but poorly localized box gets down-weighted.

Training uses three losses combined with weights (1.0, 1.0, 0.25): a region-level sigmoid focal loss matching object descriptions to proposals (a proposal counts as positive if its IoU with a ground-truth box exceeds 0.5), an image-level sigmoid focal loss matching images to short and long captions, and an IoU regression loss applied only to positive proposals. Text is encoded by the same LLM backbone as the visual tokens, using separate ⟨local_text⟩ and ⟨global_text⟩ tokens so that object-matching and image-matching do not interfere.

Training used 16 GPUs, a batch size of 2 images per GPU, a learning rate of 2e-5, two epochs, and a frozen vision encoder. Images were resized so the visual token count ranged from 900 to 1200.

Why This Matters

Impact on research. ObjEmbed proposes that an embedding model can carry localization awareness by construction rather than bolting a detector on top of a contrastive encoder. It also shows that a single model can serve region-level and image-level retrieval simultaneously without sacrificing either, which challenges the common split between "embedding models" and "detectors." The oracle experiment gives a clear diagnostic: the bottleneck is proposal recall, not representation quality, pointing follow-up work toward the proposal stage.

Real-world applications (as named in the paper):

• Autonomous driving, where distant traffic signs are small and easy to miss in a global image representation.
• Robotics, where manipulating small parts requires fine-grained object retrieval and precise localization.
• Digital content safety moderation, which needs region-level understanding of what is in an image.
• More generally, any retrieval system where the query describes an object occupying a small part of a larger image.

Industry relevance. The model is trained on 1.3M images with a batch size of 2 per GPU on 16 GPUs, so it is within reach of a well-resourced industry lab rather than requiring web-scale compute. The code is released at https://github.com/WeChatCV/ObjEmbed. The efficiency design — 8 tokens per object, all objects in one forward pass, no autoregressive decoding — matters for production search and moderation pipelines. Authors are affiliated with WeChat Vision, Tencent Inc. and Sun Yat-sen University.

Future Directions

1. Closing the proposal-recall gap. The oracle experiment shows a 12.2% AP gain on COCO and 17.6% on LVIS from ground-truth boxes; the paper suggests fine-tuning the proposal network on target datasets or using human-annotated boxes as ways to recover this.

2. Box regression support. Table 12 addresses "supporting box regression," but the content is not available in the provided text, so what that experiment concludes cannot be summarized here. Extending from box quality prediction to full box refinement is a natural next step.

3. Scaling the training data. The paper repeatedly notes ObjEmbed uses "a relatively small-scale training set" compared with large-scale contrastive models, and still performs competitively. It also explicitly hopes ObjEmbed can serve as a general-purpose baseline, implying larger-scale training as an open opportunity.

4. Broader task coverage. ObjEmbed currently supports detection, REC, local retrieval, and global retrieval through a fixed set of templates and task instructions. Whether the same object-centric embeddings transfer to tasks such as segmentation, tracking, or video object retrieval is not addressed.

Target Audience

Researchers and engineers working on multimodal embedding models, open-vocabulary detection, referring expression comprehension, and image retrieval. It is most useful to readers already comfortable with vision-language architectures and detection benchmarks who want to understand how to make a single MLLM-based model serve both object-level and image-level tasks. Practitioners building production search, moderation, or robotics perception systems will also find the efficiency and proposal-recall analysis directly actionable.

Authors’ abstract

Aligning objects with corresponding textual descriptions is a fundamental challenge and a realistic requirement in vision-language understanding. While recent multimodal embedding models excel at global image-text alignment, they often struggle with fine-grained alignment between image regions and specific phrases. In this work, we present ObjEmbed, a novel MLLM embedding model that decomposes the input image into multiple regional embeddings, each corresponding to an individual object, along with global embeddings. It supports a wide range of visual understanding tasks like visual grounding, local image retrieval, and global image retrieval. ObjEmbed enjoys three key properties: (1) Object-Oriented Representation: It captures both semantic and spatial aspects of objects by generating two complementary embeddings for each region: an object embedding for semantic matching and an IoU embedding that predicts localization quality. The final object matching score combines semantic similarity with the predicted IoU, enabling more accurate retrieval. (2) Versatility: It seamlessly handles both region-level and image-level tasks. (3) Efficient Encoding: All objects in an image, along with the full image, are encoded in a single forward pass for high efficiency. Superior performance on 18 diverse benchmarks demonstrates its strong semantic discrimination.