WeVisOne-VL: Unified Visual Perception from Objects to Pixels

AI summary

Overview

Research area: Computer Vision — unified vision-language models (VLMs) for fine-grained visual perception (detection, referring localization, counting, OCR, segmentation, and matting).

Technical level: Advanced. The paper assumes familiarity with vision-language model training, structured token prediction, deformable-attention decoders, and reinforcement learning (GRPO).

Scope: A technical report from WeChat Vision, Tencent Inc., describing WeVisOne-VL, a family of 2B and 4B vision-language models built on Qwen3-VL that unifies object-level localization and pixel-level prediction (masks and alpha mattes) in a single model, together with the data engine and three-stage training curriculum used to build it.

What This Paper Is About

Existing vision-language models remain unreliable for fine-grained visual perception: they hallucinate objects that are not present, misread unfamiliar referring expressions, and miss non-salient or densely packed instances, leading to incomplete recall. WeVisOne-VL addresses this by casting a wide range of visual tasks as structured next-token prediction — generating labels and coordinates as text — and then attaching a small pixel decoder so the same model can also emit masks and alpha mattes at any user-specified granularity (points, boxes, masks, mattes). The goal is a single deployable model that is both accurate at object-level grounding and capable of dense pixel output, without relying on a separate large segmentation model.

Key Contributions

1. A unified object-to-pixel task formulation. All tasks are expressed as structured next-token prediction using a dedicated coordinate vocabulary (<0> to <1000> for normalized coordinates) plus a special <OBJ> token emitted after each predicted box. The hidden state of <OBJ> acts as an instance-level latent representation that the pixel decoder consumes. Absent targets are indicated by filling the coordinate field with None.

2. A lightweight pixel decoder (~9.6M parameters). Integrated into Qwen3-VL, it consists of a visual feature adapter that organizes multi-level ViT features into a shared memory, a deformable query decoder (six deformable blocks) initialized from the <OBJ> hidden state and predicted box, and an instance-conditioned dense prediction stage. Segmentation and matting share these components, with matting adding a refinement module for fine boundaries and fractional opacity.

3. A unified data engine (WDE). An annotation inventory of over 100 million images and approximately 3.6 billion bounding-box annotations, drawing on GRIT, CC3M, LAION, SA-1B, and task-specific datasets. It organizes category concepts and referring expressions in a shared knowledge graph, combines complementary localization experts (SAM3, an in-house DINO-style open-vocabulary detector, Rex-Omni) with Qwen3-VL verification, extends boxes to masks via SAM2, builds hard positives and hard negatives from reviewed model errors, and runs a dedicated matting pipeline (BiRefNet and MatAnyone2 candidates plus a trained annotator built on a frozen SAM2.1-Base+ with a new alpha refinement head).

4. A training recipe and an open release. Three-stage curriculum supervised fine-tuning followed by mixed-task reinforcement learning with GRPO, plus public release of code, weights, and a project page.

Main Findings

• Object detection leads among VLMs at comparable scale. In Table 2, WeVisOne-VL-4B reaches mean F1 of 57.1 on COCO, 55.3 on LVIS, 71.3 on Dense200, and 44.7 on VisDrone. The 2B model reaches 55.2, 53.3, 61.4, and 40.4 respectively. For comparison, Rex-Omni-3B scores 52.9 / 46.9 / 58.3 / 35.8 (COCO / LVIS / Dense200 / VisDrone mean F1) and LocateAnything-3B scores 54.7 / 50.7 / 58.7 / 39.9.

• Large gains on dense and long-tailed detection. On Dense200 mean F1, WeVisOne-VL-4B reaches 86.9 at IoU 0.5 and 71.3 mean, versus 78.4 / 58.3 for Rex-Omni-3B and 74.0 / 58.7 for LocateAnything-3B. On LVIS, mean F1 rises to 55.3 (4B) from 44.0 for Qwen3-VL-4B.

• Competitive with a specialized detector. GroundingDINO-T scores 56.6 mean F1 on COCO against 57.1 for WeVisOne-VL-4B, but only 38.8 on LVIS, 33.1 on Dense200, and 38.5 on VisDrone, versus 55.3, 71.3, and 44.7 for WeVisOne-VL-4B.

• Small models also benefit. WeVisOne-VL-2B exceeds all listed Qwen3-VL and Qwen3.5 baselines on every reported mean F1 column except COCO, where Qwen3.5-9B reports 49.5 and WeVisOne-VL-2B reports 55.2 (i.e., the 2B model leads there as well).

• Rejection behavior is explicitly trained. Absent targets are represented by None, hard negatives are constructed from reviewed model errors where no object satisfies all query conditions, and in RL a negative sample earns reward 1 only when the model predicts no object.

• RL is applied to boxes, not masks. For segmentation and matting, the paper states RL uses only box-based rewards, without evaluating predicted masks or alpha mattes; the grounding reward is the sum of cubed IoU over Hungarian-matched pairs, normalized by the number of ground-truth plus predicted boxes, with G = 8 sampled responses.

• Scope of evaluation. The abstract reports leading performance on both object-level and pixel-level tasks across over 30 benchmarks. The provided content includes only the object detection table (COCO, LVIS, Dense200, VisDrone); the pixel-level and remaining task results are not included in the excerpt, so specific numbers for counting, pointing, OCR, referring segmentation, and matting are not reported here.

Methodology in Plain English

Building the training data. Instead of scraping and labeling images task by task, the authors built a pipeline. First, they organise object categories and referring expressions (phrases like "the red car on the left") in a single knowledge graph seeded from Wikipedia and WordNet, and use it to decide what images to collect. For detection, two different detectors propose candidate boxes, and a vision-language model (Qwen3-VL) checks each candidate against the query, resolving conflicts and removing duplicates. For referring expressions, SAM3 and Rex-Omni propose regions, and Qwen3-VL again verifies whether each region actually satisfies the attributes, states, actions, and relations in the phrase. Verified boxes are then turned into binary masks with SAM2, so each instance carries a box and a mask under the same query. Hard examples are built deliberately: a query becomes a hard negative only after a human-style review of the whole image confirms no object matches, and only when the current model wrongly predicts something for it.

Matting data. Binary masks lack soft edges. For each instance, the pipeline produces two candidate alpha mattes (from BiRefNet and MatAnyone2) and accepts the BiRefNet one only if a set of geometric checks pass (for example IoU with the binary mask ≥ 0.5, foreground precision ≥ 0.8, area ratio in [0.5, 2], box IoU ≥ 0.7, and a centroid offset no greater than 0.15 times the box diagonal). A dedicated matting annotator — a frozen SAM2.1-Base+ with a trainable alpha refinement head — is then trained and applied to re-annotate the whole corpus.

Model. An image and instruction go into Qwen3-VL. The model writes out an answer as text: object labels plus coordinate tokens, and an <OBJ> token after each box for pixel tasks. The small decoder reads the <OBJ> hidden state, the predicted box, and multi-level ViT features, and produces a full-image mask or alpha matte — fusing coarse and fine features, using the instance query to generate a per-instance "kernel" applied to the pixel features, then refining the mask over three upsampling stages.

Training. Three supervised stages: (1) task alignment — the vision encoder and projector are frozen and only the language model is trained on 90M examples for 13K steps, with samples capped at 50 boxes or points per category to avoid repetitive output collapse; (2) resolution and density scaling — everything is unfrozen, input resolution rises to 1,960 visual tokens, up to 500 boxes/points per response, context extended to 8,192 tokens, roughly 40M examples for 9K steps; (3) alternating token–pixel optimization — the decoder is first warmed up on 250M segmentation samples for 100K steps, then the VLM and decoder are jointly trained for 9,000 steps alternating text and pixel tasks 1:1, then the decoder alone is fine-tuned on 130M matting samples for 100K steps.

Reinforcement learning. Because token-level likelihood does not directly optimise task quality, the authors score candidate prompts by uncertainty, 4·μ·(1−μ)·σ over sampled rewards, keeping moderately difficult prompts with high reward variance. They select about 140K such samples, supplement with box-supervised and pointing data to 178K total, and run GRPO with grounding rewards computed on complete parsed responses.

Why This Matters

Research impact. The paper argues that a single VLM can replace the usual stack of a grounding VLM plus a separate large segmentation model, by treating segmentation and matting as a by-product of the same autoregressive output. The data engine's approach — using detector variation for recall, a VLM for verification, and reviewed model errors as a data-acquisition signal — is a reusable recipe for building fine-grained supervision at scale.

Real-world applications (as named in the paper):

• Content creation
• Industrial inspection
• Autonomous driving
• Embodied intelligence

Industry relevance. The reported strengths map onto deployment concerns the paper highlights: high recall in crowded scenes (Dense200, VisDrone), long-tail categories (LVIS), and correctly rejecting queries when the target is absent — a common failure mode that erodes trust in production systems. The model is released at 2B and 4B scales with open weights, code, and a project page, making it practical for teams that cannot serve a much larger model.

Future Directions

• Closing the RL gap for dense outputs. The paper explicitly limits RL to box-based rewards and does not evaluate predicted masks or alpha mattes during reinforcement learning; extending the reward to pixel quality is a natural open question.
• Scaling beyond 2B and 4B. Only two model sizes are instantiated in this report, so how the unified formulation behaves at larger and smaller scales is not established here.
• Harder matting and boundary fidelity. The matting pipeline relies on accepted expert candidates and a trained annotator with heuristics (fixed thresholds, fallbacks to the binary mask when crops are small or inference fails); improving alpha quality where those fallbacks trigger remains open.
• Sustaining the data flywheel. The engine assumes each training round's reviewed errors feed back into acquisition and annotation; how this loop behaves over many iterations, and whether it saturates, is not reported in the provided content.

Target Audience

Researchers and engineers working on vision-language models, visual grounding, open-world detection and segmentation, and referring segmentation; applied teams in industry who need a single deployable model for mixed localization and pixel tasks; and practitioners interested in data-engine design and curriculum-plus-RL training recipes for multimodal models. Readers seeking a beginner-level introduction to VLMs will find the paper dense — prior familiarity with Qwen-style architectures, Deformable DETR decoding, and GRPO is assumed throughout.

Authors’ abstract

Existing vision-language models (VLMs) remain unreliable for fine-grained visual perception. In this report, we present WeVisOne-VL (WeChat Vision All-in-One), a powerful VLM that unifies object-level and pixel-level perception through structured next-token prediction. It supports multiple visual tasks including object detection, referring localization, counting, OCR, segmentation, and matting, enabling localization at any user-specific granularity, from points and boxes to masks and alpha mattes. Architecturally, we integrate a lightweight pixel decoder into the VLM via a special <OBJ> token to generate instance-level pixel outputs. We train the model with a three-stage curriculum SFT followed by mixed-task RL. Across over 30 benchmarks, WeVisOne-VL demonstrates leading performance on both object-level and pixel-level tasks, while maintaining robust rejection for object-absent queries. By releasing WeVisOne-VL to the open-source community, we hope to advance VLMs from broad visual understanding toward truly reliable perception systems for real-world deployment.