Jag: Direct Box Prediction for Efficient Visual Grounding

AI summary

Overview

Research area: Computer vision and multimodal machine learning — specifically visual grounding, where a model must locate the object described by a natural-language referring expression by returning a bounding box.

Technical level: Intermediate. The paper assumes familiarity with multimodal language models, bounding-box regression, and standard grounding benchmarks, but its central idea is conceptually simple and explained clearly.

Scope: The paper introduces Jag, a method that adapts a compact pretrained multimodal model (Qwen3.5-0.8B) to predict an entire bounding box in a single forward pass instead of generating coordinates as text, and evaluates its accuracy, latency, and memory cost against the base model and specialized grounding models.

What This Paper Is About

Most multimodal models handle visual grounding by writing out a box as a sequence of coordinate numbers, which are then parsed from text — an approach that adds sequential decoding work on top of the image–language processing the task already requires. Jag instead treats the box as a geometric answer: it reads a hidden state the model already produces from the image and the referring expression, and maps it directly to four continuous numbers describing the box. The goal is to keep the pretrained image–language machinery intact while making localization accurate and fast.

Key Contributions

1. Direct continuous box prediction from an existing input state. Jag adapts Qwen3.5-0.8B so that a lightweight MLP regression head reads the last valid input-token state and predicts all four box coordinates (center x, center y, width, height) at once, with no dedicated regression token and no coordinate generation or text parsing.

2. Geometric supervision with joint adaptation. L1 and GIoU losses are used not only to fit the new head but also to update the language backbone (including input embeddings) and the visual merger, so localization losses shape the multimodal representations rather than only the coordinate mapping.

3. Ablation evidence that adaptation, not just a new head, drives accuracy. Component ablations show that jointly adapting the backbone and head substantially outperforms training the regression head alone.

4. Accuracy–efficiency characterization against specialized models. Jag is compared to its base model, NExT-Chat, LocateAnything, and Hi-Token on three grounding benchmarks and on single-request latency and peak GPU memory.

Main Findings

• Consistent gains over the base model on all five test splits. RefCOCO testA rises from 84.27 to 93.90, testB from 74.72 to 88.36, RefCOCO+ testA from 76.53 to 90.69, RefCOCO+ testB from 62.57 to 80.08, and RefCOCOg test from 77.96 to 87.72 (Acc@0.5). The absolute gains are +9.63, +13.64, +14.16, +17.51, and +9.76 respectively, with the largest improvements on the initially weaker testB splits and on RefCOCO+.

• Higher accuracy than NExT-Chat on every split. Jag beats NExT-Chat (89.66 / 77.04 / 83.76 / 66.19 / 79.28) by 4.24 to 13.89 accuracy points, with the largest gap on RefCOCO+ testB — notable because NExT-Chat also regresses continuous boxes through an MLP.

• Approximately on par with LocateAnything overall, better on three of five splits. Jag is highest on both RefCOCO+ splits and RefCOCO testA; LocateAnything is higher on RefCOCO testB (89.26 vs. 88.36) and RefCOCOg (88.54 vs. 87.72). Pooled across all test expressions, Jag reaches 88.30% versus 88.00% for LocateAnything and 79.57% for NExT-Chat.

• Lowest measured single-request latency. Jag runs at 65.97 ms mean end-to-end latency: 19.60× faster than Base, 1.82× faster than NExT-Chat, 3.62× faster than LocateAnything, and 9.35× faster than Hi-Token. A separate within-backbone comparison gives an 8.90× speedup over the fine-tuned autoregressive model.

• Small memory footprint. Peak process memory is 2.52 GiB — 84.22% lower than NExT-Chat, 73.88% lower than LocateAnything, and 68.10% lower than Hi-Token.

• Joint adaptation matters far more than head-only training. Component ablations report that updating the language backbone, visual merger, and regression head together improves Acc@0.5 by more than 60 points on each dataset relative to training the head alone in that setting.

• Readout and schedule choices are secondary. Averaging referring-expression token states substantially reduces accuracy, while adding a dedicated input marker stays within one point of the default on all three datasets. Immediate joint training (skipping the head-only stage) also stays within one point of the default, with no consistent advantage for the two-stage schedule.

• Continuous regression beats parallel discrete classification. Using a shared backbone and readout over multiple random seeds, continuous regression scores 89.57 / 83.22 / 85.12 versus 73.52 / 68.13 / 68.04 for four independent categorical heads on RefCOCO / RefCOCO+ / RefCOCOg, and 90.02 / 84.57 / 86.41 for the fine-tuned autoregressive (AR) baseline on the same subset.

• Close to autoregressive prediction at full data, weaker at strict thresholds. On the full training set, Jag is within one point of AR everywhere (91.28 vs. 90.93 on RefCOCO, 85.80 vs. 85.82 on RefCOCO+, 87.72 vs. 88.48 on RefCOCOg). Paired 95% intervals include zero on RefCOCO and RefCOCO+, while RefCOCOg favors AR (interval [−1.41, −0.13]). At IoU 0.75 and 0.9, AR is more accurate on all three datasets, so the gap is mainly in precise box placement rather than in the standard 0.5 threshold.

• Gains hold across target sizes, expression lengths, and spatial keywords. Every nonempty subgroup improves over Base. On RefCOCOg, medium-area targets go from 70.96% to 82.16%, a gain 2.12 points larger than on large targets (interval [+0.05, +4.26]). On RefCOCO+, expressions containing spatial keywords gain 20.06 points versus 15.09 without them, a contrast of +4.97 points ([+1.95, +8.10]). Expression length shows no consistent gradient.

• Errors are reduced where the base model fails outright. Net increases over Base are 1,240, 1,667, and 937 correct expressions on RefCOCO, RefCOCO+, and RefCOCOg, compared with 53, 111, and 78 invalid outputs produced by Base; Jag produces none.

• Larger training budgets keep helping. The smallest subset already exceeds Base on all three benchmarks, and RefCOCO+ climbs from 80.33% to 85.80% at the full budget (the paper notes this combines more examples and more optimizer updates).

Methodology in Plain English

Starting point. The authors take an off-the-shelf compact multimodal model, Qwen3.5-0.8B, and leave its vision encoder, visual merger, and language backbone architecture unchanged. The image is encoded, its features are mapped into the language model's input space, and the model processes the resulting sequence of visual and text tokens exactly as it would for any other prompt.

Where the box comes from. Rather than asking the model to write coordinates, they locate the last valid position in the input sequence using the attention mask. Because that position comes after all the visual and text tokens, its hidden state can already reflect both the image and the referring expression. A small regression head — layer normalization, one hidden projection with GELU, and a four-number output passed through a sigmoid — converts that single vector into a box. Prediction bypasses the model's vocabulary output entirely.

How the box is represented. Boxes are stored as normalized center coordinates plus width and height rather than corner coordinates, which keeps all four values in a fixed [0, 1] range. They are converted back to corners inside the loss and again at inference, with clipping to the image applied only at the very end.

How it is trained. Two losses are combined: an L1 loss on the four numbers, and a GIoU loss that measures the geometric overlap and enclosure of the predicted and ground-truth rectangles, computed on unclipped boxes so out-of-image extent is still penalized. Training runs in two stages with the same objective: first the backbone is frozen and only the new head is fitted, then the head is trained jointly with the language backbone (including input embeddings) and the visual merger, while the vision encoder stays frozen. The head has 528,900 parameters for this backbone.

How it is compared. Evaluation uses the official RefCOCO, RefCOCO+, and RefCOCOg splits with Acc@0.5 (percentage of predictions with IoU at least 0.5). Training pools all three training partitions, totaling 321,327 referring expressions over 28,158 images, with 30,969 test expressions. Baselines include the untouched base model, the released NExT-Chat-7B-336, the released LocateAnything-3B, and Hi-Token for inference cost. Confidence intervals come from paired image-cluster resampling with 2,000 draws, and a multi-seed comparison covers the output-formulation ablation.

Why This Matters

Impact on research. The paper isolates a specific design question in multimodal grounding: whether the box should be produced as language or as geometry. Its ablations give concrete evidence that the representation itself must be adapted for localization, not just the output head, and that continuous regression over simultaneous discrete classification is the stronger formulation on these benchmarks. It also documents an accuracy trade-off — direct prediction is close to autoregressive decoding at IoU 0.5 but behind at IoU 0.75 and 0.9 — which frames precise localization as a separate problem from coarse grounding.

Real-world applications:

• Interactive assistants and accessibility tools where a user points at an object by description and needs a fast visual response.
• Robotics and embodied agents that need a target region quickly enough to act on it, rather than after a long text generation.
• Image editing and annotation tools that convert a typed phrase into a selectable region, including semi-automatic dataset labeling.
• Content moderation or retrieval pipelines that must localize described content in large image collections under tight per-request budgets.

Industry relevance. The reported combination of competitive accuracy, 65.97 ms latency, and 2.52 GiB peak memory points to deployment on modest hardware, where per-request cost and memory — not just leaderboard accuracy — determine feasibility. Removing coordinate-token generation and output parsing also removes a brittle text-parsing step from the serving pipeline.

Future Directions

• Extending beyond one object in one static image. The authors explicitly note that multiple referents, video, and segmentation require task-specific design and evaluation, and are not addressed here.
• Closing the strict-IoU gap. Since AR is more accurate at IoU 0.75 and 0.9 on all three datasets, follow-up work could target precise box placement while retaining single-pass inference.
• Revisiting the two-stage schedule. Immediate joint training performed within one point of the staged default, raising the question of whether the head-only warm-up stage is needed at all, and whether different adaptation strategies for the backbone do better.
• Domain transfer and deployment profiling. The paper calls for evaluating specialized domains on representative data, and notes that latency depends on input resolution, expression length, hardware, and serving configuration — so reported speedups should be re-measured under intended operating conditions.

Target Audience

Researchers and engineers working on visual grounding, referring expression comprehension, or multimodal model adaptation will get the most from this paper, particularly those interested in replacing autoregressive coordinate output with direct regression. It is also useful for practitioners choosing a grounding model under latency and memory constraints, and for readers tracking how task-specific supervision can specialize compact pretrained multimodal backbones.

Authors’ abstract

Visual grounding combines image--language understanding with geometric prediction, yet autoregressive multimodal models often express a box as a coordinate sequence. Sequential decoding adds computation to the multimodal processing needed for localization. Adapting these representations to box geometry offers a route to direct prediction. We present Jag, which adapts a compact pretrained multimodal model for direct continuous box prediction. A lightweight regression head reads the last valid input-token state, conditioned on the image and referring expression, and predicts all four box coordinates together. Geometric supervision jointly adapts the language backbone, visual merger, and regression head, allowing localization losses to shape both the multimodal representations and the coordinate mapping. Inference retains image--language processing and returns the box in one forward pass, without coordinate generation or text parsing. Across benchmarks, Jag consistently improves grounding accuracy over its base model across the observed target sizes and expression groups. Component ablation show that jointly adapting the representations and head substantially improves accuracy over training the head alone. Against existing specialized grounding models, it achieves competitive accuracy with lower single-request latency and GPU memory use. These results show how supervision tailored to the geometric answer can specialize a compact pretrained multimodal model for accurate grounding and efficient direct prediction. Project page: https://xyzzzh.github.io/Jag/