MJev-Spatial: System One Spatial Intelligence via Native Attention Matching

AI summary

Overview

Research area: Computer vision and multimodal spatial reasoning — specifically, converting vision-language models (VLMs) from text-generating answerers into fast, single-pass classifiers for spatial questions.

Technical level: Advanced. The paper assumes familiarity with transformer attention (queries/keys), autoregressive decoding, cross-entropy training, and distributed batching. The core idea is explainable to a newcomer, but the architecture and training recipe sections are specialized.

Scope in one sentence: The paper proposes MJev-Spatial, a "System 1" spatial decision model built on Qwen3.5-4B that answers choice, numerical, and pointing questions by classifying over finite candidate sets with the backbone's own pretrained attention projections, and reports accuracy, latency, and training-throughput results across twelve spatial benchmarks.

What This Paper Is About

Spatial questions asked of a VLM — which relation holds, how far away something is, where to place an object — have small answer spaces (one of a few options, a scalar, a 2D point), yet standard VLMs answer them by generating free-form text and then parsing that text back into an answer. That costs sequential decoding and answer parsing on every query, which is a poor fit for judgments that need to happen at interaction rates. The paper's goal is to make these three spatial output types into a single classification problem, so a spatial answer costs one forward pass and zero generated tokens, while still matching or beating autoregressive models on accuracy.

Key Contributions

1. A unified System 1 formulation for spatial intelligence. Option selection, numerical estimation, and image pointing are all cast as classification over finite candidate sets — supplied options for choice tasks, discretized value bins for numbers, and image regions for points — so one scorer handles all three output types.

2. Native attention matching as the classifier. Instead of adding a prediction head, MJev-Spatial scores each candidate using the final full-attention layer's pretrained query and key projections (the decision marker supplies the query, each candidate's option-end marker supplies a key, averaged over 16 query heads with d = 256 and one key head shared per four query heads). No new parameters are introduced.

3. Coarse-to-fine refinement plus a multi-point extension. Numerical tasks use up to two classification stages; single-point prediction uses three 9-way region decisions with crops re-encoded from original pixels between stages (an effective 27×27 grid). Multi-point prediction expands multiple branches of the same trained tree, requiring no additional training or learned stop decision.

4. An efficient training recipe and data pipeline. All refinement stages of a question are supervised jointly with teacher forcing in one forward pass, with per-question loss averaged over stages; length-aware global batching and cross-GPU load balancing raise training throughput by 30.8% over the original batching strategy. Training uses 12.3M question–answer pairs converted from existing spatial datasets.

Main Findings

• Highest overall score among eight models: 65.64%. MJev-Spatial beats the next-best model, Qwen3-VL-8B-Instruct at 62.79%, and ranks first on seven of the twelve benchmarks.

• It beats its own base model everywhere. Against unmodified Qwen3.5-4B (54.80% overall), MJev-Spatial improves on all twelve benchmarks, including +22.89 percentage points on RoboSpatial VQA and +12.50 points on RefSpatial-Bench.

• Lowest mean query latency: 160.6 ms on an NVIDIA B200 GPU. That is a 2.5× speedup over the fastest autoregressive baseline and a 5.6× speedup over the base model, giving the best accuracy–latency trade-off in the comparison.

• Length estimation reaches 0.103 m MAE on the VST-300 development set with the final model.

• Coarser discretization beat finer discretization. Cutting positive leaf cells from 960 (32 fine bins) to 216 (4/8 bins) raised quantization MAE from 0.0391 to 0.1540 but lowered prediction MAE from 0.3277 to 0.3202 m — finer representational precision did not translate into better learned predictions.

• Argmax decoding beat conditional expectation. With the same 216-cell model, argmax gave 0.3202 m MAE versus 0.3300 m for conditional expectation (AbsRel 29.03% vs 30.04%).

• Crop re-encoding is essential for pointing. Removing crops from the trained model collapsed RefSpatial accuracy from 51.00% to 9.00%; retraining without crops recovered only 46.50%. Crop re-encoding was the best of four visual strategies (51.00% RefSpatial, 56.84% Point-Bench), ahead of original-view context, ViT token replay, and KV replay.

• Native Q/K scoring is competitive and cheapest. It gave the best RefSpatial accuracy (51.00%) and lowest length MAE (0.3341 m), while a linear classifier was better on Where2Place (67.99) and RoboSpatial pointing (65.57), and KEV was better on RoboSpatial VQA (68.54). The authors chose native Q/K for minimal architectural change, fewest trainable parameters, and lowest measured training cost.

• Probability-tree decoding improves multi-point counting. Replacing repeated continue/stop decisions raised Point-Bench counting success from 19.39% to 38.78%; root pruning kept that rate while cutting decision nodes from 91 to 51.10 and inference time from 4.716 s to 2.638 s per question (1.79× speedup). A 10% peak threshold (38.78%) outperformed 5% (30.10%) and 20% (37.24%).

• Batching changes pay off. On a fixed 72K-QA corpus with global batch 256 and eight L20D GPUs, original ordering reached 99.06 QA/s at 35.17% padding; global balanced grouping reached 129.54 QA/s at 1.77% padding — the reported 30.78% throughput gain — while global grouping with a buffer of 128 reached 129.39 QA/s at 1.03% padding but with a 19.77% step coefficient of variation.

Methodology in Plain English

The team starts from a simple observation: spatial answers are small. A multiple-choice question has a handful of options; a distance can be binned into intervals; a point can be narrowed down to a region. So instead of asking the model to write an answer, they write out every possible answer as text, let the model read those candidate descriptions alongside the image and question, and then ask which candidate the model's internal representations best match.

Concretely, each candidate description ends with an end marker, and the whole list ends with a decision marker. The model reads the image, question, any previous decisions, and all candidates in one causal pass. The hidden state at the decision marker becomes a "query," and the hidden state at each candidate's end marker becomes a "key." The score for a candidate is the average scaled dot product between them, using the model's existing pretrained projection weights after 31 language layers of Qwen3.5-4B. A softmax over these scores picks the winner. Nothing new is added to the network, and no answer text is generated.

For numbers and points, one decision is not precise enough, so they subdivide and ask again. A length is first placed in one of 30 logarithmically spaced intervals up to 16384 m (plus exact-zero and overflow classes), then narrowed into 4 or 8 linear cells; counts use two 32-way decisions to cover 0–1023 exactly; angles use twelve 30-degree intervals with 32 cells each. For pointing, three 9-way decisions over a 3×3 grid give a 27×27 effective grid, and after each of the first two choices the selected rectangle is cropped from the original image and re-encoded so the next decision sees local detail.

Training is done in one pass per question by exploiting teacher forcing: the ground-truth answer fixes the entire target path and any crops ahead of time, so all stages can be laid out in causal order and all their losses computed together. Each question's stage losses are averaged so a three-stage point example counts the same as a one-stage choice example. Because examples differ wildly in length and crop count, sequences are sorted by length within each global batch and distributed across GPUs with a longest-work-first rule, cutting padding and synchronization stalls. The language backbone is fully fine-tuned while the vision encoder and merger stay frozen.

Why This Matters

Impact on research. The paper challenges a default assumption in multimodal work — that spatial answers must be produced generatively. It shows a compact, parameter-free classification readout can beat the same model's autoregressive behavior on every benchmark tested, and it reports honest negative results (finer codebooks hurting accuracy, extra heads not consistently winning), which is useful for anyone designing decision interfaces or grounded interfaces for VLMs.

Real-world applications:

• Robotics manipulation and navigation, where an agent needs "where do I place this?" or "how far is that obstacle?" answered within a control loop rather than after a text-generation delay.
• Augmented reality, where a phone or headset must place, anchor, or annotate virtual content on a real surface at interactive latency.
• Warehouse and logistics automation, where counting objects and locating them in an image feed must keep pace with conveyor-rate throughput.
• Assistive technology for visually impaired users, where a scene question needs a spoken answer in well under a second.

Industry relevance. The speedup matters commercially: 160.6 ms mean latency and 2.5×–5.6× speedups translate into fewer GPUs per query stream, and the 30.8% training-throughput gain lowers the cost of retraining on new spatial data. Because the approach adds no parameters and reuses the base checkpoint's projections, it is an attractive retrofit for teams already serving a Qwen-family model.

Future Directions

• Generalizing beyond the three output types. The framework works because each task's answers can be enumerated; it is unclear how it would extend to outputs without a natural finite candidate set, or to quantities the adapter currently rejects (unsupported units, ambiguous frames).
• Closing the remaining gaps. MJev-Spatial loses to a linear classifier on Where2Place (59.98 vs 67.99) and to KEV on RoboSpatial VQA (68.06 vs 68.54), so the question of when a learned head is worth the extra parameters is left open rather than settled.
• Better multi-point and counting decoding. The probability-tree extension currently comes with a 10% peak threshold tuned empirically (5% and 20% both score lower) and heavy expansion cost before pruning; adaptive thresholds or learned grouping are natural next steps.
• Scaling and backbone transfer. All results use Qwen3.5-4B; whether native attention matching holds at larger scales, on other backbones, or combined with the parameter-efficient tuning methods the paper cites as applicable is not reported.

Target Audience

Researchers and engineers working on multimodal models, embodied AI, and robotics perception who care about inference latency as much as benchmark accuracy. It is also valuable for practitioners building grounded or "decision-interface" systems on top of pretrained VLMs, and for graduate students studying how attention projections can be repurposed as classifiers. Readers wanting a first look at spatial VLM benchmarks will find the twelve-benchmark comparison table useful, but should expect to consult the appendices for sequence layout and decoding rules.

Authors’ abstract

Spatial intelligence lets an agent perceive and act in the physical world, and many of its judgments must be made at interaction rates: which spatial relation holds, how far an object is, or where to place it. Vision-language models (VLMs) produce these answers by free-form text generation, paying for autoregressive decoding and answer parsing even though the output space is compact: one of a few options, a scalar, or a 2D point. In light of this, we introduce MJev-Spatial, a System 1 multimodal decision model that tackles all three types of spatial understanding tasks within a unified classification framework. Specifically, MJev-Spatial treats each candidate as a class and scores it using the backbone's pretrained query and key projections, without adding parameters. The candidates are the given options for choice tasks, or value bins and image regions refined through coarse-to-fine classification for numerical and point prediction, respectively. At inference, each classification decision takes a single forward pass and generates no answer tokens. To train on heterogeneous spatial data efficiently, we supervise all coarse-to-fine stages jointly with teacher forcing in one forward pass, and combine this with length-aware global batching and cross-GPU load balancing, raising training throughput by 30.8% over the original batching strategy. Built on Qwen3.5-4B and trained on 12.3M question-answer pairs converted from existing spatial datasets, MJev-Spatial achieves the highest overall score (65.64%) among eight tested models, ranks first on seven of twelve benchmarks, and outperforms its unmodified autoregressive base model on all twelve, while running 2.5 times faster than the fastest autoregressive baseline and 5.6 times faster than the base model.