WinARDet: Compressed Window Auto-Regression for Long-Horizon Vision-Language Grounding

AI summary

Overview

Research area: Computer vision / multimodal large language models (MLLMs) — specifically autoregressive object detection and dense vision-language grounding.

Technical level: Advanced. The paper assumes familiarity with autoregressive decoding, KV caching, attention sinks, and coordinate-token serialization used by MLLM detectors.

Scope: The paper diagnoses why MLLM-based detectors collapse into repeating the same bounding boxes in dense scenes, and proposes a windowed, compressed-history autoregressive decoding scheme (WinAR) that fixes this across multiple decoders, decoding modes, and output formats.

What This Paper Is About

MLLM-based detectors write out object locations as sequences of coordinate tokens, one after another. In crowded images with hundreds or thousands of objects, these models eventually start emitting the same box over and over until the decoding budget runs out — wasting the remaining budget on duplicates instead of finding new objects. The paper traces this to attention being "taken over" by the ever-growing history of previously generated coordinates, and proposes to compress each completed chunk of the answer into a small set of latent tokens so the model remembers compactly while still generating each local chunk exactly.

Key Contributions

1. Diagnosis of a long-horizon failure mode. The authors identify a "box repetition loop" in Full-History Auto-Regression (FHAR) and link it to a historical-answer attention takeover: attention mass shifts from the initial system tokens (attention sinks) to the growing raw answer history, and this shift coincides with the onset of repetition.
2. Compressed Window Auto-Regression (WinAR). A new paradigm that partitions a long answer into consecutive windows, keeps exact token-level autoregression only inside the current window, and replaces each completed window with a fixed set of latent tokens produced by a Window History Compressor.
3. A Window History Compressor design. An L-token window is reduced to 2B + N latent tokens: B boundary tokens at each end are kept verbatim to preserve exact local context, while the middle L − 2B tokens are summarized by N queries initialized via average-pooling over uniform intervals and refined with cross-attention.
4. Broad empirical validation. Experiments across scene densities (DOTA2 & SODA-A, RSDense), decoding strategies (next-token prediction, parallel box decoding, hybrid), and output formats (boxes, polygons), plus a demonstration that window-wise training cuts peak GPU memory.

Main Findings

• Repetition escalates with decoding horizon. Over the first 400 box predictions on RSDense, the redundancy rate of the FHAR baseline rises from roughly 10% to near 100%, while new-object discovery falls from about 16% to nearly zero (Fig. 2a).
• Attention takeover precedes the loop. Attention to the historical answer grows with the decoding horizon and rises sharply near the repetition onset, with a pronounced drop in attention to the initial system tokens.
• Large gains on the densest benchmark. On RSDense, RexOmni-WinAR improves F1@IoU=0.5 from 32.16 (FHAR* baseline) to 39.36 and Recall from 36.82 to 45.23, while cutting Mean R_dup from 35.48% to 13.00% and Mean R_over from 3.14 to 0.87.
• Smaller but consistent gains on the general benchmark. On DOTA2 & SODA-A, WinAR improves Recall from 66.94 to 68.32 and F1 from 67.59 to 67.97, with Mean R_dup falling from 2.43% to 1.26%.
• Dense categories benefit most. On RSDense, Ship F1 rises from 51.71 to 61.74 and Small Vehicle F1 from 19.94 to 26.96; Recall goes from 57.13 to 65.35 and from 22.65 to 30.06, respectively.
• GRPO post-training did not cure the loop. The FHAR*-GRPO variant reaches F1 31.98 on RSDense (versus 32.16 for FHAR*), suggesting the failure is architectural rather than a reward-shaping issue.
• Generalizes across decoding paradigms. With LocateAnything's parallel box decoding on RSDense, WinAR improves F1 from 26.15 to 27.54 and reduces Mean R_dup from 14.89% to 2.13%; under hybrid decoding F1 goes from 49.76 to 50.82 and Mean R_dup from 10.60% to 1.75%; under next-token prediction Mean R_dup drops from 5.90% to 0.49% with comparable F1 (54.87 vs. 54.40).
• Generalizes to polygon output. On polygon detection, F1 improves from 47.25 to 48.93 on DOTA2 (Mean R_dup 6.34% to 2.46%) and from 38.64 to 44.00 on SODA-A (Mean R_dup 3.72% to 1.93%).
• Training memory drops. Peak GPU memory for RexOmni training falls from 57.62 GB (FHAR*) to 42.97 GB (WinAR); for LocateAnything, WinAR reduces peak memory by 34.1%, enabling training on eight NVIDIA A40 GPUs (48 GB) without MagiAttention.
• Raw throughput is not always better, but effective throughput is. Boxes/s rises only modestly (6.05 to 6.68), yet Valid Boxes/s — throughput after removing redundant boxes — rises from 3.57 to 5.40.
• Compression frequency trades repetition for accuracy. Shortening L from 300 to 100 on RSDense drops Mean R_dup from 5.38% to 0.89% but also lowers F1 from 44.78 to 43.71.
• Exact boundary tokens matter more than extra latent capacity. With L = 200, removing boundary preservation (B = 0) and raising N to 20 lowers F1 from 67.22 to 64.82 on DOTA2 & SODA-A and from 44.56 to 32.47 on RSDense relative to (B, N) = (4, 6).
• A 6-layer compressor is the sweet spot. Four layers reach 37.73 F1 on RSDense; six layers reach 39.36 with the lowest Mean R_dup (13.00%); eight layers reach 39.32 with Mean R_dup rising to 16.87%.

Methodology in Plain English

Step 1 — understand the failure. The authors take a supervised fine-tuned RexOmni model (built on Qwen2.5-VL-3B, with coordinates quantized into 1,000 bins and the last 1,000 vocabulary entries repurposed as coordinate tokens) and track, at every decoding step, how much attention the coordinate-token queries place on four regions of the context: system tokens, visual tokens, prompt tokens, and the historical answer. They compare this attention profile against the rate of duplicate boxes and the rate of newly discovered objects.

Step 2 — change how history is carried. Instead of conditioning on every previously generated coordinate token, they cut the answer into fixed-length windows (L = 300 tokens for RexOmni; 50 atomic blocks of 6 tokens for LocateAnything). Inside the current window, generation is ordinary next-token prediction. Once a window finishes, it is compressed.

Step 3 — the compressor. Each completed window is split into front boundary tokens, middle tokens, and back boundary tokens. The boundary tokens (B = 6 each side) are passed through untouched so category phrases or box structures at the cut points are not mangled. The middle tokens are divided into N = 8 equal intervals, each average-pooled into one query, then refined with stacked cross-attention (keys and values come from the middle tokens plus learnable window-local positional embeddings) and feed-forward layers. The L-token window becomes 2B + N = 20 latent tokens.

Step 4 — training. Each window is trained in sequence: all preceding windows appear only via their compressed latents, and cross-entropy is applied to the current window. Raw window activations can be thrown away once compressed, which is where the memory saving comes from. Because window-wise training exposes long samples more often, they build exposure-matched FHAR* baselines that oversample long samples roughly ⌈T/L⌉ times.

Step 5 — inference. Generate the first window, compress it, drop the raw tokens, then generate the next window conditioned on the original multimodal prefix, the accumulated latent memory, and the current window's local tokens. Repeat until the end-of-sequence token or the decoding budget.

Step 6 — evaluate. Category-averaged Recall/Precision/F1 at IoU 0.5 on DOTA2 & SODA-A and on RSDense (712 images containing more than 200 objects, 412 objects on average), plus two redundancy metrics: R_dup (duplicate predictions over total predictions) and R_over (excess predictions over ground-truth count).

Why This Matters

Research impact. The paper reframes dense detection with MLLMs as a long-horizon generation problem rather than a localization problem. It supplies a mechanistic explanation (attention redistribution away from attention sinks) that connects the attention-sink literature to structured generation failures, and shows that the fix is a context-management change rather than more data, more post-training, or a bigger model.

Real-world applications:

• Remote sensing and aerial survey — counting ships, small vehicles, and other objects in high-resolution satellite and drone imagery, where scenes routinely contain hundreds of instances (the DOTA2 and SODA-A settings).
• Retail shelf auditing — dense product counting in shelf images such as SKU110K, where near-identical items tempt a model into duplicate predictions.
• Traffic and infrastructure monitoring — vehicle and vessel census from overhead cameras, where missing objects matters more than raw throughput.
• Layout-heavy document and scene parsing — any structured extraction task (polygons, keypoints, oriented boxes) where output sequences are long and repetitive.

Industry relevance. The memory result matters practically: the same hardware can train models that previously needed larger GPUs or specialized attention kernels. The Valid Boxes/s improvement means the same inference budget returns more usable detections, which directly affects cost per image at deployment scale.

Future Directions

• Adaptive window boundaries. Windows are currently fixed-length (300 tokens, or 50 blocks). Learned or content-aware boundaries that respect category and box structure could avoid splitting semantic units and reduce the reliance on preserved boundary tokens.
• Better history compressors. The depth ablation shows a plateau at six layers, and the boundary ablation shows that raw latent capacity cannot substitute for exact local tokens. Whether a different memory design (e.g., retrieval over compressed windows) outperforms uniform pooling remains open.
• Reducing per-window overhead. WinAR does not consistently improve raw throughput because of compression cost and KV-cache prefilling at each boundary; removing this overhead is an explicit open engineering problem.
• Interaction with reinforcement learning. GRPO on the FHAR* baseline gave no gain on RSDense; whether reward-based post-training becomes effective once the repetition loop is removed, or whether window-level rewards are needed, is untested.
• Beyond geometry. The paper tests boxes and polygons only; whether compressed-window history helps other long structured outputs (keypoints, OCR, segmentation token sequences) is not reported.

Target Audience

Researchers and engineers working on MLLM-based detection and grounding, long-context autoregressive decoding, and efficient multimodal training. It is most useful to those who already understand transformer attention and KV caching and are dealing with dense-scene detection, structured sequence generation, or memory-constrained fine-tuning of vision-language models. Practitioners deploying detection models on overhead imagery or high-density counting tasks will find the efficiency and redundancy results directly actionable.

Authors’ abstract

MLLM-based detectors formulate vision-language grounding as autoregressive coordinate generation, yet in dense scenes, they often collapse into a box repetition loop, repeatedly emitting identical boxes until the decoding budget is exhausted. We trace this failure to historical-answer attention takeover: Under full-history autoregression, repeated coordinate tokens in the growing answer history attract disproportionate attention, crowding out the attention sinks needed to attend to new visual evidence. % To break this loop, in this paper, we propose \textbf{Compressed Window Auto-Regression} (WinAR). This new mechanism partitions a long answer sequence into consecutive generation windows and restricts full token-level autoregression to the current window only while compressing each previous window into a fixed set of latent tokens that condition subsequent decoding. This design reduces the repeated exposure to raw historical coordinates that triggers the attention takeover, yet preserves cross-window dependencies essential for coherent multi-object generation. On RSDense, which contains 200-2,000 objects per image, our model WinARDet improves RexOmni F1-score (IoU=0.5) from 32.16 to 39.36 over the baseline, while reducing the redundancy rate from 35.48% to 13.00%. WinARDet also generalizes across decoding strategies and output formats, including parallel box decoding and polygon generation, providing a practical paradigm for long-horizon vision-language grounding across scene densities.