OBB-PBD: Extending Parallel Box Decoding to Oriented Bounding Boxes for Referring Expression Grounding in Aerial Imagery

AI summary

Overview

Research area: Computer vision and vision-language models — specifically referring expression grounding (localizing an object described in natural language) in aerial/UAV imagery, combined with oriented bounding box (OBB) geometry and parallel coordinate decoding.

Technical level: Advanced. The paper assumes familiarity with vision-language model architectures, LoRA fine-tuning, autoregressive vs. non-autoregressive token decoding, rotated IoU metrics, and the angle-periodicity problems of oriented detection.

Scope: The paper extends LocateAnything's Parallel Box Decoding (PBD) interface from a four-coordinate horizontal-box block to a six-slot oriented-box block, validates the extension with diagnostic probes, and measures its accuracy on the AVVG aerial grounding benchmark.

What This Paper Is About

Most grounding models emit box coordinates one token at a time, and almost all of them emit horizontal bounding boxes — a poor fit for aerial objects that appear at arbitrary rotations. Parallel Box Decoding predicts all of a box's geometry slots in one forward pass, but its released form only covers four-coordinate horizontal boxes. This paper asks whether that block-aligned interface can be extended to a six-slot oriented representation (center, size, and a double-angle orientation pair), what breaks when you try, and whether the resulting model actually uses the image and prompt rather than just fitting the training loss.

Key Contributions

1. A structurally validated six-slot OBB extension of PBD. The four horizontal corner coordinates are replaced by center-and-size, and two orientation slots — sin(2θ) and cos(2θ) — are added, giving block_size=8 and n_future_tokens=6. The codec's round-trip error is measured separately from model accuracy (mean round-trip IoU 0.9903 on train, 0.9823 on validation, minimum 0.8379), so representation error is distinguished from grounding error.

2. A diagnostic protocol for block-parallel extensions. Image/prompt-shuffle controls plus implementation visibility and leakage probes test whether the model depends on the image and whether the reported metric reflects the trained decoding path. Applied here, the probes exposed a silent failure: in the backbone's training path, the coordinate slots saw none of the image or prompt tokens while training loss still decreased normally.

3. A bounded, controlled performance result. Reported on the complete 6139-record AVVG test split (120 images) with checkpoint selection kept disjoint from that split, across three training seeds, plus a three-arm decomposition that separates the joint effect of domain adaptation and decoding-path change, the six-slot block format alone, and learning oriented geometry.

4. An explicit scope boundary. The paper states it does not compare head-to-head against joint-OBB-regression architectures such as SATGround, positioning its contribution as the PBD coordinate-block mechanism rather than joint OBB regression as such.

Main Findings

• Large gain over the zero-shot baseline. Rotated IoU rises from 0.1253 (zero-shot LocateAnything-3B) to 0.303 ± 0.012 (mean ± standard deviation over three seeds), an improvement of 0.178 ± 0.012, or roughly 142% relative. Axis-aligned IoU rises from 0.1512 to 0.334 ± 0.012.

• Threshold accuracy and angle error. Acc@0.50 goes from 15.1% to 37.1 ± 1.5%; Acc@0.75 from 3.3% to 22.0 ± 0.8%; angle MAE is 9.98 ± 0.34 degrees for OBB-PBD (the baseline row reports no angle MAE, shown as a dash). Decoded rate is 99.9% for both.

• A GT-derived horizontal-envelope reference scores 0.6037 rotIoU. This row is a computed statistic (the tightest axis-aligned rectangle around each ground-truth polygon), not a trained model, and the authors caution it is not a formal upper bound for axis-aligned prediction. Its Acc@0.75 is 16.7%, not 0%, because an axis-aligned box can exceed 0.75 rotIoU against a near-square or near-axis-aligned target.

• Learning orientation is what helps; the format switch alone does not. A matched HBB-envelope control — identical block size, collator, loss weights, and LoRA targets, but trained on axis-aligned envelopes — shows the orientation-learning step adds 0.055 ± 0.008 rotIoU, with per-seed 95% confidence intervals from +0.046 to +0.060 (smallest lower bound +0.036), an 18.7–23.7% relative increase. The format-switch step's rotIoU interval includes zero in all three seeds (−0.005 to +0.000).

• A silent loss of visual conditioning was found and fixed. The causal-prefix length was being computed from a position-ID pattern that training never produces. After recovering it from the mask-token region, the first coordinate slot's attention mask on the same training batch goes from 7 of 894 keys (none of them image or prompt tokens) to all 894.

• Four additional backbone defects caused crashes rather than silent errors and are corrected; they are listed in Appendix A.

• The normalized directional-loss gate was kept after testing a "corrected" version. The gate uses r_norm = max(w/W, h/H)/min(w/W, h/H), which is the image-normalized side ratio, not the physical aspect ratio w/h. Retraining with the physical gate (same seed-1 initialization and data order) reached rotIoU 0.3008 (95% CI [0.2799, 0.3228]) versus 0.3124 for the otherwise identical normalized-gate seed-1 run (paired difference −0.012, per-record image-cluster CI [−0.019, −0.004]). The authors note this lies within the spread of the three normalized-gate seeds (0.290–0.312) and that a single physical-gate run cannot separate the gate's effect from training randomness, so the normalized gate was retained.

• The HBB-PBD arm has a training-data asymmetry. Because the four-slot corner format cannot represent envelopes touching an image corner, that constraint was applied as a training-time exclusion filter, making this arm's training set about 10.3% smaller (20,182 vs. 22,495 records); at test time, corner-touching cases (~4% of the split) are scored as decode failures rather than excluded. The authors flag this explicitly.

• Qualitative failure mode is misidentification, not bad geometry. In one shown example, both models place a well-formed box with 8.2° angle error on a neighboring car rather than the one described, scoring rotIoU 0.00 for both. In the other two examples, both models find the right car but the baseline's axis-aligned box cannot represent its tilt (OBB-PBD rotIoU 0.82 and 0.79 versus baseline rotIoU 0.53 and 0.60).

• Not reported in the supplied content: the image/prompt-shuffle diagnostic results (Section 4.3) and the decoding-speed/efficiency comparison (Section 4.4). The text is truncated partway through Section 4.2, so those numbers cannot be stated here.

Methodology in Plain English

The model being extended is LocateAnything-3B: a Qwen2.5-3B-Instruct language model paired with a MoonViT vision encoder. Normally, such a model writes out a box one coordinate token at a time. PBD instead uses a trigger token, <box>, followed by a single forward pass that predicts every coordinate slot at once; the closing </box> token comes at the next step.

To make that block oriented, the authors change what the slots mean. Instead of four corner coordinates, they use the box's center (c_x, c_y), its width and height (w, h), and two orientation values, sin(2θ) and cos(2θ). The double-angle trick avoids the jump that a raw angle would have at the 0°/180° boundary while still respecting a rectangle's 180° symmetry. Because a rectangle with (w, h, θ) is the same box as (h, w, θ+90°), every box is first canonicalized to w ≥ h so training and evaluation always speak the same language. Every number is quantized into 1001 levels (0 through 1000), and signed values are rescaled into [0, 1] first.

Training uses LoRA (rank 16) on the language model, LoRA on the last six MoonViT vision blocks, and full training of the mlp1 projector, with the projector at one tenth the adapters' learning rate (1×10⁻⁵) and gradients clipped at norm 1.0. Beyond the standard next-token cross-entropy, three differentiable auxiliary terms are computed from soft-argmax expectations over each slot's 1001 level probabilities: a Smooth-L1 spatial loss on the four geometric slots (weight 2.5), a circular consistency loss pushing (ŝ, ĉ) onto the unit circle (weight 0.25), and a directional cosine loss gated to boxes with r_norm ≥ 1.05 (weight 0.5).

The data is the AVVG subset of the RefGeo collection: 26,451 usable records after removing 31 out-of-frame annotations, split into a 421-image training set (22,495 records) and a 74-image validation set (3,956 records) disjoint from the 120-image test split. An audit found 97.85% of boxes are not axis-aligned. Training runs 5 epochs on three NVIDIA RTX 6000 Ada GPUs, three seeds per model, with the best-validation-rotIoU checkpoint selected per run and then evaluated once on the test split. Evaluation uses a direct PBD decoder rather than the upstream generate() path, because the latter only implements multi-token prediction for the original six-token HBB block. Rotated IoU is computed with Shapely 2.1.2 polygon intersection, and angle error uses a periodic distance modulo 180° so that 1° versus 179° counts as 2°, not 178°.

Why This Matters

Impact on research. The paper's most transferable lesson is methodological: a decreasing loss curve did not reveal that coordinate slots had lost all access to the image and prompt. Any group extending block-parallel or otherwise block-aligned decoding should expect the same class of silent failure, and the shuffle controls and attention-visibility probes here give a cheap template for catching it. The three-arm decomposition also demonstrates how to attribute a gain to "new output format" versus "actually learning the new geometry" — a distinction that would be easy to conflate.

Real-world applications:

• UAV surveying and inspection — locating a specifically described vehicle, vessel, or structure in overhead imagery where objects are elongated and rotated.
• Maritime and port monitoring — ship detection and identification, where heading and aspect ratio carry real meaning and horizontal boxes waste substantial area.
• Disaster response and search-and-rescue — directing an operator to "the truck nearest the collapsed building" in dense aerial scenes.
• Agriculture and infrastructure auditing — counting and localizing oriented assets such as parked machinery, rows, or panels from drone flights.

Industry relevance. Geospatial analytics platforms increasingly pair natural-language querying with imagery; the ability to return a correctly oriented footprint, in a single parallel decoding step rather than one token per coordinate, matters for both query latency and downstream geometric use. The paper's honest reporting of a modest absolute rotIoU (0.303) also signals that this remains a research-stage capability rather than a deployable one.

Future Directions

• Closing the absolute-accuracy gap. At 0.303 rotIoU and 22.0% Acc@0.75, oriented grounding is far from usable in production. The authors' own intuition — that the qualitative failures are referring errors (picking the wrong neighboring car) rather than geometry errors — points toward better relational and comparative language understanding rather than better angle regression.
• Head-to-head comparison with joint OBB regression. The paper explicitly declines to compare against SATGround-style regression heads under a matched protocol. A controlled comparison of the block-aligned token interface versus a dedicated regression head on the same data would clarify when each design wins.
• Fixing the directional-loss gate properly. The gate was designed for near-square boxes but, on non-square 4000×2250 imagery, ends up excluding boxes whose aspect ratio is close to the image's (~16:9) while near-square boxes still receive the directional gradient. A single physical-gate run was not enough to separate the effect from training noise; a multi-seed study is the obvious next step.
• Reporting the withheld analyses. The image/prompt-shuffle diagnostic and the decoding-speed comparison are referenced but not present in the supplied content, and broader efficiency characterization — the stated motivation for parallel decoding — would show whether the six-slot extension preserves PBD's speed advantage.

Target Audience

Researchers and engineers working on vision-language grounding, remote-sensing vision-language models, or oriented object detection who already understand transformer decoding and rotated-box geometry. It is also valuable to practitioners interested in evaluation rigor — particularly those who build on top of an existing backbone and need to know that a falling loss curve is not evidence of a working model. Readers looking for a tutorial on visual grounding or for a state-of-the-art aerial grounding system will find it too specialized and too candidly scoped for either purpose.

Authors’ abstract

Referring expression grounding in overhead imagery requires oriented bounding boxes rather than horizontal ones, since aerial objects appear at arbitrary orientations. Most grounding models emit box coordinates sequentially; Parallel Box Decoding (PBD) instead predicts all geometry slots in one forward pass, but only for four-coordinate horizontal boxes. We extend this block-aligned interface to a six-slot oriented representation using a double-angle parameterization. Rather than assuming correctness from a decreasing loss curve, we verify the extension with image/prompt-shuffle controls and visibility/leakage probes, which show that the corrected model depends on both the image and the prompt rather than on prompt statistics alone. On the aerial vehicle visual grounding (AVVG) benchmark, whose targets are genuinely oriented, the extended model improves rotated intersection over union (rotIoU) from 0.1253 for the zero-shot baseline to 0.303 +/- 0.012 (mean +/- standard deviation over three training seeds) across the complete 6139-record test split spanning all 120 images. Learning oriented boxes rather than their axis-aligned envelopes adds 0.055 +/- 0.008 rotIoU beyond a matched control (per-seed 95% confidence intervals all excluding zero), whereas block expansion alone yields no rotIoU gain distinguishable from zero.