Token-Budgeted Candidate Retrieval for 3D Scene-Grounded VLMs: Category Names vs. Pairwise Geometry

AI summary

Overview

Research area: 3D scene-grounded vision-language models (VLMs), specifically the upstream retrieval step that decides which objects and relations from a scene graph are serialized into a fixed-size prompt.

Technical level: Intermediate. The core question is intuitive, but the paper assumes familiarity with 3D referring-expression benchmarks (ScanRefer, Nr3D, Sr3D), Recall@K, Acc@0.25 (3D IoU), and token-budget accounting.

Scope: The paper compares training-free, category-name-based retrieval against a deterministic pairwise-geometry scorer (QCRR) on ScanNet-derived indoor benchmarks, then measures how each selector affects downstream grounding accuracy in one 3DGraphLLM checkpoint.

What This Paper Is About

A 3D scene-grounded VLM facing a query like "the lamp on the desk" cannot serialize an entire scene graph into its prompt, so it must choose a small set of objects and relations first. The paper asks which selection strategy earns that limited budget: matching the query to object category names, or scoring pairs of objects with hand-written geometric rules (left/right, above/below, near/far, on/inside, overlap). The authors build a controlled protocol for this comparison, introduce a geometry-based selector as an analysis instrument, and check whether retrieval gains actually translate into better end-to-end grounding.

Key Contributions

1. A plug-in retrieval protocol. A retrieve(q, boxes, names) interface with a normalized token-ratio metric (τ = retrieved GPT-2 tokens / full-prefix tokens), per-query logs, and fixed budgets of M = 60 relations and K = 20 objects matching 3DGraphLLM evaluation.

2. QCRR (Query-Conditioned Relation Retrieval). A deterministic, training-free pairwise-geometry scorer that parses spatial cues phrase-first, scores ordered object pairs on offsets, distance, 3D IoU and containment, down-weights relations touching structural categories (wall, floor, ceiling, door, window, doorframe, stair rail) by ×0.1, and applies a multiplicative noun prior of β = 1.5. It runs in O(N²) at 49 ms/query on Nr3D and 64 ms/query on Sr3D on a single CPU core.

3. Multi-axis stratification. Results are broken down by cue type, scene size, verbatim vs. paraphrased category mention, the ReferIt3D uses_spatial_lang flag, relation budget, and per-class disambiguation difficulty.

4. A corrected end-to-end harness. After auditing their own setup, the authors found three bugs (relations passed as prose rather than through the embedding pathway, raw ScanNet ids used instead of model object indices, and zero-filled object features from a key-format mismatch) and re-ran downstream evaluation on 3DGraphLLM.

Main Findings

• Category names beat pairwise geometry at every reported K. On ScanRefer gt_val (9,508 queries), MiniLM reaches R@1 0.461 vs. QCRR's 0.202, and the additive noun baseline reaches R@10 0.864 vs. QCRR's 0.626. On Nr3D, MiniLM R@10 is 0.941 vs. QCRR's 0.634. Gaps widen at higher recall.

• Two small frozen encoders are close to each other. MiniLM (all-MiniLM-L6-v2, ~22M parameters) and BGE-small (bge-small-en-v1.5, ~33M) reach R@10 0.911 and 0.913 on ScanRefer and 0.941 and 0.947 on Nr3D; they differ by at most 0.6 pp on ScanRefer, with a 5.0 pp gap at Sr3D R@1.

• QCRR's own recall is largely lexical. Removing the noun prior drops its R@10 by 19.8 pp (0.626 to 0.428); removing hub suppression costs 3.8 pp, and replacing max-edge aggregation with a sum of the top three relations costs 9.0 pp at R@10.

• Geometry still beats the right baselines. QCRR beats query-agnostic kNN-graph selection, whose Sr3D R@1 of 0.016 falls below Random's 0.022 while QCRR reaches 0.254. On the 848-query (8.9%) paraphrased ScanRefer bucket, QCRR R@10 is 0.440 vs. Cue-only 0.288 (+15.2 pp), a ranking flip seen at every K; on the 8,660 exact-match queries Cue-only instead leads at R@10 (0.756 vs. 0.652).

• Richer object text raises the semantic ceiling. Using leave-one-out descriptions of the same instance lifts MiniLM from R@1 0.461 to 0.587 and R@10 0.911 to 0.971, though such captions are unavailable at deployment.

• Retrieval ranking carries downstream. On 300 ScanRefer queries (seed 42) under a matched object budget, Acc@0.25 spans 0.410 to 0.697: kNN-graph 0.410, Random 0.443, QCRR 0.557, noun baseline 0.647, full graph 0.670, MiniLM 0.673, oracle 0.697. QCRR beats Random by +0.114 (95% CI [0.034, 0.193]) and kNN by +0.147 [0.068, 0.226]; MiniLM beats QCRR by +0.116 [0.039, 0.194].

• GT-in-budget explains most of the spread. Random places the target in the serialized graph 0.60 of the time, QCRR 0.82, the learned retriever 0.96, the oracle 1.00. Improving retrieval without getting the target into the prompt has limited downstream effect.

• Scale and orientation both hurt geometry. Recall@10 falls with scene size for all tabulated methods; at N ≥ 61, Cue-only + noun holds 0.609 vs. QCRR's 0.487. Vertical cues are easiest (R@10 0.773 on 1,303 queries) and left/right hardest (0.696 on 3,957); 2D PCA room-axis alignment on a separate 4,272-query left/right slice gains +0.027 R@1 and +0.047 R@5, but does not close the gap.

• Outdoor transfer is near chance. On CityRefer val (5,934 queries, six scenes, indoor cues reused), MiniLM R@1 is 0.013, QCRR 0.008, noun 0.011, Random 0.004; the authors exclude outdoor numbers from the main tables.

Methodology in Plain English

The authors fix one scene representation — N axis-aligned 3D boxes plus a category string per box — and one query string, then let different selectors choose which top-K objects and top-M relations enter the prompt. All selectors see identical inputs, so differences come from the selection rule alone.

Three families are compared. The simplest is training-free heuristics: random selection, a query-agnostic k-nearest-neighbor graph over objects, a centroid heuristic, and "Cue-only" scoring where each object is scored against spatial words in the query, optionally with an additive bonus when the query mentions its category. The second family is QCRR: it parses spatial phrases from the query (multi-word triggers like "in front of" fire before single words like "left"), scores every ordered object pair with analytic terms mapped to a logistic over world-frame center offsets, distance normalized by scene diagonal, 3D IoU, and containment, suppresses relations attached to structural clutter, and gives each object the maximum score among its incident relations. The third family feeds the query and each object's category name into a frozen sentence encoder and takes cosine similarity.

Budgets are made comparable by reporting a token ratio rather than a raw relation count, so a selector is not rewarded merely for serializing more text. Downstream, each selector is wired into 3DGraphLLM's native graph pathway with the same object budget, and accuracy is measured as Acc@0.25 in the model's object-index space.

The paper is also candid about self-correction: an earlier version of the harness made Random, kNN, and QCRR look statistically indistinguishable downstream. The audit traced this to prose-formatted relations, misaligned label indexing, and zero-filled features, after which the native full-graph configuration reached Acc@0.25 0.67 with the released checkpoint loading 448 LoRA adapters and zero unexpected keys.

Why This Matters

The paper is a caution against assuming that better geometric reasoning is the right lever for context selection. On indoor queries that name a category — the common case — a cheap frozen text encoder matching the query to object labels outperforms a hand-engineered pairwise geometry scorer, and the downstream gap is large enough to change conclusions about which retriever to ship. The results also supply a concrete decision rule: use geometry when the ground-truth category string is absent from the query, or when a class has many same-instance candidates; stop at semantics when the class is essentially unique.

Real-world applications:

• Indoor service robots and manipulators that must resolve "the mug next to the sink" from a detected scene graph within a prompt-length limit.
• AR/VR and smart-home assistants answering spatial questions about a scanned room where the whole graph cannot fit in context.
• Embodied agents and navigation systems that need a predictable, low-latency shortlist before invoking an LLM planner.
• Benchmark and tooling builders who need a budget-fair way to compare context selectors without rewriting the evaluation analysis.

Industry relevance centers on cost: matching a query against category strings is far cheaper than pairwise scoring over all object pairs, and the paper quantifies that trade-off in milliseconds and in accuracy points. The released protocol is designed so new selectors can be swapped in without re-deriving the analysis, which suits teams iterating on retrieval policies for grounded assistants.

Future Directions

• Multi-target and abstention-aware retrieval. The authors did not run QCRR under the F1 protocol on Multi3DRef because it cannot emit an empty answer when no class matches; zero-target subsets are where lexical matching already scores 0.695 F1@0.25 (ZT-wo-d, 528 queries) against 0.000 for random. Adding abstention is an open engineering problem.
• Better room-axis estimation. Left/right cues trail vertical ones because ScanNet's world x-axis is not aligned with the room; PCA recovers only +0.027 R@1 on the 4,272-query slice, and the authors suggest a dedicated layout estimator as a better source of room axes.
• Richer object text at inference time. The leave-one-out bound (R@10 0.971) shows headroom for semantic retrieval, but those captions come from other ScanRefer annotations and are unavailable at deployment; generating equivalent descriptions on the fly is untested.
• Broader backbone and geometry coverage. Only one 3DGraphLLM checkpoint and axis-aligned boxes are evaluated; oriented or articulated object representations and additional scene-LLM backbones are untested, and the oracle at 0.697 is not statistically above the full graph at 0.670, suggesting remaining error comes from more than the object budget.

Target Audience

Researchers and engineers building 3D scene-grounded LLM systems, particularly those working on context or subgraph selection under token limits. It is also useful for practitioners who need a defensible baseline before investing in geometric reasoning modules, and for benchmark designers interested in budget-fair retrieval metrics and in how harness bugs can silently flatten comparisons between methods. Readers without background in 3D referring benchmarks will need to consult the cited ScanRefer, Nr3D, and Sr3D papers first.

Authors’ abstract

Under a fixed context budget, which objects and relations should a 3D scene-grounded VLM see first? On ScanNet indoor referring benchmarks (Dai et al., 2017), matching the query to object category names outperforms pairwise geometry scoring at every reported K. The gaps are largest at high recall. Analytic pairwise cues beat per-object cue scoring and query-agnostic kNN graphs, including on paraphrases. MiniLM and BGE-small still score higher on those splits. We evaluate token-budgeted candidate retrieval with axis-aligned boxes, a token-ratio metric, training-free baselines, two small encoders that match q to each object's category name, and QCRR (Query-Conditioned Relation Retrieval), a deterministic pairwise-geometry scorer. Across ScanRefer gt_val plus a custom 20% scan-id split of Nr3D and Sr3D (35,504 queries; not official ReferIt3D val), additive noun matching reaches R@10 0.864 on ScanRefer against QCRR's 0.626; MiniLM and BGE-small reach 0.911 to 0.913 on ScanRefer and 0.941 to 0.947 on Nr3D. A corrected 3DGraphLLM harness (native Acc@0.25 0.67 on the full graph, n=300 ScanRefer queries) shows that context selection changes downstream accuracy under a matched object budget (Random 0.44, QCRR 0.56, MiniLM 0.67, oracle 0.70). Semantic retrieval beats hand-designed geometry on that harness. We will release the protocol, retriever implementations, and per-query outputs so new selectors can be swapped in without rewriting the analysis.