CoverPrune: Coverage-Driven Token Pruning for 3D VLMs via Optimal Transport

AI summary

Overview

Research area: Computer vision and multimodal machine learning, specifically efficiency for 3D Vision-Language Models (3D VLMs) through training-free visual token pruning.

Technical level: Advanced. The paper assumes familiarity with Vision-Language Models, attention, token pruning, and Optimal Transport, and it derives a semi-relaxed OT formulation with greedy subset selection.

Scope: The paper proposes and evaluates CoverPrune and CoverPrune-Lite, two training-free token pruning modules that select a compact set of visual tokens for 3D VLMs by maximizing coverage of visual evidence rather than maximizing diversity, and validates them on ScanQA, SQA3D, Scan2Cap, and VSI-Bench using GS-Reasoner and VLM-3R as base models.

What This Paper Is About

3D Vision-Language Models can reason about space, but a single input can produce hundreds or even thousands of visual tokens, so inference becomes dominated by the quadratic cost of attention and by growing KV caches. Existing pruning methods pick tokens by attention saliency or by maximizing diversity, which in 3D scenes tends to discard the shared prototype tokens that carry repeated multi-view evidence and break geometric consistency. The paper's goal is to redefine token pruning for 3D VLMs as preserving coverage of visual evidence, and to make that objective practical to optimize at inference time.

Key Contributions

1. A new pruning paradigm. CoverPrune is a training-free framework that shifts the pruning objective from maximizing token diversity to preserving visual evidence coverage, formulated through Optimal Transport (OT) so that retained tokens act as prototypes that must explain the full token set.

2. Three tailored OT designs. The paper introduces a Feature-Spatial-Temporal (FST) transport cost that combines semantic similarity, 3D spatial proximity, and temporal coherence; informativeness-aware target capacities derived from local FST distinctiveness; and a Spatial-Guided Greedy Selection (SGS) algorithm that approximates the semi-relaxed OT objective.

3. A lightweight accelerated variant. CoverPrune-Lite replaces iterative global transport solving with a Morton-code 3D-aware ordering, capacity-guided grouping into K groups, and one prototype per group, achieving O(N log N) inference-time complexity.

4. State-of-the-art results on 3D benchmarks. Across Scan2Cap, ScanQA, SQA3D, and VSI-Bench, the methods rank at or near the top under matched token budgets and degrade far more gracefully than prior methods as the retention ratio drops.

Main Findings

• Strong results on general 3D tasks at 20% retention. Against a Vanilla GS-Reasoner baseline scoring 100.00 average relative retention on each of Scan2Cap, ScanQA, and SQA3D, CoverPrune reaches 101.68, 100.95, and 99.67, and CoverPrune-Lite reaches 101.31, 102.46, and 97.45. CoverPrune also reports absolute Scan2Cap B-4 of 50.10 versus 47.60 for Vanilla, and ScanQA CIDEr of 103.17 versus 102.60 for Vanilla.

• The advantage widens under aggressive pruning. At 10% retention CoverPrune reaches 99.04 on Scan2Cap, 100.00 on ScanQA, and 97.45 on SQA3D, while baselines such as EgoPrune fall to 80.69, 80.66, and 86.86 on the same benchmarks.

• VSI-Bench spatial reasoning, GS-Reasoner base model. With a full-token Vanilla average of 64.70, CoverPrune scores 59.76 at 20% retention, 58.27 at 15%, 56.83 at 10%, and 52.66 at 5%; CoverPrune-Lite scores 59.43, 57.72, 56.94, and 52.88 at the same ratios. The paper reports that at 20% token retention CoverPrune preserves 92.4% of full-token performance.

• Graceful degradation under extreme compression. The paper states that its methods show substantially higher scores than the strongest competitor at 10% and 5% retention, and Figure 1 reports over 90% performance retained with 15% visual tokens on VSI-Bench and near-zero performance loss with 10% visual tokens on general 3D tasks.

• VSI-Bench with a different base model. With VLM-3R and a Vanilla average of 60.90, CoverPrune scores 54.69 at 10% and 51.28 at 5%, while CoverPrune-Lite scores 54.74 and 51.56; the strongest listed baseline at 5%, DTC, scores 49.83.

• Complementary behavior between the two variants. The paper observes that CoverPrune-Lite tends to do better on global layout-focused Room Size tasks, attributed to its geometry-aware ordering preserving coarse scale signals, while full CoverPrune shows advantages on fine-grained Relative Direction tasks through full coverage optimization.

• Baseline weakness in diversity-driven pruning. EgoPrune, described as attention-diversity fused merging, drops most sharply; at 20% retention on general 3D tasks it reaches only 88.79 on Scan2Cap, 88.67 on ScanQA, and 92.04 on SQA3D.

Methodology in Plain English

The input is a set of visual tokens from a video or multi-view observation, each with a feature embedding, an estimated 3D global coordinate, and a timestamp. Given a retention ratio R, the method must keep K = ceil(R·N) tokens. The module is inserted between the visual-geometric encoder and the 3D VLM, immediately before the LLM prefill stage, and is training-free.

The central idea is to treat the kept tokens as a small set of "prototypes" that must distribute their representational mass across all original tokens, and to choose the set that minimizes the resulting transport cost. This is an Optimal Transport problem: a cost matrix measures how badly one token substitutes for another, a transport plan moves mass from kept tokens to original tokens, and a low-cost plan means the kept tokens explain the originals well.

To make this work for 3D reasoning, the paper defines three pairwise discrepancies: feature discrepancy (1 minus cosine similarity), 3D spatial distance, and a temporal term ReLU(tau_source - tau_target) that penalizes covering an earlier observation with a later one. These are min-max normalized per sample, combined with weights lambda_f, lambda_x, and lambda_tau (all set to 1 in the experiments), and the spatial term is passed through a log-based nonlinear mapping phi_kappa(x) = log(1 + kappa·x) / log(1 + kappa) to expand the dynamic range for near-field distances.

Rather than weighting all original tokens equally, each token gets a capacity based on how distinctive it is from its n nearest neighbors in 3D space, using the same feature, spatial, and temporal discrepancies with weights alpha_f, alpha_x, and alpha_tau (all set to 1). Distinctive tokens receive more capacity, so dense but redundant regions cannot dominate the objective.

Selecting a subset that minimizes this OT cost is a coupled discrete-continuous problem and is NP-hard. The paper relaxes the strict target marginal constraint to an inequality (semi-relaxed OT: P·1 = u, P^T·1 ≤ v) and adds an entropic regularizer, which makes the induced set function amenable to greedy optimization. To avoid re-solving a global transport problem at every greedy step, Spatial-Guided Greedy Selection restricts marginal-cost evaluation to the g nearest target tokens in 3D space, solving one semi-relaxed OT problem per step and then picking the candidate that minimizes residual-capacity-weighted local cost.

CoverPrune-Lite skips the iterative solving entirely: it sorts tokens with a Morton code space-filling curve so that adjacent list entries are spatially close, walks the sorted list accumulating target capacity until it reaches 1/K to form K groups, and selects one prototype per group by minimizing the capacity-weighted within-group cost. The paper shows this is a block-constrained version of the OT objective whose optimal coupling degenerates to P_qj = v_j for tokens inside group q, exactly recovering the within-group selection rule.

Why This Matters

• Impact on research: The paper challenges the default assumption behind most training-free pruning work, that similar tokens are redundant, and argues that for spatial reasoning the similar tokens are often the representative evidence that multi-view geometry depends on. It also opens systematic token-pruning evaluation on VSI-Bench, which the paper notes has substantially higher complexity and reasoning difficulty than the general 3D QA tasks where performance "has largely saturated."

• Real-world applications:

  • On-device or embedded spatial assistants that must answer questions about a room, scene, or headset capture with a limited compute and memory budget.
  • Robotics and embodied agents that need to interpret multi-view or video observations in real time, where KV cache growth is a practical limit.
  • Augmented and virtual reality systems that build scene understanding from egocentric video and must run under tight latency constraints.
  • Long-video 3D inspection or documentation pipelines, where the number of frames and therefore tokens grows without bound.

• Industry relevance: The method is training-free and plug-and-play, inserted before the LLM prefill stage without changing base-model configurations, which lowers the barrier to adoption for teams already running 3D VLMs. It is also described as compatible with diverse acceleration frameworks, and the Lite variant targets minimal overhead, both of which matter for serving costs.

Future Directions

• Whether the coverage objective transfers beyond 3D. The paper's motivation is specific to multi-view and geometric evidence; it is not reported whether the same coverage-with-compactness formulation helps 2D image or generic video understanding, where the evaluated baselines originate.

• Runtime and memory measurements. The paper reports theoretical complexity, cubic-time for the iterative greedy procedure and O(N log N) for CoverPrune-Lite, but the visible content does not report measured latency, throughput, or memory savings, so the practical speedup over baselines remains unquantified here.

• Hyperparameter sensitivity. The cost weights lambda_f, lambda_x, lambda_tau, the capacity weights alpha_f, alpha_x, alpha_tau, the neighborhood sizes n and g, and the mapping phi are all fixed to specific values; the truncated content includes the beginning of an ablation table but no complete sensitivity analysis, so how much performance depends on these choices is not fully reported.

• Extension to other 3D VLM backbones and richer settings. Only GS-Reasoner and VLM-3R are instantiated, with uniform 32-frame sampling and coordinates estimated without ground truth. Whether the same behavior holds for other geometric encoders, longer sequences, or noisier coordinate estimates remains open.

Target Audience

Researchers and engineers working on efficient multimodal inference, 3D vision-language models, and spatial reasoning benchmarks; practitioners who need to deploy 3D VLMs under tight token, memory, or latency budgets; and readers interested in Optimal Transport as a tool for subset selection and data summarization. Some background in VLM architecture and attention is helpful for following the optimization sections, while the motivation and results sections are accessible to a broader machine learning audience.

Authors’ abstract

While 3D Vision-Language Models (3D VLMs) have demonstrated remarkable spatial reasoning capabilities, they suffer from massive visual token counts that create severe computational bottlenecks during inference. Existing token pruning methods primarily rely on diversity-based selection, discarding similar tokens to maximize dispersion. However, in 3D environments, this approach frequently drops representative prototype tokens in favor of outliers, breaking the multi-view consistencies and geometric structures essential for spatial reasoning. In this paper, we propose a paradigm shift for 3D VLM token pruning: from maximizing diversity to preserving visual evidence coverage. We introduce CoverPrune, a training-free framework that formulates inference-time token pruning as an Optimal Transport (OT) problem. To overcome the intractable combinatorial subset selection inherent in this formulation, we design the Feature-Spatial-Temporal (FST) transport cost and target capacity, along with an efficient Spatial-Guided Greedy Selection (SGS) algorithm to approximate the OT objective. Furthermore, we propose CoverPrune-Lite, an accelerated variant utilizing spatially structured local matching for minimal overhead. Extensive experiments across multiple 3D visual-spatial reasoning benchmarks demonstrate that our methods achieve state-of-the-art token efficiency, maintaining robust reasoning performance even under highly aggressive pruning budgets. Visit our project website at https://github.com/Brucess/CoverPrune.