Depth-Guided Video Object Counting in Crowded Scenes

AI summary

Overview

• Research area: Computer vision — multimodal (RGB-D) video object counting, depth-guided detection, and cross-frame tracking/de-duplication in crowded, occluded scenes.
• Technical level: Advanced. The paper builds on open-vocabulary detectors (GroundingDINO/CountGD-Box), transformer decoder heads, cross-modal attention with a depth-derived bias, FiLM modulation, Bayesian association costs, and Hungarian matching.
• Scope in one sentence: The paper introduces a depth-guided detector (DG-Det) plus a depth-guided tracking and de-duplication pipeline (DG-Track), and releases the synchronized RGB-D RGBD-VideoCount dataset, reporting a 62.01% reduction in MAE versus existing baselines.

What This Paper Is About

Counting all instances of a prompted target category (by text or visual exemplar) inside a video is hard when objects are densely packed and mutually occluded, because RGB only captures appearance cues such as color and texture. The authors argue that tightly adjacent, often visually identical objects still have distinguishable depth differences and spatial hierarchy in 3D, so they inject depth into both the single-frame detector and the cross-frame association stage. The goal is a unified framework that produces accurate video-level counts inside a specified region of interest (ROI), together with a new dataset that supplies synchronized depth for this task.

Key Contributions

1. Introducing depth cues to video individual counting in crowded scenes, which the authors state alleviates the limitations of RGB-only methods under occlusion (motivated by real scenarios such as shelf inventory checks and warehouse inspections).
2. A depth-guided RGB-D detection and de-duplication framework — DG-Det (multi-scale RGB-D cross-attention with Depth Affinity Bias, an occlusion-aware prediction head, and FiLM-based global feature generation) plus DG-Track (depth-aware association cost and Occlusion-Adaptive Temporal Voting) — that better distinguishes occluded instances while reducing missed detections and duplicate counting.
3. Releasing RGBD-VideoCount, a dataset with synchronized RGB-D video streams, multiple object categories coexisting per sequence, instance-level bounding boxes, exemplar boxes, and video-level ROIs: 195 video clips across 6 categories, 2,032 finely annotated frames and 77,638 instance boxes, captured with an Intel RealSense D455.
4. Code and dataset release at https://github.com/streamer-AP/DG-Net and https://huggingface.co/datasets/aerospace123/RGBD-VideoCount.

Main Findings

• Overall improvement: The method achieves a 62.01% reduction in MAE compared to existing baselines, with consistent improvements in RMSE.
• Text-only prompt results (RGBD-VideoCount): The authors' full-depth model reaches validation MAE/RMSE of 14.55/25.32 with AP 20.16 and AP50 43.26, and test MAE/RMSE of 12.68/15.87 with AP 17.52 and AP50 45.68, versus CountVID at validation 24.77/28.76 (AP 11.59, AP50 25.76) and test 23.69/30.78 (AP 9.38, AP50 23.21).
• Text + exemplar prompt results: CountVID reports validation 19.68/26.87 (AP 18.73, AP50 47.65) and test 20.98/27.73 (AP 17.08, AP50 48.42); the proposed model reports validation 13.32/19.68 (AP 23.96, AP50 50.12) and test 7.97/11.44 (AP 20.78, AP50 50.39).
• Strong single-frame detection is not enough: The SAM 2.1 baseline achieves an excellent AP50 but a high MAE, showing that without explicit occlusion modeling, frequent ID switches and fragmented trajectories lead to severe over-counting.
• Depth helps only when fused properly: Naive RGB-D channel concatenation leads to many missed detections; DepthMOT, which uses depth only in tracking, performs comparably to pure RGB trackers (test MAE/RMSE 23.57/32.65, AP 4.08, AP50 9.30), while DepTR-MOT (depth in detection and tracking) reaches test 14.33/25.75, AP 3.40, AP50 13.28.
• Real depth beats pseudo-depth: Using pseudo-depth, the model reports validation 16.95/28.39 (AP 21.95, AP50 45.16) and test 13.75/20.81 (AP 16.97, AP50 44.79), worse than the full real-depth setting. Predicted depth maps suffer from "edge smoothing" that erases spatial boundaries between overlapping objects, whereas the sparse, noisy ground-truth depth keeps sharp depth changes.
• Depth coverage ablation: RGB-only (0%) gives validation 19.88/28.46 and test 15.47/25.26; 25% partial depth gives validation 16.15 ± 0.29 / 30.25 ± 1.57 and test 14.74 ± 0.17 / 35.85 ± 0.82; 50% gives validation 16.52 ± 0.40 / 28.87 ± 0.61 and test 12.60 ± 0.36 / 19.66 ± 0.93; 75% gives validation 14.18 ± 0.27 / 29.13 ± 1.10 and test 12.91 ± 0.21 / 36.44 ± 0.82; 100% gives validation 13.33/19.68 and test 7.98/11.45.
• Module-by-module ablation: Starting from RGB-only at validation 18.72/32.40 and test 18.07/26.80, adding Cross-Attention Fusion gives 17.12/28.77 and 14.93/20.89; adding Depth Affinity Bias gives 14.26/26.08 and 12.20/15.99; adding the Occlusion Head gives the best result of 13.33/19.68 and 7.98/11.45.
• Robust hyperparameter: Model performance remains stable when the Depth Affinity Bias coefficient β varies within the range 0.5–3.0, with only minor MAE and RMSE fluctuations.
• Adaptive voting helps: Occlusion-Adaptive Temporal Voting reduces MAE and RMSE by 1.91 and 2.75 respectively compared with a fixed-threshold temporal filtering scheme.
• Depth-noise robustness: Under depth corruption, Edge Blur yields a test MAE of 8.50 and Random Holes 8.23, close to the 7.97 obtained with clean depth; Gaussian noise and Salt & Pepper noise degrade performance more, raising MAE to 11.16 and 10.66.
• Additional evaluation: The method is also evaluated on FSCD-147 (Nguyen et al., 2022) using pseudo-depth estimated by Depth Anything V2, with details and qualitative results in the supplementary material.

Methodology in Plain English

The framework takes a synchronized RGB-D video, an open-vocabulary text query, a visual exemplar, and a video-level ROI, and outputs a count.

Detection (DG-Det). A BERT encoder processes the text query, keeping tokens for the category name. A GroundingDINO backbone and FPN extract multi-scale RGB features (levels 2–5), while a separate depth encoder (ResNet-18 + FPN with the first convolution changed to single-channel input, initialized from the channel-wise mean of RGB pretrained weights) produces a matching depth pyramid. At every scale, RGB features become queries and depth features become keys/values in a cross-attention module, with a Depth Affinity Bias subtracted from the attention logits: the bias is proportional to the absolute difference of normalized depth values, so interactions between regions that are far apart in 3D but nearby in the image plane are suppressed. Depth normalization uses a log-inverse transformation and quantile normalization. Because the Transformer needs four input scales but the backbone outputs three, a FiLM-based global feature generation step applies global average pooling and global max pooling on the depth map to produce channel-wise scaling and shifting parameters, which modulate the top-level fused feature before downsampling to create the extra coarse level. An occlusion-aware head predicts a continuous occlusion score in [0, 1] per object, computed during preprocessing from the spatial coverage of neighboring objects and their relative depth (a neighbor that overlaps and is closer to the camera contributes to occlusion).

Tracking and de-duplication (DG-Track). Raw per-frame detections are refined with a two-stage filter (confidence threshold, then post-NMS ROI overlap), keeping each candidate's box, decoder feature, sampled depth, and occlusion probability. Online tracking (based on ObjectTracker) computes an association cost from the negative log-posterior of a match, combining position distance, appearance cosine distance, center-depth difference, and an IoU-based geometric prior, with weights tied to modality reliability; the cost matrix is normalized and solved with the Hungarian algorithm, accepting only matches below a cost threshold. Trajectories are then validated by a support score that combines the number of hit frames against an occlusion-adaptive minimum hit count with local hit density against a frame-density threshold; a trajectory is valid if the score is at least 1, and the number of valid trajectories is the final count.

Training. The model is initialized from CountGD_Box with the pretrained backbone frozen; the depth encoder, cross-attention fusion module, and detection head are optimized. The total loss combines classification and GIoU losses with an L1 bounding-box regression loss (to improve localization under dense overlap) and a BCE occlusion loss. Optimization uses AdamW with batch size 1 and standard scaling, cropping, and flipping augmentation.

Why This Matters

• Research impact: The work extends open-vocabulary counting from images into the video domain with a modality (depth) that prior video counting datasets do not provide, and it reframes the task's core difficulty as both intra-frame separation of visually ambiguous neighbors and video-level cross-frame de-duplication. The released RGBD-VideoCount benchmark (2,032 finely annotated frames, 77,638 instance boxes, 6 categories) targets depth-assisted detection, association, and de-duplication, while allowing multiple object categories to coexist in one video, which the authors describe as closer to real shelf and inventory scenarios.
• Real-world applications:
  • Shelf inventory checks and automated out-of-stock warnings from dynamic video streams.
  • Warehouse inspections where counts of stored goods must be verified automatically.
  • Retail or storage scenarios involving dense, similarly packaged products where RGB alone cannot separate overlapping items.
  • Robotic or camera-equipped inspection systems that already carry an RGB-D sensor such as an Intel RealSense D455.
• Industry relevance: Counting from video is framed as a prerequisite for automated inventory management; the framework is built on existing open-vocabulary detectors and released with code and dataset, lowering the barrier to adoption. The comparison against DepthMOT and DepTR-MOT shows that depth alone is not the source of the gain — effective cross-modal fusion and occlusion-aware de-duplication are.

Future Directions

• Improving pseudo-depth for deployment without depth sensors: The gap between real depth (test MAE 7.97) and pseudo-depth (test 13.75) is attributed to edge smoothing; sharper monocular depth estimation is the obvious lever.
• Handling sensor noise and missing depth: Since Gaussian noise (MAE 11.16) and Salt & Pepper noise (MAE 10.66) hurt more than Edge Blur (8.50) or Random Holes (8.23), robustness to corrupted local depth ordering in the fusion bias is an open problem.
• Extending to more categories and scenes: RGBD-VideoCount covers 6 categories (beverage, bag, book, plant, box, cabinet) with per-category video counts of 94, 8, 42, 12, 55 and 42; broader category coverage and generalization beyond shelves and warehouses remain untested.
• Understanding partial-depth behavior: Partial depth improves MAE but can cause RMSE fluctuations from cross-modal inconsistency, which the paper reports but does not fully explain; also, the occlusion score construction and several training/implementation details are deferred to supplementary material.

Target Audience

• Researchers in computer vision working on object counting, open-vocabulary detection, and multimodal RGB-D perception.
• Multi-object tracking researchers interested in depth-aware association costs and occlusion-adaptive trajectory validation.
• Practitioners building automated inventory, shelf-auditing, or warehouse inspection systems from video.
• Dataset and benchmark developers looking for a depth-equipped video counting benchmark with multi-category sequences, instance boxes, exemplar boxes, and ROIs.

Authors’ abstract

Our primary objective is to advance video object counting in crowded scenes, aiming to robustly count all instances of a target category based on given text or visual prompts. Existing methods rely on RGB information, limiting their discriminative ability in crowded and occluded conditions. To address this, we propose a Depth-Guided Detector (DG-Det) along with a general post-processing pipeline. By integrating depth cues with multi-scale RGB-D cross-attention and explicit occlusion prediction, our method enhances spatial understanding and achieves robust detection in crowded and occluded scenes. Furthermore, we introduce a unified de-duplication framework to eliminate cross-frame redundant counting. To facilitate future research, we also release a new RGB-D Video Object Counting dataset featuring depth information and multiple object categories persequence. Extensive experiments demonstrate that our method achieves a 62.01\% reduction in MAE compared to existing baselines, and also produces consistent improvements in RMSE. We provide the source code at https://github.com/streamer-AP/DG-Net and the dataset at https://huggingface.co/datasets/aerospace123/RGBD-VideoCount.