CountGD++: Generalized Prompting for Open-World Counting

AI summary

Overview

Research area: Computer vision — open-world object counting, multi-modal prompting, and vision-language models.

Technical level: Advanced. The paper assumes familiarity with transformer detection architectures (Grounding DINO, Swin Transformer, BERT), focal loss, Hungarian matching, and RoIAlign.

Scope in one sentence: The paper introduces CountGD++, a counting model that accepts positive and negative text and visual prompts, automatically derives visual exemplars ("pseudo-exemplars"), accepts exemplars from external natural or synthetic images, and can serve as a counting expert agent for an LLM.

What This Paper Is About

Open-world counting models let a user say what to count, using text or by drawing boxes around example objects in the image. Two things were impossible before this work: saying what not to count, and reusing a visual example across many images without re-annotating each one. The paper's goal is to expand prompt flexibility so users can specify targets with positive and negative text and visual examples, have visual examples generated automatically, and draw those examples from images outside the one being counted.

Key Contributions

1. Negative prompting for counting. CountGD++ accepts any number of negative text prompts and negative visual exemplars alongside positive ones. Negatives act as filters during inference: an object query is only counted if its highest positive similarity exceeds the confidence threshold and exceeds its maximum similarity to all negative prompts. This is trained with a focal loss over a query-prompt similarity matrix.

2. Pseudo-exemplars. The model's own top-scoring output boxes are cast as visual exemplars and fed back for a second forward pass, automating exemplar annotation from text alone. When both positive and negative text are available, both positive and negative pseudo-exemplars can be produced. In videos, pseudo-exemplars from the current frame become dynamic exemplars for the next frame, letting the visual description evolve as objects deform.

3. External exemplars. The input image and the exemplar image are processed in separate streams rather than one unified stream, so exemplars can come from natural or synthetic images different from the image being counted. A user annotates one image once and applies it to an entire dataset.

4. CountGD++ as an LLM vision expert agent. Three pipelines are described: generating synthetic exemplars via an image-generation API; an iterative image agent that refines counts by re-feeding high-confidence boxes; and an iterative video agent that updates exemplars frame by frame.

Main Findings

• Pseudo- and synthetic exemplars improve text-only counting on FSCD-147. With text only, CountGD++ (Ours_t) reaches MAE 16.55 / RMSE 129.76. Adding pseudo-exemplars (Ours_t+p) gives MAE 10.29 / RMSE 33.52 with AP 37.78 and AP50 68.90. Adding synthetic exemplars as well (Ours_t+p+s) gives MAE 8.39 / RMSE 27.03 with AP 38.93 and AP50 71.35. For comparison, CountGD scores MAE 12.98 / RMSE 98.35 and CountSE scores MAE 7.84 / RMSE 82.99 (CountSE does not output boxes; the paper notes it has similar MAE but much higher RMSE).

• Negative prompts produce large gains on Blood Cell Detection. Without negatives, CountGD++ scores MAE 11.56 / RMSE 15.69 (positive prompts only) and 11.62 / RMSE 15.84 (external positive exemplar). With negative text plus internal negative exemplars, MAE drops to 1.73 / RMSE 3.06, AP 0.46, AP50 0.71. With external negative exemplars, MAE 1.52 / RMSE 2.42, AP 0.54, AP50 0.80. CountGD scores MAE 10.99 / RMSE 14.64; CountGD-Box scores MAE 11.34 / RMSE 15.42, AP 0.25, AP50 0.45.

• Negative prompts also transform OmniCount (Fruits) results. CountGD scores MAE 2.76 / RMSE 3.11; CountGD-Box scores MAE 2.83 / RMSE 3.15 with AP 0.47 / AP50 0.61. CountGD++ with internal negatives reaches MAE 0.41 / RMSE 1.51, AP 0.62, AP50 0.83; with external negatives, MAE 0.49 / RMSE 1.63, AP 0.60, AP50 0.80.

• State-of-the-art crowd counting on ShanghaiTech with text plus pseudo-exemplars. Part A: MAE 116.0 / RMSE 234.0. Part B: MAE 28.0 / RMSE 50.0. CountGD-Box scores 132.2 / 253.9 (A) and 32.2 / 57.9 (B); CountSE scores 129.7 / 258.3 (A) and results on Part B are not available; CLIP-Count scores 192.6 / 308.4 and 45.7 / 77.4; Grounding DINO 394.9 / 537.5 and 58.3 / 99.3; OWLv2 420.2 / 553.3 and 81.5 / 126.5. The paper states this reduces CountSE's MAE by over 10 percent and RMSE by 9 percent on Part A.

• PrACo: negatives nearly eliminate false positives when the target is absent. On the Negative Label Test, CountGD++ with positive and negative text achieves NMN 0.07, PCCN 97.99, CntP 0.90, CntR 0.96, versus 0.88, 62.86, 0.86, 0.96 with positive text only. DAVE with negatives scores NMN 0.08, PCCN 97.62, CntP 0.84, CntR 0.80; TFPOC scores NMN 0.75, PCCN 66.04, CntP 0.69, CntR 0.85.

• Pseudo-exemplars outperform manually annotated exemplars on video crystals. On Science-Count (Crystals), CountGD++ reaches MAE 10 / RMSE 12.3, versus CountVid with manually annotated exemplars at MAE 12 / RMSE 13.5 and CountVid text-only at MAE 69.1 / RMSE 86. In the text-only setting the paper reports MAE and RMSE divided by a factor of about 7. The paper attributes the win over manual exemplars to evolution over time and to using up to 10 pseudo-exemplars per frame rather than the 3-8 given to CountVid.

• External exemplars generalize. The paper reports that external exemplars may generalize better than internal ones when they depict the object more clearly or under more representative conditions, and that CountGD++ given only positive text and exemplars generally matches CountGD and CountGD-Box on counting while outperforming CountGD-Box on detection.

Methodology in Plain English

CountGD++ extends CountGD-Box, which itself extends Grounding DINO. A Swin Transformer image encoder processes three kinds of input with shared weights: the image being counted, the positive exemplar image, and each negative exemplar image. The user's positive and negative exemplars are turned into feature vectors by RoIAlign, and the text goes through a BERT-base text encoder in the format "positive . negative1 . negative2 ." In the Feature Enhancer, visual exemplar tokens and text tokens of the same class attend to each other, but prompts for different classes do not attend to each other — an explicit design choice the authors ablate. Cross-attention then fuses the prompt features with the image patch tokens. The top k = 900 image tokens most similar to the prompts become cross-modality queries and pass through a 6-block decoder to produce 900 object queries. An Object Filtering Module keeps only queries that pass two tests: a sigmoid similarity to the positive prompt above a confidence threshold, and a higher similarity to the positive prompts than to any negative prompt. The rest are enumerated to give the count, and boxes come from an MLP regression head.

For training, the model needs images with multiple labeled object categories, which standard counting datasets lack. The authors synthesize them by applying mosaic construction to create images with several categories. The classification term is a focal loss over the matrix of sigmoid similarity scores, with target entries set to 1 for matching query-prompt pairs and 0 otherwise, which pushes queries away from prompts they do not correspond to. Localization adds center, height/width, and generalized IoU terms. Loss weights are λ_loc = 5, λ_GIoU = 2, λ_cls = 2, borrowed from CountGD-Box with no further tuning. Training uses FSC-147 plus 1000 synthetic mosaic images; no fine-tuning is done on any other dataset. A confidence threshold of σ = 0.23 is used everywhere without optimization.

Pseudo-exemplars are simple in operation: run the model on text, take the top N scoring boxes (N = 3 in the standard FSC-147 setting, where three exemplars are annotated per image), and feed them back with the text for a second pass. With both positive and negative text, positive pseudo-exemplars come from queries closer to the positive prompt and negative pseudo-exemplars from queries closer to the negative prompt.

Why This Matters

Impact on research. The paper reframes what a counting prompt can be — no longer just "what to count" but also "what not to count," where examples come from, and whether they are annotated by hand at all. It supplies a concrete mechanism (contrastive query-prompt filtering with focal loss) for negative prompting in dense prediction, and it adds counting to the list of tasks an LLM can delegate to a specialized vision tool.

Real-world applications named in the paper:

• Counting different blood cell types for medical diagnosis.
• Measuring the formation rate of growing crystals in x-ray videos to develop more sustainable materials.
• Distinguishing ripe from unripe fruits in agriculture.
• Crowd counting in dense scenes, and counting vehicles in aerial drone imagery (CARPK).

Industry relevance. The external-exemplar capability converts a per-image annotation cost into a one-time cost, which matters for anyone deploying counting at dataset scale. Automated pseudo-exemplars remove annotation from the loop entirely for text-only workflows. The agent pipelines let an LLM orchestrate counting as one tool among several, which is how many production multi-modal systems are being assembled. Code is released at https://github.com/niki-amini-naieni/CountGDPlusPlus/.

Future Directions

• Reducing the dependence on the LLM controller. Synthetic exemplars in the FSCD-147 experiments were generated using GPT-5 as the LLM in the pipeline; how much the pipeline degrades with a smaller or open controller is not reported.
• Handling negative prompts that overlap with positives. The design prevents prompts from different negative classes attending to each other. How the model behaves when negative classes are visually similar to each other or ambiguously defined is not reported.
• Extending dynamic exemplars beyond two forward passes per frame. The video setup uses 10 pseudo-exemplars per frame carried forward; whether error accumulation over long videos is a problem is not reported.
• Broader benchmark coverage. The paper states results on PairTally and CARPK are in the appendix, but the provided content truncates the PairTally appendix table after GeCo, so CARPK figures and the full PairTally comparison are not available here.

Target Audience

Researchers and engineers working on object counting, open-vocabulary detection, and vision-language prompting; practitioners who need to count a specific object class across many images or video frames without annotating each one; and teams building LLM agent systems that need a reliable counting tool. Readers should be comfortable with transformer-based detection architectures and standard counting metrics (MAE, RMSE, AP, AP50), though the core ideas — negative prompts, pseudo-exemplars, and external exemplars — are explainable without that background.

Authors’ abstract

The flexibility and accuracy of methods for automatically counting objects in images and videos are limited by the way the object can be specified. While existing methods allow users to describe the target object with text and visual examples, the visual examples must be manually annotated inside the image, and there is no way to specify what not to count. To address these gaps, we introduce novel capabilities that expand how the target object can be specified. Specifically, we extend the prompt to enable what not to count to be described with text and/or visual examples, introduce the concept of `pseudo-exemplars' that automate the annotation of visual examples at inference, and extend counting models to accept visual examples from both natural and synthetic external images. We also use our new counting model, CountGD++, as a vision expert agent for an LLM. Together, these contributions expand the prompt flexibility of multi-modal open-world counting and lead to significant improvements in accuracy, efficiency, and generalization across multiple datasets. Code is available at https://github.com/niki-amini-naieni/CountGDPlusPlus.