CountFormer: A Transformer Framework for Learning Visual Repetition and Structure in Class-Agnostic Object Counting

AI summary

Overview

• Research area: Computer vision, specifically class-agnostic and exemplar-free object counting using density regression on the FSC-147 benchmark.
• Technical level: Intermediate. The paper assumes familiarity with vision transformers, self-supervised pretraining, density-map counting, and standard counting metrics (MAE, RMSE).
• Scope: The paper studies whether swapping in a self-supervised foundation encoder (DINOv2) plus explicit 2D positional embeddings, inside an otherwise unchanged CounTR-style density-regression pipeline, changes structural consistency when counting objects with no exemplars provided at test time.

What This Paper Is About

Counting models that generalize to unseen object classes usually rely on example patches (exemplars) or text prompts given at test time. This paper asks what happens when you remove both and rely purely on the image encoder's representations. The authors take a standard density-regression counting framework, replace the image encoder with the self-supervised DINOv2 foundation model, fuse in explicit two-dimensional positional embeddings, and decode with a lightweight convolutional network whose summed output is the final count.

Key Contributions

1. A controlled integration of a DINOv2 vision transformer into a standard exemplar-free density-regression framework that is architecturally aligned with CounTR, with no change to the loss formulation or the inference protocol.
2. A simple two-dimensional positional embedding fusion step applied before density decoding, intended to give transformer token features explicit spatial grounding while keeping the architecture lightweight and computationally comparable to prior approaches.
3. Beyond standard MAE and RMSE reporting on FSC-147, the paper adds qualitative analysis of part-level overcounting in structurally complex objects, plus a diagnostic sensitivity analysis of extreme dense-scene cases.
4. Explicit characterization of observed failure modes in highly dense scenes with weak inter-object boundaries, and clarification of how sensitive evaluation metrics are to a few extreme samples, without altering the official benchmark results.

Main Findings

• Competitive but not state-of-the-art quantitative results: On the official FSC-147 split, CountFormer reaches MAE 19.06 and RMSE 118.45 on the test set, and MAE 19.11 and RMSE 65.58 on the validation set. For comparison, the table lists CounTR at 14.71 MAE / 106.87 RMSE (test) and RCC at 17.12 MAE / 104.53 RMSE (test), so the proposed method does not beat those baselines on aggregate error.

• Structural grouping improves on composite objects: In the "glasses" example reported in Figure 5, the ground truth count is 96, CounTX predicts 185, and the proposed model predicts 98. The authors attribute this to DINOv2 features activating on whole-object instances rather than on individual sub-components such as lenses.

• Smoother density maps: The generated density maps show smoother spatial distributions, whereas CounTX produces activations that more closely resemble point-level responses, according to the qualitative comparison in Figure 4.

• Metrics are dominated by a few extreme scenes: Excluding four extreme high-density cases for analysis only drops test MAE from 19.06 to 13.14 and test RMSE from 118.45 to 33.05, and drops validation MAE from 19.11 to 15.64 and RMSE from 65.58 to 48.12. The authors emphasize these four images are not excluded from the official benchmark evaluation.

• Systematic undercounting in dense scenes: In the failure case in Figure 6, the ground truth is 512 objects and the model predicts 400. The authors link this to overlapping density activations where objects lack clear boundaries.

• A well-counted dense example is also reported: Figure 2 shows an input image with a ground truth of 149 objects for which the model's density map summed to a prediction of 149.

• Apparent reporting inconsistency: Figure 4(c) lists ground truth 96, CounTX 185, and the proposed model 122, while Figure 5, describing the "glasses" example, lists the same ground truth and CounTX values but a proposed-model prediction of 98. The paper does not reconcile these two numbers.

• Additional qualitative comparisons: Figure 4 also reports ground truth 21 / CounTX 15 / proposed 21; ground truth 118 / CounTX 82 / proposed 118; ground truth 48 / CounTX 50 / proposed 48; ground truth 79 / CounTX 143 / proposed 83; and ground truth 17 / CounTX 15 / proposed 17.

Methodology in Plain English

The input image goes through a pretrained DINOv2 vision transformer, which acts as a frozen-style feature extractor trained self-supervised (through knowledge distillation across multiple image views) rather than on counting labels. The resulting token features are added to explicit two-dimensional positional embeddings so the model retains a sense of where each token sits in the image. Those spatially grounded features are reshaped into 2D maps and passed through a convolutional decoder with four upsampling stages, ending in a linear layer that produces a single-channel density map at the original image resolution. Summing the values in that density map gives the object count. Training uses images resized to 256 x 256 and center-cropped to 224 x 224, with horizontal and vertical flips, random rotations within 30 degrees, and normalization. Optimization uses AdamW with batch size 8 and learning rate 6.25e-6 on an RTX 3070 Ti Mobile GPU, training for 200 epochs and selecting the checkpoint with the lowest validation MAE. At inference, a square sliding window with a stride of 128 pixels scans the image, with the overlapping-region handling taken from the CounTR procedure.

Evaluation uses the FSC-147 dataset, which contains 6135 images across 147 object classes, with 7 to 3731 objects per image and an average of 56. Performance is measured with Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) computed over predicted versus ground-truth counts.

Why This Matters

Impact on research. The paper reframes a counting result as a representation question rather than an architecture question: it holds the loss and inference protocol fixed and changes only the encoder, which isolates how foundation-model features affect part-whole coherence. It also makes a methodological point that aggregate MAE and especially RMSE can be dominated by a handful of extreme dense images, which matters when comparing counting methods on FSC-147 and similar benchmarks. The reported numbers on model parameters, inference speed, and cross-dataset transfer are not reported here.

Real-world applications.

• Counting items in retail or warehouse scenes (the paper's examples include Lego pieces, pens, and shirts) where the object class is not known in advance.
• Estimating crowds or gatherings where no labeled exemplars are available.
• Biological or scientific surveys such as counting birds, one of the categories visualized in the paper.
• Inventory or quality-control checks on densely packed manufactured parts, where the paper's failure analysis on small, similar, weakly separated objects is directly relevant.

Industry relevance. Because the method is exemplar-free and adds only a positional fusion step and a lightweight convolutional decoder on top of an existing pretrained backbone, it fits deployment settings where annotating examples per new object type is impractical. The paper's honest finding that dense, weakly separated objects cause systematic underestimation identifies the operating condition under which such a system should not be trusted without additional resolution or multi-scale handling.

Future Directions

1. Raising the input resolution above the 224 x 224 crop used here, since the authors attribute suppressed boundary detail partly to input resolution.
2. Multi-scale aggregation to reduce overlapping density activations and the resulting underestimation in densely packed scenes.
3. Broader cross-dataset evaluation beyond FSC-147 to test whether the structural benefits of foundation features generalize.
4. Clarifying or correcting the discrepancy between the reported proposed-model predictions on the "glasses" example in Figure 4(c) and Figure 5, and investigating annotation granularity ambiguity (for example stud-level versus piece-level counting) that the authors say disproportionately affects RMSE.
5. Testing whether architectural changes, which the authors deliberately avoided in order to isolate representation effects, could close the remaining MAE/RMSE gap to CounTR and RCC.

Target Audience

Researchers and graduate students working on class-agnostic, few-shot, or zero-shot object counting; practitioners evaluating whether self-supervised foundation encoders such as DINOv2 are worth substituting into an existing density-regression pipeline; and benchmark-focused readers interested in how sensitive FSC-147 MAE and RMSE are to a small number of extreme dense images. Readers looking for a new state-of-the-art counting architecture or a novel loss function will not find one here, since the paper positions itself as a controlled study of representation effects.

Authors’ abstract

Humans can often count unfamiliar objects by observing visual repetition and composition, rather than relying only on object categories. However, many exemplar-free counting models struggle in such situations and may overcount when objects contain symmetric components, repeated substructures, or partial occlusion. We introduce CountFormer, a controlled adaptation of a density-regression framework inspired by CounTR, where the image encoder is replaced with the self-supervised vision foundation model DINOv2. The resulting transformer features are combined with explicit two-dimensional positional embeddings and decoded by a lightweight convolutional network to produce a density map whose integral gives the final count. Our goal is not to propose a new counting architecture, but to study whether foundation-based representations improve structural consistency under a strictly exemplar-free setting. On FSC-147, CountFormer achieves competitive performance under the official benchmark (MAE 19.06, RMSE 118.45). Qualitative analysis suggests fewer part-level overcounting errors for some structurally complex objects, while overall error remains broadly consistent with prior approaches. Sensitivity analysis shows that evaluation metrics are strongly affected by a small number of extreme high-density scenes. Overall, the results highlight the role of representation quality in exemplar-free object counting.