2D Gaussians Spatial Transport for Point-supervised Density Regression

AI summary

Overview

Research area: Computer vision, specifically point-supervised density regression and the use of optimal transport (OT) and 2D Gaussian Splatting in deep networks.

Technical level: Advanced. The paper builds on optimal transport theory (Wasserstein distance, Sinkhorn iterations), Bayesian transport plans, and differentiable 2D Gaussian Splatting.

Scope: The paper proposes Gaussian Spatial Transport (GST), a framework that pre-computes a fixed transport plan from an image and its point annotations using 2D Gaussian Splatting, replacing the iterative optimal transport computation used during training.

What This Paper Is About

Point-supervised density regression tasks, such as crowd counting and landmark detection, provide only object locations as annotation, not contours. Optimal transport methods match predicted density maps to these annotations well, but they require solving a costly inner-loop transport problem at every training iteration. This paper asks whether a good transport plan can be pre-computed once from the image itself, before any network training, so that the training loss becomes a single matrix multiplication.

Key Contributions

1. A spatial transport framework that decouples transport plan generation from network optimization: the plan is pre-computed as a fixed transport kernel, turning costly iterative OT into one efficient matrix multiplication during training.
2. A connection between Gaussian Splatting and spatial transport: a Bayesian framework (Theorem 1) derives the transport kernel from quantities obtained by 2D Gaussian Splatting, with kernel elements defined as a normalized conditional probability P(x_i | y_n).
3. Application of GST to representative computer vision tasks: crowd counting and landmark detection, with the code released at https://github.com/infinite0522/GST.
4. A supplementary mechanism for pixel-to-background correspondence, which assigns pixels far from the nearest foreground Gaussian to a virtual background object with density zero, avoiding the OT practice of assigning all pixels to targets.

Main Findings

• JHU-Crowd++ (crowd counting): GST reaches MAE 53.9 and MSE 225.4, the lowest reported in Table 1, compared with APGCC at 54.3/225.9, PET at 58.5/238, GL at 59.5/259.5, UOT at 60.5/252.7, DMCount at 61.6/256.1, Bayesian Loss (BL) at 75.0/299.9, and the direct-regression L2 Baseline at 81.7/304.5.
• UCF-QNRF (crowd counting): GST reaches MAE 80.7 and MSE 131.1. The paper reports MSE reductions of 17.2 against balanced OT (DMCount), and 11.2 and 16.4 against unbalanced OT (UOT and GL, respectively). APGCC reports 80.1/136.6, so GST is lower on MSE but slightly higher on MAE.
• NWPU (crowd counting): GST reaches MAE 74.4 and MSE 306.2, tying PET's MAE of 74.4 while having lower MSE than PET's 328.5, and outperforming MAN (76.5/323.0), ChfL (76.8/343), GL (79.3/346.1), UOT (87.8/387.5), DMCount (88.4/388.6), BL (105.4/454.2), and the L2 Baseline (126.2/528.2).
• MPII Human Pose (landmark localization): GST achieves the best mean PCKh@0.5 across all joints, outperforming the baseline HRNets by 0.6% on both HRNet-W32 (91.0 vs 90.4) and HRNet-W48 (91.1 vs 90.5), and with over 1.0% gains on shoulders, hips, knees, and ankles. The paper attributes weaker head performance (96.2 and 96.4) to annotation bias, since MPII head joints are annotated at the periphery rather than the center.
• Ablation on JHU++ with VGG backbone: L2 baseline 81.70 MAE / 304.50 MSE; heuristic Gaussian plan with empirical sigma = 8 gives 65.45/268.50; GST with pixel-to-annotation correspondence only gives 60.17/247.61; full GST with both pixel-to-annotation and pixel-to-background correspondence gives 58.30/239.58.
• Backbone consistency: GST surpasses OT on both VGG and Transformer architectures across all three counting datasets. On VGG, GST is 82.5/139.2, 58.3/239.6 and 80.3/325.7 versus OT's 87.2/150.8, 61.6/256.1 and 88.4/388.6 on UCF-QNRF, JHU++ and NWPU respectively.
• Runtime: On JHU++ with a VGG backbone, GST takes 15h32min of training versus 28h36min for OT, roughly halving OT's runtime on an NVIDIA 4090. The pre-computation step itself takes 3–7 seconds per high-resolution image, remaining under 10 seconds for resolutions up to 4K and over 4,500 annotated points.
• Complexity: Pre-computation costs O(pq) for p annotated points and q pixels, and occurs before training, reducing theoretical training complexity to O(1). OT instead costs O(kpq) per training step, with k = 100 in DMCount and k = 1,000 in HDM-HPE.
• Transport plan quality: Plans based on ground truth scales (56.60 MAE / 238.24 MSE) and on 2DGS-estimated scales (GST, 58.30/239.58) outperform both the OT plan (61.55/256.10) and the heuristic plan (65.45/268.50), showing GST estimates head scales from point annotations alone.
• Deformity elimination: Without the shape control regularization (delta = 1.5), over-elongated Gaussians cover irrelevant background regions and cause overlapping, ambiguous transport regions.
• Cut-off distance: The optimal cut-off distance d is 3, aligning with the 3-sigma principle in 2DGS rendering, and GST consistently outperforms OT within the 2.4 to 3.6 range.

Methodology in Plain English

Density regression models predict a density map over image pixels, but supervision is usually just a list of point locations. Optimal transport offers a principled way to compare the prediction against those points, but it must solve an optimization problem inside every training step.

GST changes the order of operations. First, it reconstructs the input RGB image with 2D Gaussian Splatting, representing the image as a set of anisotropic Gaussian ellipses with color and opacity. Some Gaussians are pre-assigned one-to-one to annotation points (the foreground set, whose size equals the number of annotations), while the rest model unannotated regions. A shape regularization keeps ellipses from becoming excessively elongated, and any pixel far from the nearest foreground Gaussian is assigned to a virtual background object with zero ground-truth density.

These Gaussians provide a way to estimate the conditional probability of a pixel given an annotation. A theorem shows that a valid transport plan can be written as a diagonal matrix of the source distribution multiplied by a "transport kernel," whose entries are normalized conditional probabilities. Because the annotations are uniform, this kernel simplifies to P(x_i | y_n) divided by its sum over annotations, so it becomes a fixed matrix that can be computed before training starts.

At training time, the predicted density map is pushed forward to annotation space by a single matrix multiplication with the pre-computed kernel. The loss is simply the L1 discrepancy between the transported density and the ground truth annotation map (the transport cost term is dropped because it is constant once the plan is fixed). This avoids the inner optimization loop entirely, while the training target is still aligned with the ideal counting objective.

Why This Matters

Impact on research. The paper argues that GST is a distinct paradigm from optimal transport rather than an approximation of it, because the transport plan is derived from the RGB image itself instead of from the network's density estimate. It also opens a new application for Gaussian Splatting, moving beyond image reconstruction, compression and super-resolution into spatial transport and pixel-to-annotation correspondence.

Real-world applications (mapped from the tasks evaluated in the paper):

• Crowd counting and crowd density monitoring for public safety and event management, on datasets such as UCF-QNRF, JHU-Crowd++ and NWPU.
• Human pose estimation and body-joint localization from single images, evaluated on MPII.
• General landmark localization tasks with point-level supervision only, where drawing full contours or heatmaps for every object is expensive.
• Large-scale object counting and dense environment analysis, where an image may contain thousands of targets and annotation budgets are limited.

Industry relevance. Because annotation cost is a practical bottleneck in dense counting tasks, a method that works from point annotations alone and roughly halves OT training time is attractive for production training pipelines. The reported pre-computation cost of 3–7 seconds per high-resolution image, staying under 10 seconds even at 4K resolution with over 4,500 points, suggests the approach is tractable on realistic imagery, and the authors note this could be reduced further with optimized Gaussian Splatting implementations.

Future Directions

• Replacing the hard binding of Gaussian kernels to annotations during splatting optimization with more flexible soft-assignment mechanisms, which the authors identify as a limitation.
• Incorporating explicit depth modeling: single-image 2D Gaussian Splatting lacks it, and the authors suggest extending to 3D Gaussians for sparse-view reconstruction to improve spatial reasoning in multi-view scenes.
• Validating GST in a broader range of applications beyond crowd counting and landmark detection.
• Further reducing the pre-computation runtime through optimized Gaussian Splatting implementations.

Target Audience

Researchers and practitioners working on point-supervised learning, density regression, crowd counting, and human pose or landmark localization, especially those already familiar with optimal transport losses such as DMCount, UOT, GL and Bayesian Loss. It is also relevant to anyone tracking new applications of Gaussian Splatting beyond rendering and reconstruction. Readers without background in OT or differentiable rendering will find the theoretical sections demanding, since the paper assumes familiarity with Wasserstein distance, marginal constraints and Sinkhorn-based solvers.

Authors’ abstract

This paper introduces Gaussian Spatial Transport (GST), a novel framework that leverages Gaussian splatting to facilitate transport from the probability measure in the image coordinate space to the annotation map. We propose a Gaussian splatting-based method to estimate pixel-annotation correspondence, which is then used to compute a transport plan derived from Bayesian probability. To integrate the resulting transport plan into standard network optimization in typical computer vision tasks, we derive a loss function that measures discrepancy after transport. Extensive experiments on representative computer vision tasks, including crowd counting and landmark detection, validate the effectiveness of our approach. Compared to conventional optimal transport schemes, GST eliminates iterative transport plan computation during training, significantly improving efficiency. Code is available at https://github.com/infinite0522/GST.