Bipartite Mode Matching for Vision Training Set Search from a Hierarchical Data Server

AI summary

Overview

Research area: Computer vision and data-centric transfer learning — specifically, searching a large "data server" for a training set that matches an unlabeled target domain, evaluated on object re-identification (re-ID) and object detection.

Technical level: Intermediate. The paper assumes familiarity with clustering, Fréchet Inception Distance (FID), graph/bipartite matching, and unsupervised domain adaptation (UDA).

Scope: The paper proposes a hierarchical data server plus a bipartite mode matching (BMM) algorithm that selects source clusters to build target-specific training sets, and compares them against random selection, NDS, SnP, TL;DR, and CCDR on four re-ID targets and two detection targets.

Authors: Yue Yao (Shandong University, China), Ruining Yang, Tom Gedeon; affiliations listed as Shandong University, China; Northeastern University, United States; Curtin University, Australia. Code is released at https://github.com/yorkeyao/BMM. arXiv:2601.09531v1 [cs.CV], 14 Jan 2026, CC BY 4.0.

What This Paper Is About

The paper tackles a situation where the target domain is accessible but real-time annotation of it is not feasible, so a substitute labeled training set must be pulled from a large pre-existing data server. The core problem is that target domains contain distinct "modes" (semantic clusters describing the data distribution), and if the selected training set lacks those modes, model performance is compromised.

The goal is to build a training set from the server whose mode structure mirrors the target's, so a model trained on it transfers well to the target. Rather than iterating on the search algorithm alone, the authors optimize the structure of the data server itself by making it hierarchical, then match target modes to source modes one-to-one.

Key Contributions

1. A hierarchical data server design. Server images are featurized, split into J clusters with balanced k-means (so each mode at a level has consistent size: |C_k| = m_t/K), then merged bottom-up by agglomerative clustering into a dendrogram of all cluster sizes found during merging, yielding H modes. H is not a hyperparameter but a result of merging; for a full binary tree, H = 2J - 1.

2. A bipartite mode matching (BMM) framework. Target data is flat-clustered into L modes; every (source, target) mode pair is weighted by FID; the minimum-weight assignment is solved with the Hungarian algorithm, and the selected source clusters form the searched training set (duplicates removed).

3. Demonstration that the joint design beats existing training set search methods. On four re-ID targets (AlicePerson, Market, AliceVehicle, VeRi) and two detection targets (ExDark, Region100), BMM produces consistently smaller FID and higher rank-1/mAP/mAP@50 than random selection, NDS, SnP, TL;DR, and CCDR.

4. Evidence that BMM is orthogonal to model-centric UDA. Combining the searched training set with pseudo-labelling methods MMT (re-ID) and AT (detection) improves accuracy further, and an ablation shows the hierarchical server and mode matching only help when used together.

Main Findings

• Direct match with duplicates is worse than bipartite matching. On AlicePerson with 5% pruning, direct match with duplicates reaches FID 51.07, SSIM 15.85 and mAP 20.14; direct match without duplicates reaches FID 60.84, SSIM 21.07 and mAP 22.07; BMM reaches FID 51.93, SSIM 20.45 and mAP 26.08 (Table 2). Duplicate matches occurred in 4 out of 20 cases when targeting AlicePerson.

• Balanced k-means improves the hierarchical server. Baseline: FID 51.93, mAP 26.08. Balanced: FID 50.48, mAP 28.16 (Table 1), under 5% pruning targeting AlicePerson.

• Re-ID results at 5% pruning. AlicePerson: server FID 81.67; BMM FID 51.93, rank-1 49.28, mAP 26.08 versus Random 81.41/33.16/14.49, NDS 61.01/44.63/22.81, SnP 60.64/47.26/25.45, TL;DR 62.98/43.08/21.95, CCDR 60.52/48.47/25.04. Market: BMM FID 27.05, rank-1 53.03, mAP 28.39. AliceVehicle: BMM FID 21.96, rank-1 45.08, mAP 23.84. VeRi: BMM FID 15.98, rank-1 72.69, mAP 38.55.

• Re-ID results at 20% pruning. AlicePerson: BMM FID 51.53, rank-1 49.68, mAP 26.97. Market: 27.54/60.49/35.08. AliceVehicle: 21.64/47.34/26.18. VeRi: 15.72/75.36/42.05.

• Search without pruning already reduces the domain gap. "Searched via BMM" rows on the full server: AlicePerson FID 51.08, rank-1 50.63, mAP 27.09; Market 27.68/60.57/36.05; AliceVehicle 21.27/47.84/27.24; VeRi 15.32/77.19/42.18, versus the data server's 81.67, 37.53, 43.95 and 24.39 respectively.

• Detection results at 5% pruning (Table 4). ExDark: server FID 104.98; BMM FID 56.34, mAP 34.83, mAP@50 76.57, mAP@75 25.83 versus Random 105.74/23.50/56.13/14.49, NDS 64.35/30.34/65.34/22.18, SnP 59.78/32.15/67.18/25.69, TL;DR 66.25/28.32/65.11/22.07, CCDR 57.98/32.96/67.67/25.03. Region100: server FID 251.47; BMM FID 140.07, mAP 23.08, mAP@50 46.34, mAP@75 18.08 versus CCDR 142.48/22.25/44.49/19.68. The running text cites BMM reaching "an FID of 56.26, mAP of 42.23%, and mAP@50 of 81.69%", which differs from the Table 4 values; the paper does not reconcile the two.

• Hierarchical clustering removes the need for a hyperparameter "sweet point." With flat source clustering, both the source and target cluster numbers must be tuned to reach a working setting; with hierarchical source clustering, performance holds as long as the number of source clusters is not considerably small.

• Hyperparameter behaviour. Accuracy stabilizes at J = 128 source clusters; target accuracy plateaus at L = 10, and FID reaches its lowest value of 9.8 at L = 20, with mAP at 83.0%. Figure 4's caption states that L > 20 and > 200 source clusters (depth > 8) suffice for high accuracy; Figure 4 also fixes target cluster number at 20 when varying source clusters, and source cluster number at 128 when varying target clusters.

• Ablation. Replacing flat with hierarchical clustering alone gives no noticeable improvement, and swapping greedy search for mode matching while keeping flat clustering also gives no gain; only both together produce significant improvement over the baseline.

• Joint use with UDA raises accuracy sharply. Using MMT raises accuracy from 26.08% (direct transfer with the searched set) to 78.95%; when joint training with MMT, BMM moves from 76.98% to 78.95% compared to greedy search.

• Time complexity. Overall O(J^3), comprising O(L^2) for target flat clustering, O(J^2 log J) for source hierarchical clustering, and matching; per-target matching is O(log J * J * L). The hierarchical construction is a one-time offline process and need not be rebuilt for new target domains.

Methodology in Plain English

The authors treat the problem as matching two sets of clusters.

First, they featurize every image in the server with a feature extractor pre-trained on Imagenet, producing d-dimensional vectors. They run balanced k-means to partition the feature space into J equally sized clusters (each cluster center is called a "mode"), then merge the closest clusters step by step bottom-up (agglomerative clustering) until everything is one cluster. Every cluster that appears along the way — large or small — is kept as a usable mode, giving H modes at different levels of semantic granularity. This mirrors how web search engines organize pages hierarchically so relevant content can be retrieved quickly.

Second, they flat-cluster the unlabeled target set into L modes. Flat clustering is used on the target deliberately: hierarchical clusters at different levels would correlate with one another and make it hard to tell redundant modes apart.

Third, they build a complete undirected bipartite graph whose vertices are the H source modes and L target modes, and whose edge cost is the FID between the two modes' features. Since H > L (in practice H is set about two orders of magnitude larger than L), they solve the minimum-cost one-to-one assignment with the Hungarian algorithm. This prevents several target modes from collapsing onto the same source mode, which is what a naive greedy direct match does. The selected source clusters are concatenated, duplicates dropped, and optionally pruned to a target budget (5% or 20% of source IDs for re-ID; 5% of images for detection).

Models used: IDE (Zheng et al. 2016) for re-ID and RetinaNet (Lin et al. 2017) for detection, with direct transfer unless stated otherwise. The detection server combines seven datasets — ADE20K, COCO, BDD, CityScapes, DETRAC, Kitti, VOC — totaling 176,491 images; ExDark (7,363 low-light images) and the Region100 benchmark (footage from static cameras in 100 regions, first 70% for training and remaining 30% for validation and testing, used in the 2nd CVPR DataCV Challenge) serve as targets. Evaluation reports FID, rank-1, rank-5 and mAP for re-ID, and mAP, mAP@50 and mAP@75 at IoU 0.5 and 0.75 for detection.

Why This Matters

Impact on research: Prior training-set-search work iteratively improves the search algorithm; this paper shifts attention to the structure of the data server itself. It positions BMM as a data-centric method orthogonal to model-centric UDA, meaning the two can be stacked rather than competing. It also generalizes beyond re-ID — where the closest prior method, SnP, was tailored — to object detection.

Real-world applications named or implied by the paper:

• Autonomous driving, where annotating every driving scene is costly and target scenes vary.
• Large-scale surveillance systems, such as the static-camera footage from 100 world regions in Region100.
• Object re-identification for persons and vehicles, where appearance and filming conditions shift between cameras.
• Medical imaging and other expert-dependent domains where real-time annotation needs specialized expertise.

Industry relevance: The hierarchical server is built once offline and reused across target domains, with per-target matching at O(log J * J * L) — the paper argues this keeps deployment cost manageable. The method also lets an organization assemble a competitive labeled training set from data it already owns when fresh annotation is not feasible, and the released code lowers the barrier to trying it.

Future Directions

• Reconciling the detection figures: the running text and Table 4 report different BMM numbers for ExDark (FID 56.26 / mAP 42.23% / mAP@50 81.69% in text versus 56.34 / 34.83 / 76.57 in the table). Clarifying this, and verifying the Region100 mAP@75 result where CCDR (19.68) exceeds BMM (18.08), would sharpen the detection claims.

• Extending the hierarchical server beyond the two task families tested. The re-ID server composition is described only by reference to prior settings rather than by dataset list or image count, unlike the seven-dataset detection server, so broader benchmark coverage is an open step.

• Exploring how mode granularity could be chosen automatically. The paper notes semantic granularity is hard to control and that matching "apple" to "fruit" or to "painted apples" would be wrong; the hierarchy mitigates this but the level of merge is not learned per target.

• Studying the combination of BMM with a wider range of model-centric UDA and pseudo-labelling methods, since only MMT and AT are evaluated, and quantifying how much of the gain comes from the data side versus the adaptation side.

Target Audience

Researchers and practitioners working on unsupervised domain adaptation, data-centric machine learning, dataset selection/coreset construction, and training set search from large data pools. It is most useful to readers already comfortable with clustering, FID, and assignment problems, and to engineers building person or vehicle re-identification and object detection systems who need to assemble a training set from existing data rather than annotate a new target domain. Beginners can follow the motivation and results but will likely need background reading on domain gaps and bipartite matching.

Authors’ abstract

We explore a situation in which the target domain is accessible, but real-time data annotation is not feasible. Instead, we would like to construct an alternative training set from a large-scale data server so that a competitive model can be obtained. For this problem, because the target domain usually exhibits distinct modes (i.e., semantic clusters representing data distribution), if the training set does not contain these target modes, the model performance would be compromised. While prior existing works improve algorithms iteratively, our research explores the often-overlooked potential of optimizing the structure of the data server. Inspired by the hierarchical nature of web search engines, we introduce a hierarchical data server, together with a bipartite mode matching algorithm (BMM) to align source and target modes. For each target mode, we look in the server data tree for the best mode match, which might be large or small in size. Through bipartite matching, we aim for all target modes to be optimally matched with source modes in a one-on-one fashion. Compared with existing training set search algorithms, we show that the matched server modes constitute training sets that have consistently smaller domain gaps with the target domain across object re-identification (re-ID) and detection tasks. Consequently, models trained on our searched training sets have higher accuracy than those trained otherwise. BMM allows data-centric unsupervised domain adaptation (UDA) orthogonal to existing model-centric UDA methods. By combining the BMM with existing UDA methods like pseudo-labeling, further improvement is observed.