Neighbor-aware Instance Refining with Noisy Labels for Cross-Modal Retrieval

AI summary

Overview

• Research area: Cross-modal retrieval (CMR) with noisy labels — learning image-text representations that stay accurate when the training annotations contain errors.
• Technical level: Advanced. The paper assumes familiarity with cross-modal embedding spaces, contrastive/margin losses, Wasserstein barycenters, and Expectation-Maximization.
• Scope in one sentence: The paper proposes NIRNL, a framework that combines a margin-based representation regularizer with a neighborhood-consistency partitioner that splits training data into pure, hard, and noisy subsets, each optimized with its own loss, and evaluates it on three benchmarks under noise rates of 0.2, 0.4, 0.6, and 0.8.

What This Paper Is About

Cross-modal retrieval models normally assume that every image-text pair is labeled correctly, but real web-scale data contains mislabeled and semantically ambiguous pairs, and this noise degrades retrieval accuracy. Existing fixes fall into three families — robust losses, label correction, and filtering out noisy samples — and the authors argue each one sacrifices something: robust losses tolerate noise without raising the performance ceiling, label correction can introduce new errors, and sample selection wastes data and is threshold-sensitive. NIRNL's goal is to combine all three paradigms so that model performance ceiling, calibration reliability, and data utilization are balanced at the same time.

Key Contributions

1. A unified robust cross-modal learning framework (NIRNL) that brings together robust learning, label calibration, and instance selection. The authors state this is the first work to balance model performance ceiling, calibration reliability, and data utilization rate in multi-modal learning with noisy labels.
2. Neighbor-aware Instance Refining (NIR), which dynamically partitions training instances into pure, hard, and noisy subsets by perceiving the global neighborhood distribution, using cross-modal neighborhood consensus rather than small-batch small-loss criteria.
3. Customized optimization strategies for each subset — cross-entropy for pure, instance-weighted cross-entropy for hard, and label correction plus Mean Absolute Error for noisy — so that all available data is exploited while error propagation is limited.
4. Cross-modal Margin Preserving (CMP), which constrains the relative distance between positive and negative pairs so that positive-pair similarity exceeds negative-pair similarity by a predefined margin, improving representation discriminability.

Main Findings

• Best MAP across all three benchmarks: On Wikipedia, NIRNL reaches a mean MAP of 46.6 versus 44.9 for the strongest baseline (RONO). On XMedia, NIRNL reaches a mean of 91.8 versus 88.6 for RSHNL and 89.5 for RONO. On INRIA-Websearch, NIRNL reaches a mean of 52.0 versus 49.5 for RSHNL.
• Robustness holds at the highest noise rate: At a noise rate of 0.8 on Wikipedia, NIRNL scores 41.7 (I2T) and 39.4 (T2I), while several baselines collapse far lower (for example DRCL at 16.1 / 14.7 and DHRL at 24.0 / 24.5).
• Unsupervised baselines are noise-immune but capped: DGCPN, CIRH, and UCCH do not use supervision so their scores do not change with the noise rate, yet their Wikipedia mean of 29.2, 23.1, and 38.0 respectively sits below robust supervised methods.
• Supervised baselines are noise-sensitive: ALGCN, GNN4CMR, and DRCL degrade sharply as the noise rate rises; on Wikipedia, ALGCN falls from 41.9 / 40.1 at 0.2 to 13.4 / 14.1 at 0.8.
• Precision-recall curves favor NIRNL: Under the 0.6 noise rate, the NIRNL curves in Figure 2 lie above those of other methods on all three datasets, meaning higher retrieval precision at the same recall.
• Ablation shows every component matters (0.6 noise rate): Removing CMP (NIRNL-1) causes the largest drop, to a Wikipedia mean of 24.4, XMedia 40.8, and Websearch 8.3. Discarding the noisy subset (NIRNL-2) gives Wikipedia 44.8, XMedia 88.6, Websearch 46.7. Treating hard samples as pure without loss weighting (NIRNL-3) gives Wikipedia 47.1, XMedia 90.3, Websearch 51.1. Removing barycenter alignment (NIRNL-4) gives Wikipedia 40.5, XMedia 90.8, Websearch 50.4.
• Clean samples are progressively absorbed: Figure 4 tracks instance counts over epochs at 0.6 noise and shows that NIRNL incorporates the majority of clean samples into training over time.
• NIRNL outperforms RSHNL over training: Figure 3 shows NIRNL consistently ahead of RSHNL on all three datasets; the authors note RSHNL shows signs of overfitting to noisy labels on Wikipedia and underperforms on XMedia and INRIA-Websearch.

Methodology in Plain English

The model learns two separate encoders — one for images, one for text — that map both modalities into a shared semantic space where similarity can be measured directly. On top of this basic setup, the paper adds two modules that run in parallel.

The first, Cross-modal Margin Preserving (CMP), is a hinge-style constraint applied to all samples: for each anchor, its similarity to its true match must exceed its similarity to a non-matching sample by at least a margin M. This pulls true pairs together and pushes mismatched pairs apart, making the embedding space better organized regardless of noise.

The second, Neighbor-aware Instance Refining (NIR), decides how much to trust each training pair. For every image and every text, it looks at the K nearest neighbors in that modality's feature space and builds a soft label by counting how many neighbors belong to each class. It then compares that soft label's argmax with the given ground-truth label, separately for the image side and the text side. If both modalities agree with the given label, the pair is pure. If only one agrees, it is hard. If neither agrees, it is noisy.

Each subset gets a different treatment. Pure pairs are trained with standard cross-entropy against class centers ("semantic barycenters," computed as a 2-Wasserstein barycenter per class and solved with Expectation-Maximization). Hard pairs get the same cross-entropy but weighted by a per-instance factor so more trustworthy pairs count more. Noisy pairs have their labels replaced: the soft label distributions from the two modalities are fused, the highest-scoring class becomes the corrected label, and the pair is trained with the noise-tolerant Mean Absolute Error loss. The total objective sums the three subset losses plus a weighted CMP term, where the subset losses apply only to their own subsets and CMP applies to everything.

Why This Matters

• Research impact: The paper argues that robust learning, label calibration, and instance selection each have structural weaknesses in isolation, and demonstrates that combining them — with a tri-partition rather than a binary clean/noisy split — improves both accuracy and robustness at high noise rates. This challenges the common practice of simply discarding suspected noisy samples.
• Real-world applications:
  • Web-scale image-text search, where captions and alt-text are frequently wrong or ambiguous.
  • Product catalog retrieval in e-commerce, where seller-supplied titles and categories are inconsistent.
  • Medical or scientific image-text archives assembled from heterogeneous sources with variable annotation quality.
  • Video-image retrieval pipelines, which the paper cites as a target scenario alongside image-text retrieval.
• Industry relevance: The method is a training-time technique that sits on frozen backbones, so it can be added to an existing retrieval pipeline without retraining the feature extractors. Any organization building cross-modal search from scraped or crowdsourced data faces exactly the labeling-noise problem this paper addresses. The code is released at https://github.com/perquisite/NIRNL.

Future Directions

• The paper's conclusion does not specify a future work plan, so the following are open questions the work raises rather than stated author intentions.
• Scalability of the neighborhood step: NIR requires computing K-nearest neighbors and per-class Wasserstein barycenters with EM. Whether this remains practical at the scale of the full INRIA-Websearch set (over 70,000 pairs, 353 categories) rather than the 14,698-pair subset used here is not reported.
• Noise type generality: All experiments use symmetric label noise at rates 0.2 through 0.8. Behavior under asymmetric or instance-dependent noise, which is more typical of real annotation error, is not reported.
• Sensitivity to key hyperparameters: The margin M, the balance coefficient α, the number of neighbors K, and the regularization coefficient λ are all introduced but the paper does not report a sensitivity analysis for them.
• Extension beyond image-text: The formulation is written generically over two modalities, but only image-text retrieval is evaluated; the paper does not report results for the video-image or other modality pairings it cites as motivation.

Target Audience

Researchers and graduate students working on cross-modal retrieval, learning with noisy labels, or weakly supervised multimodal learning. It will also interest practitioners building production image-text search systems from imperfectly labeled web data, and readers already familiar with robust loss design, sample selection, or label correction who want to see these three families unified into one framework. The dense notation and probabilistic machinery (Wasserstein barycenters, EM, hinge formulations) make it a poor fit for beginners.

Authors’ abstract

In recent years, Cross-Modal Retrieval (CMR) has made significant progress in the field of multi-modal analysis. However, since it is time-consuming and labor-intensive to collect large-scale and well-annotated data, the annotation of multi-modal data inevitably contains some noise. This will degrade the retrieval performance of the model. To tackle the problem, numerous robust CMR methods have been developed, including robust learning paradigms, label calibration strategies, and instance selection mechanisms. Unfortunately, they often fail to simultaneously satisfy model performance ceilings, calibration reliability, and data utilization rate. To overcome the limitations, we propose a novel robust cross-modal learning framework, namely Neighbor-aware Instance Refining with Noisy Labels (NIRNL). Specifically, we first propose Cross-modal Margin Preserving (CMP) to adjust the relative distance between positive and negative pairs, thereby enhancing the discrimination between sample pairs. Then, we propose Neighbor-aware Instance Refining (NIR) to identify pure subset, hard subset, and noisy subset through cross-modal neighborhood consensus. Afterward, we construct different tailored optimization strategies for this fine-grained partitioning, thereby maximizing the utilization of all available data while mitigating error propagation. Extensive experiments on three benchmark datasets demonstrate that NIRNL achieves state-of-the-art performance, exhibiting remarkable robustness, especially under high noise rates.