3rd Place Solution to ICCV LargeFineFoodAI Retrieval

AI summary

Overview

• Research area: Fine-grained image retrieval (computer vision), specifically food image retrieval, presented as a competition solution paper.
• Technical level: Advanced. The paper assumes familiarity with metric-learning losses (ArcFace, Circle loss), knowledge distillation, test-time augmentation, and classical retrieval reranking algorithms (diffusion, k-reciprocal reranking).
• Scope: A description of the 3rd place entry to the ICCV LargeFineFoodAI Retrieval Competition on Kaggle, covering training, ensembling, and a reranking scheme that combines diffusion and k-reciprocal reranking.

What This Paper Is About

The paper describes a system that, given a query food photo, finds the most visually similar images in a large gallery. The task is hard because the food categories used for testing are completely different from those used for training, and many images are difficult even for humans to tell apart. The goal is to maximize retrieval accuracy, measured as mAP@100, on 10,000 query images against 209,562 gallery images.

Key Contributions

1. A training recipe combining ArcFace and Circle loss as a weighted sum, where the Circle loss weight is set to 1 divided by the batch size, letting ArcFace dominate early training and Circle loss contribute later.
2. A four-model pipeline using ResNeXt152, ResNeSt269, and ResNeSt200 backbones, all distilled with a temperature-3 KD loss combined 1:1 with cross entropy, followed by test-time augmentation and feature-level ensemble.
3. A new reranking method that runs diffusion and k-reciprocal reranking independently and fuses them into a single score, S_final = S - λ × D, where S is the diffusion similarity matrix, D is the k-reciprocal distance matrix, and λ = 1.
4. A reported 3rd place finish with 0.81219 mAP@100 on the public leaderboard and 0.81191 on the private leaderboard.

Main Findings

• Final scores: 0.81219 mAP@100 on the public leaderboard and 0.81191 on the private leaderboard.
• Dataset scale: Training used the LargeFineFoodAI dataset of 317,277 images across 1000 food classes; testing used 500 additional food classes with 10,000 query images and 209,562 gallery images, with no class overlap between train and test.
• TTA gains: Test-time augmentation with five crops, resize, horizontal flip, and rotation produced roughly 2 to 4 percent gains.
• Ensemble gains: Adding features from the selected models and applying L2 normalization gave roughly 1 to 2 percent improvement.
• Reranking gains on a single model: Diffusion gave about 8 percent improvement and k-reciprocal reranking about 6 percent, measured on a single model.
• Combined reranking: In earlier 3-model ensemble experiments, the combined diffusion-plus-k-reciprocal method improved about 0.33 percent over the best single retrieval algorithm.
• Epoch sensitivity: For ResNeSt269, retrieval score peaked at epoch 15 (0.6473) and declined afterward even as recognition accuracy kept rising: epoch 10 gave 0.845 accuracy / loss 1.9585 / retrieval 0.59204; epoch 20 gave 0.8944 / 1.4986 / 0.64581; epoch 24 gave 0.8992 / 1.362 / 0.64034; epoch 30 gave 0.9031 / 1.1578 / 0.62799; epoch 40 gave 0.9061 / 0.7997 / 0.59956. Epoch 15 gave 0.8831 / 1.7001 / 0.6473.
• Backbone detail: Training used eight NVIDIA Tesla-V100 GPUs, 512x512 inputs, and 416x416 for ResNeSt269 because the authors found smaller image size better for retrieval; features were taken after the final GAP layer, normalized, and were 1x2048 per image.
• Negative results on other post-processing: PCA reduced feature dimensions and improved a single model by about 0.4 percent, but combining it with whitening hurt performance. DBA improved performance somewhat, but adding AQE caused degradation. The paper states these methods generally did not work together with diffusion, which requires feature dimensions to be irrelevant.

Methodology in Plain English

The team treats each food photo as a vector, or feature, produced by a convolutional network, and retrieves gallery images whose vectors are closest to a query's vector.

Training: three backbone architectures — ResNeXt152, ResNeSt269, and ResNeSt200 — were trained with standard image augmentations (random scale and center crop, horizontal flip with probability 0.5, 15-degree rotation, color jitter) using SGD with momentum 0.9 and cosine annealing. The loss combined ArcFace (margin 0.2, scale 32) with Circle loss (m 0.25, delta 32), weighted as δ0 = 1 and δ1 = 1/batch size. The authors note the abstract refers to four basic models, while the backbone list contains three architectures; the later ensemble section also names three models used for further ensembling.

Distillation: teacher models were distilled into the trainees with a knowledge-distillation loss on soft labels and cross entropy on hard labels at equal 1:1 weight and temperature 3.

Inference: each test image was passed through four test-time augmentations, and features from models scoring above 0.66 under nearest-neighbor search after TTA were added together and L2-normalized.

Reranking: two classical methods were run separately on the 2048-dimensional ensembled features. Diffusion produced a 10000 x 209562 similarity matrix, using a subgraph scale of 10000 and 70 nearest neighbors per node. K-reciprocal reranking produced a same-shaped distance matrix, using k1 = 260 reciprocal nearest neighbors, a local query expansion parameter k2 = 30, and a weighting coefficient of 0.3 between Euclidean and Jaccard distance. The two outputs were fused by subtracting λ times the distance matrix from the similarity matrix with λ = 1, and the top 100 results per query were returned by sorting this fused score.

Why This Matters

The paper documents a competition-winning-style pipeline for fine-grained retrieval where training and test categories do not overlap, which is a realistic and difficult setting for deployment. It shows that metric-learning loss combinations, distillation, TTA, ensembling, and classical reranking stack into measurable gains, and it adds a concrete fusion recipe for two well-known reranking methods.

Real-world applications:

• Visual food search and menu or calorie estimation apps that let users photograph a dish and find matching items.
• E-commerce and grocery catalog search, where a shopper's photo must retrieve visually similar products.
• Inventory and quality control in food production, matching photographs of products against a reference gallery.
• General fine-grained retrieval systems in other domains (fashion, plant or animal species, retail) that face the same unseen-class transfer problem.

Industry relevance: the pipeline is built from off-the-shelf components (three common backbones, standard losses, classical reranking) trained on eight NVIDIA Tesla-V100 GPUs, which makes it reproducible for teams with comparable compute. The reported negative results for PCA, whitening, DBA, and AQE are also practically useful, since they indicate which popular post-processing tricks failed in this setting.

Future Directions

• Test the combined diffusion-plus-k-reciprocal reranking directly on the final ensembled features; the paper states this was not done due to submission limits.
• Tune the fusion hyperparameter λ in S_final = S - λ × D, which was fixed at 1 for simplicity and never finetuned because of submission limits.
• Apply the 416x416 input size that helped retrieval to more than one model; the authors changed only ResNeSt269 because of time and compute constraints.
• Investigate why retrieval score peaks early in training (epoch 15 for ResNeSt269) while recognition accuracy continues to improve, and use that finding to select checkpoints more systematically.
• Revisit post-processing methods such as PCA, whitening, DBA, and AQE, which the paper reports as limited or harmful, particularly in combination with diffusion.

Target Audience

Researchers and engineers working on image retrieval, metric learning, or fine-grained visual recognition, especially those building competition or production systems with large galleries and unseen test categories. The paper is most useful to readers who already understand embedding-based retrieval and want a practical account of how training losses, distillation, TTA, ensembling, and reranking interact, including which techniques did not work.

Authors’ abstract

This paper introduces the 3rd place solution to the ICCV LargeFineFoodAI Retrieval Competition on Kaggle. Four basic models are independently trained with the weighted sum of ArcFace and Circle loss, then TTA and Ensemble are successively applied to improve feature representation ability. In addition, a new reranking method for retrieval is proposed based on diffusion and k-reciprocal reranking. Finally, our method scored 0.81219 and 0.81191 mAP@100 on the public and private leaderboard, respectively.