3rd Place Solution to Large-scale Fine-grained Food Recognition

AI summary

Overview

• Research area: Computer vision, specifically large-scale fine-grained image classification applied to food recognition, presented as a Kaggle competition solution.
• Technical level: Advanced. The paper assumes familiarity with metric-learning losses (ArcFace, Circle loss), knowledge distillation, test-time augmentation, and model ensembling.
• Scope: A competition report describing how a team combined two training configurations, two loss functions, eight convolutional backbones, distillation, test-time augmentation, and weighted ensembling to place 3rd in the LargeFineFoodAI-ICCV Workshop-Recognition challenge.

What This Paper Is About

The authors tackle the problem of telling apart many visually similar food categories — a fine-grained recognition task where differences between classes can be subtle. They describe their entry to the LargeFineFoodAI-ICCV Workshop-Recognition challenge, which uses Food1K (1000 food categories, 500,000 images) drawn from a larger 1500-category dataset. The goal was to maximize classification accuracy on a held-out test set and, in practice, to finish as high as possible on the competition leaderboard; their final pipeline placed 3rd.

Key Contributions

1. A combined metric-learning loss. The authors report that adding Circle loss to ArcFace loss, with a properly tuned weight, accelerates convergence and improves accuracy. The final loss is L = γ0·La + γ1·Lc, with γ0 = 1 and γ1 = 1/β where β is the training batch size. ArcFace is used with a margin of 0.2 and a scale of 32; Circle loss is used with m of 0.25 and gamma of 32.
2. A two-configuration training recipe. Two configurations (config-a: 416 input, cutmix; config-b: 512 input, random scale and center crop, horizontal flip at 0.5 probability, 15-degree rotate, color jitter) are applied across six backbone families to produce eight trained models: regnetx, rx152, effnetv2, effnet5, rs200-a, rs269-a, rs200-b, and rs269-b.
3. Distillation applied selectively. Two backbones (rx152 and rs200-b) are distilled from their trained versions using a KDloss on soft labels (the teacher's 1000-dimension output embeddings) with temperature 3 and equal 1:1 weighting with cross entropy. The paper also documents that BAKE self-distillation was tried and hurt performance.
4. An ensemble pipeline that produced the final score. Test-time augmentation with four augmentations per image plus a weighted sum of pre-softmax feature embeddings across the eight models yields 0.92093 on the public leaderboard and 0.92019 on the private leaderboard, securing 3rd place.

Main Findings

• Dataset and split sizes: Food1K contains 1000 food categories and 500,000 images, split into 317277 training, 53462 validation, and 158790 test images. The full source dataset is described as having 1500 food categories and about 80,0000 images (as printed in the paper).
• Loss functions, single-model public scores (ResNeSt269): Cross Entropy at 416 gives 0.90195; Arcface loss at 416 with cutmix gives 0.90510; Arcface loss at 512 with cutmix gives 0.90801; Arcface + Circle at 512 with cutmix gives 0.90661; Arcface + Circle at 512 without cutmix gives 0.90892, the best of these configurations.
• Backbone comparison (single model, public / with TTA): CvT 0.83787, MLP-Mixer 0.85978, RegNetX 0.87858 / 0.88742, RegNetY 0.87879, EfficientNet-B5 0.89184 / 0.89430, ResNeXt152 0.88880 / 0.89636, ResNeSt200-a 0.89610, ResNeSt200-b 0.90708 / 0.90809, ResNeSt269-a 0.90510 / 0.90833, ResNeSt269-b 0.90892 / 0.91039.
• Transformers and Mixers underperformed. ViT and CvT were tried but judged unsuitable because transformer-based models rely on pretraining on a larger dataset to improve performance in this setting; MLP-Mixer also underperformed.
• Larger finetuning image sizes hurt. Inspired by prior work, the authors finetuned on a bigger image size both for the whole model and for models with weights before the last batch normalization layer frozen; both variants degraded performance.
• BAKE self-distillation reduced accuracy. On rs269-a, BAKE lowered the score to 0.90452 from 0.90510.
• Traditional distillation helped. KDloss combined equally with cross entropy improved the two distilled backbones, rx152 and rs200-b.
• Progressive gains from post-processing (public / private): single model (ResNeSt269-b) 0.90892 / 0.90760; plus test-time augmentation 0.91039 / 0.90962; plus ensemble 0.92093 / 0.92019.
• Ensemble method comparison (RegNetY and ResNeSt269, public / private): Vote 0.91040 / 0.90960, Stacking 0.91272 / 0.91209, Logits sum 0.91339 / 0.91233 — logits sum performed best.
• Ensemble weighting scheme: Weights of 1.0 were assigned to the first five ensemble models because each scored below 0.90 on the test set; weights of 1.5 were assigned to the remaining three models, which scored nearly 0.91.
• Same-architecture diversity helped. The authors report that adding a model with the same structure sourced from a different code base improved ensemble performance.
• An inconsistency in the write-up: one sentence states config-a uses Arcface loss and config-b uses the combined loss, while a later sentence states the combined loss was used in config-a and Arcface loss in config-b. The paper does not reconcile these two statements.

Methodology in Plain English

The team started from the same dataset and the same basic approach: train convolutional neural networks to map a food photo to one of 1000 categories. Rather than searching for a single best model, they deliberately trained several models under two different settings so that later averaging would help.

The two settings differ mainly in input resolution and augmentation. One setting ("config-a") trains on 416-pixel images and adds cutmix, a technique that pastes patches of one image onto another. The other ("config-b") trains on 512-pixel images and uses random scale and center cropping, horizontal flips half the time, 15-degree rotations, and color jitter, but no cutmix. They tuned these settings on ResNeSt269 first and transferred them to other backbones. At test time they used larger images than in training (512 for config-a, 690 for config-b), reported to work better.

For the classification objective, they compared cross entropy against ArcFace and Circle loss. ArcFace, a loss that pushes classes apart by a margin, performed best. Their key trick was to add Circle loss on top, weighted by the inverse of the batch size, so that ArcFace dominates early in training and Circle loss contributes later. They trained eight models covering RegNetX, ResNeXt152, EfficientNet-B5, EfficientNetV2, and ResNeSt200 and ResNeSt269 under both configurations, using SGD with momentum 0.9 and cosine annealing.

Two of the eight models were then refined through knowledge distillation: a trained "teacher" model supplies soft labels (its 1000-dimension outputs), and the student is trained on a blend of those soft labels and the true hard labels. For the final prediction, each test image is passed through four test-time augmentations — five crops, resize, horizontal flip, and rotate — and the resulting feature vectors from all eight models are combined as a weighted sum before the final class decision.

Why This Matters

Research impact. The paper is a concrete, reproducible data point on loss-function design for fine-grained recognition: it shows that a weighted combination of ArcFace and Circle loss can outperform either alone in a competition setting, and that the best augmentation choice can change once the loss changes (removing cutmix improved the combined-loss model). It also documents negative results — transformer backbones without large-scale pretraining, MLP-Mixer, BAKE self-distillation, and large-image finetuning all failed here — which is useful information that often goes unpublished.

Real-world applications.

• Dietary tracking and calorie estimation: automatically identifying a dish from a phone photo is the first step in logging what someone ate.
• Nutrition research and public health monitoring: large-scale food-image datasets can support automated coding of eating habits across populations.
• Restaurant and food-delivery platforms: tagging, search, and cataloging of menu items at fine-grained category levels.
• Canteen or automated checkout systems: recognizing individual dishes from a tray image to speed up payment.

Industry relevance. The work is a collaboration between East China Normal University, Shanghai Jiao Tong University, and OPPO Research Institute, and the task is a phone-camera-friendly one. The competition setting (Kaggle, ICCV workshop) reflects how industry teams benchmark practical image-recognition pipelines. The paper does not report deployment on any mobile device, nor does it report inference latency, model size, or on-device performance.

Future Directions

• Resolve the loss/configuration contradiction in the paper so that the reported config-a and config-b results can be reproduced unambiguously.
• Extend to the full dataset. Only 1000 of the 1500 food categories were used for the Food1K task; the remaining categories are an obvious next test of whether the recipe scales.
• Revisit transformer backbones with large-scale pretraining, since the authors attribute the poor ViT and CvT results to the lack of such pretraining rather than to the architecture itself.
• Test the recipe on other fine-grained domains, such as animal, plant, or product recognition, where the same many-similar-class structure applies but the ArcFace margin of 0.2 and Circle loss settings of m = 0.25 and gamma = 32 may need retuning.
• Investigate why same-architecture models from different code bases improve ensembles, since the authors report the phenomenon but do not explain it.

Target Audience

• Competition participants in image classification, especially those working on fine-grained or long-tailed recognition.
• Practitioners building production food-recognition or dietary-assessment systems, who can reuse the loss combination and ensembling strategy.
• Researchers in metric learning who want a documented case study of combining ArcFace and Circle loss.
• Engineers interested in the practical value of negative results: which backbones, distillations, and finetuning strategies did not pay off under competition constraints.

Authors’ abstract

Food analysis is becoming a hot topic in health area, in which fine-grained food recognition task plays an important role. In this paper, we describe the details of our solution to the LargeFineFoodAI-ICCV Workshop-Recognition challenge held on Kaggle. We find a proper combination of Arcface loss[1] and Circle loss[9] can bring improvement to the performance. With Arcface and the combined loss, model was trained with carefully tuned configurations and ensembled to get the final results. Our solution won the 3rd place in the competition.