YOLO-IOD: Towards Real Time Incremental Object Detection

AI summary

Overview

Research area: Computer vision — incremental (continual) object detection, specifically adapting real-time YOLO detectors to learn new object classes sequentially.

Technical level: Intermediate. Readers need basic familiarity with object detection, knowledge distillation, and continual learning, but the paper's framing around three named "knowledge conflicts" is approachable.

Scope: The paper proposes a full incremental detection framework (YOLO-IOD) built on YOLO-World, three modules to counter forgetting, and a new benchmark (LoCo COCO) that removes inter-stage data leakage.

What This Paper Is About

Incremental object detection (IOD) asks a model to learn new object categories over successive training stages without forgetting the ones it already knows. Nearly all existing IOD methods are built on slower detectors like Faster R-CNN or DETR, and when those techniques are transferred to fast YOLO-style detectors, accuracy collapses. This paper diagnoses why that happens and builds a YOLO-native incremental detector that keeps real-time inference speed while adding new classes with minimal forgetting.

Key Contributions

1. Diagnosis of three knowledge conflicts specific to YOLO-based incremental detectors: foreground-background confusion (unlabeled objects from past or future tasks treated as background, worsened by Mosaic/MixUp augmentation), parameter interference (tasks compete for overlapping weight subsets), and misaligned knowledge distillation (teacher and student no longer share the same class distribution).

2. YOLO-IOD, a real-time incremental detection framework built on pretrained YOLO-World with stage-wise parameter-efficient fine-tuning, comprising three modules: Conflict-Aware Pseudo-Label Refinement (CPR), Importance-based Kernel Selection (IKS), and Cross-Stage Asymmetric Knowledge Distillation (CAKD).

3. CAKD's dual-teacher design, in which the target student detector's neck features are routed through the detection heads of both the previous-stage teacher and the current-stage teacher, letting the model distill from old and new categories asymmetrically rather than distilling only the non-conflicting old outputs.

4. LoCo COCO, a benchmark that partitions categories using co-occurrence statistics and assigns each image to exactly one stage, eliminating the inter-stage image reuse (averaging 1.84 stages per image in the prior 20+20 setup) that inflated the apparent success of pseudo-label-based IOD methods.

Main Findings

• Consistent state-of-the-art AP across single-step settings. On COCO 40+40, YOLO-IOD reaches 53.0 AP versus 51.5 for the previous best (RGR, a generative-replay method), and cuts the relative gap to joint training from 5.5% to 2.7%. On 70+10 it reaches 52.4 AP with a 3.9% relative gap — and does so without any replay buffer.

• Advantage grows in long multi-step sequences. On the 10-10 setting with eight incremental phases, YOLO-IOD ends with an 8.8% relative gap, compared with 20.3% for RGR and 48.1% for CL-DETR. Gains over RGR across multi-step protocols range from 3.3 to 6.3 AP.

• LoCo COCO reveals benchmark inflation. Every compared method loses between 0.6% and 2.0% AP when inter-stage image overlap is removed, confirming that prior benchmarks leaked training images across stages. YOLO-IOD still outperforms the previous best (GCD) by 7.5, 5.9, and 8.5 AP in the 40+40, 70+10, and 40+20 settings.

• All three modules contribute, and they compound. Ablation on 70-10: baseline pseudo-labeling 48.4 AP, +CPR 50.3, +IKS 51.5, all three 52.4. CAKD alone lifts the baseline to 50.8 AP, showing distillation is the single strongest component.

• Dual-teacher asymmetry matters. Using only the current-stage teacher helps early (plasticity), using only the old teacher helps later (stability), and the full dual-teacher configuration wins throughout.

• Kernel selection ratio has a sweet spot. Freezing too much (5%) prevents adaptation; updating too much (20%) causes forgetting. 12% for incremental stages and 20% for the base stage is optimal.

Methodology in Plain English

The authors start from an open-vocabulary detector, YOLO-World, which can already recognize arbitrary categories described by text. Each incremental stage then fine-tunes only a small, carefully chosen fraction of the network.

To handle unlabeled objects that belong to categories the model does not yet know or has already learned, the framework does two things. First, it generates pseudo-labels for those unlabeled objects and uses their confidence scores as soft targets rather than hard thresholds, adding an entropy term that keeps low-confidence predictions uncertain instead of forcing them to a class. Second, it runs the open-vocabulary detector with a broad vocabulary of 500 common categories plus 50 abstract super-categories, clusters the resulting text embeddings with frequency-weighted K-means, and folds stray detections into "unknown" super-categories so they are not mistaken for background.

To avoid wrecking previously learned weights, the authors estimate how important each convolution kernel is using Fisher Information, subtract the accumulated historical importance from the current-task importance, and fine-tune only the top-ranked kernels while freezing the rest.

For distillation, the student's intermediate features are pushed through both the old teacher's and the new teacher's detection heads. A focal weight derived from the teacher's maximum logit suppresses background regions, and the losses compare region-level feature embeddings (L2) and predicted boxes (IoU) against both teachers. This sidesteps the standard trick of distilling only old-task outputs that do not overlap with new labels, which the authors argue discards most of the useful signal for dense YOLO predictions.

The LoCo COCO benchmark is constructed by building a category co-occurrence matrix from COCO, clustering it as an undirected weighted graph, and assigning frequently co-occurring categories to the same stage. Any remaining image that spans multiple stages is randomly assigned to a single stage, so no image is ever seen twice.

Why This Matters

Real-time detectors are what actually ship in products, yet the incremental learning literature has largely ignored them because their dense prediction heads and heavy augmentation pipelines break standard IOD techniques. This paper shows the gap is fixable with targeted design rather than architectural compromise, and it demonstrates that a chunk of the reported progress in IOD has been flattered by benchmarks that quietly reuse images across stages.

Real-world applications:

• Autonomous driving and robotics — vehicles and robots encounter new object types (road debris, novel vehicle classes, delivery robots) continuously and cannot be retrained from scratch on the road.
• Video surveillance and security — camera deployments must add new threat categories (a new weapon type, a new uniform) without losing detection of everything previously configured.
• Industrial visual inspection — production lines add new defect types over time; downtime for full retraining is expensive.
• Mobile and embedded vision — YOLO-class models run on constrained hardware, so preserving real-time latency while incrementally updating is a hard deployment requirement, not a nice-to-have.

Industry relevance: Any team deploying detection models that must evolve with changing data distributions — rather than being frozen at release — benefits. The parameter-efficient kernel selection also reduces storage and compute per update, which matters when updates happen on-device or in fleet-scale pipelines.

Future Directions

• Extending the conflict analysis beyond YOLO to other single-stage dense detectors (e.g., FCOS-style, anchor-free heads) to test whether the three named conflicts generalize or are specific to YOLO's architecture and augmentation recipe.
• Closing the remaining gap to joint training. Even the best results leave a 2.7–8.8% relative gap, and the largest gaps appear in the longest sequences, suggesting long-horizon drift is not fully solved.
• Combining YOLO-IOD with replay or generative rehearsal. The paper deliberately avoids replay, but hybrid approaches might recover additional accuracy where the RelGap remains highest (the eight-phase 10-10 setting).
• Refining LoCo COCO or building successors. The benchmark's random assignment of multi-stage images and its single hard co-occurrence partition are pragmatic choices; alternative partition strategies and domain-specific co-occurrence statistics could produce even more realistic evaluation protocols.

Target Audience

Graduate students and researchers working on continual learning, incremental detection, or open-vocabulary detection; computer vision engineers who need to deploy detectors that adapt to new classes without full retraining; and benchmark designers interested in evaluation protocols that expose, rather than hide, data leakage in incremental settings. Readers with a background in object detection will get the most from the module design details, while those focused on evaluation methodology will find the LoCo COCO analysis independently useful.

Authors’ abstract

Current methods for incremental object detection (IOD) primarily rely on Faster R-CNN or DETR series detectors; however, these approaches do not accommodate the real-time YOLO detection frameworks. In this paper, we first identify three primary types of knowledge conflicts that contribute to catastrophic forgetting in YOLO-based incremental detectors: foreground-background confusion, parameter interference, and misaligned knowledge distillation. Subsequently, we introduce YOLO-IOD, a real-time Incremental Object Detection (IOD) framework that is constructed upon the pretrained YOLO-World model, facilitating incremental learning via a stage-wise parameter-efficient fine-tuning process. Specifically, YOLO-IOD encompasses three principal components: 1) Conflict-Aware Pseudo-Label Refinement (CPR), which mitigates the foreground-background confusion by leveraging the confidence levels of pseudo labels and identifying potential objects relevant to future tasks. 2) Importancebased Kernel Selection (IKS), which identifies and updates the pivotal convolution kernels pertinent to the current task during the current learning stage. 3) Cross-Stage Asymmetric Knowledge Distillation (CAKD), which addresses the misaligned knowledge distillation conflict by transmitting the features of the student target detector through the detection heads of both the previous and current teacher detectors, thereby facilitating asymmetric distillation between existing and newly introduced categories. We further introduce LoCo COCO, a more realistic benchmark that eliminates data leakage across stages. Experiments on both conventional and LoCo COCO benchmarks show that YOLO-IOD achieves superior performance with minimal forgetting.