Enhancing Object Detection with Privileged Information: A Model-Agnostic Teacher-Student Approach

AI summary

Overview

Research area: Computer vision — object detection, specifically the Learning Using Privileged Information (LUPI) paradigm applied through teacher–student knowledge distillation.

Technical level: Intermediate. Readers should have some familiarity with object detection architectures (one-stage vs. two-stage, backbones, FPN) and with knowledge distillation concepts.

Scope: A model-agnostic teacher–student methodology for injecting training-only privileged information (bounding box masks, saliency maps, depth cues) into five existing object detectors, evaluated on UAV litter-detection datasets and Pascal VOC 2012.

What This Paper Is About

Object detectors are typically trained on images with bounding box and class labels, but those annotations contain far more structured information than models currently exploit. This paper asks whether extra, fine-grained signals that exist only at training time — such as bounding box masks or depth maps — can be used to train a better detector that still runs on plain RGB images at inference. The authors formalise this as a LUPI problem and solve it with a teacher network that sees the privileged input guiding a student network that does not.

Key Contributions

1. A general, model-agnostic methodology for injecting privileged information into any deep learning-based object detector via a teacher–student architecture, not restricted by architectural choices.
2. Broad empirical validation across five open-source, state-of-the-art pretrained detectors (Faster R-CNN, SSD, RetinaNet, SSDLite, FCOS), multiple UAV-based litter detection datasets, and Pascal VOC 2012.
3. A systematic comparison of privileged information forms — saliency, depth, a fusion of these, and bounding box masks — within the teacher–student framework, with the bounding box mask achieving the highest improvement.
4. Extension of the authors' earlier work through analysis of performance across object scales, standard COCO metrics, ablation over the loss-balancing parameter, inference time and model size, plus Grad-CAM interpretability analysis.

Main Findings

• Teacher models benefit strongly from privileged input: with privileged channels, teacher strict mAP and mAR values approach 1 across within-dataset experiments, demonstrating high reliability. On SODA at 1-metre, RetinaNet reached mAP 0.94, mAP@50 0.98, and mAP@75 0.96; Faster R-CNN reached mAP 0.96, mAP@50 0.99, mAP@75 0.98.
• Architecture ranking among teachers: Faster R-CNN achieved the highest average mAP, followed closely by RetinaNet and FCOS, while SSD and SSDLite exhibited comparatively lower performance. In the SODA at 1-metre setting, SSDLite's mAP was 0.61 and its mAP@Medium was 0.00.
• Cross-dataset and cross-domain performance drops but remains robust: on BDW, teacher mAP ranged from 0.55 (SSD) to 0.23 (SSDLite); on UAVVaste, from 0.44 (Faster R-CNN) to 0.01 (SSDLite); on Pascal VOC 2012, from 0.80 (FCOS) to 0.42 (SSD).
• Students consistently beat their baselines: LUPI-trained students showed consistent gains in strict mAP and F1 score, with the largest boosts in within-dataset UAV litter detection and smaller yet meaningful gains in cross-dataset evaluations.
• Architecture-dependent gains: Faster R-CNN, FCOS and RetinaNet benefited most from teacher guidance in UAV-based scenarios, whereas SSD and SSDLite exhibited clearer improvements on Pascal VOC.
• Object scale matters: improvements were most marked for medium and large objects, while small objects showed limited improvement.
• Bounding box masks are the best privileged signal: among saliency, depth, fusion and bounding box mask representations tested on RetinaNet with the SODA 1-metre dataset, the bounding box mask yielded the highest improvement. Each mask is a black background with bounding boxes filled using grayscale values proportional to class labels, drawn in descending size order to minimise occlusion.
• Intermediate teacher weighting is optimal: in ablations over α ∈ {0, 0.25, 0.5, 0.75, 1} (where α = 0 is the baseline and α = 1 is full teacher supervision), α = 0.25 and 0.5 generally gave the best performance, α = 0.75 occasionally performed well, and α = 1 tended to underperform.
• One exception to student superiority: for Faster R-CNN on Pascal VOC, only the baseline outperformed its student counterpart, attributed to the teacher's additional region proposals introducing ambiguity in supervision, though the difference remains marginal.
• No inference penalty: baseline and student models were nearly identical in size, number of parameters, GFLOPS and FPS. For example, RetinaNet was 124.22 MB, 32.56 M parameters, 265.10 GFLOPS (39.74 FPS baseline vs 38.00 FPS student), and SSDLite was 9.42 MB, 2.47 M parameters, 0.95 GFLOPS (36.74 vs 36.75 FPS).
• Training cost roughly doubles: because both a teacher and a student must be trained, the training phase is effectively doubled, as shown in the Pascal VOC 2012 training-time comparison.
• Sharper attention: Grad-CAM visualisations on the final backbone layer show LUPI-trained students concentrating on litter objects with higher-confidence detections and fewer misclassifications, whereas baseline attention is more diffuse across the background.

Methodology in Plain English

The setup uses two networks. The teacher is the same detector as the baseline but with its input layer extended from three to four channels so it can also read a privileged signal (for example, a mask of the ground-truth boxes). The added weights are initialised with Kaiming Normal initialisation while the rest keep their pre-trained COCO weights. The student is architecturally identical to the baseline and sees only RGB images.

During training, the student is asked to do two things at once: predict detections correctly from the labels, and mimic the teacher's internal feature representation at the final backbone layer. The combined loss is (1 − α)·(detection loss) + α·(cosine distance between teacher and student features), so α controls how much the student leans on the teacher. For SSD and SSDLite the matching happens at the final convolutional layer before the auxiliary heads; for Faster R-CNN, FCOS and RetinaNet it is the last convolutional layer before the FPN.

All models were implemented using torchvision architectures, trained for 100 epochs with the Adam optimizer at a fixed learning rate of 1×10⁻³, with early stopping and checkpointing based on validation loss. Images (including privileged channels) were normalised with min-max scaling, resized to 800×800 pixels, standardised per channel to zero mean and unit variance, and non-maximum suppression with an IoU threshold of 0.5 was applied to final predictions.

Evaluation used standard detection metrics (mAP, precision, recall, F1, mAR) plus COCO-style metrics for object scales. Within-dataset experiments used SODA subsets under three scenarios: binary litter detection at 1-metre altitude without tiling, binary detection across multiple altitudes with 3×3 tiling, and multi-label detection across altitudes with 3×3 tiling. Cross-dataset experiments tested SODA-trained models on BDW (models trained at 1-metre, tested without retraining) and UAVVaste (models trained on 3×3 tiled SODA images). Pascal VOC 2012 provided a broader 20-category test. The full pipeline is publicly available on GitHub.

Why This Matters

Impact on research: LUPI in computer vision has been relatively underexplored, particularly for object detection, where the authors' earlier work is described as one of the first contributions. This paper shows that privileged information can be added without architectural change, without extra inference cost, and without requiring larger annotated datasets — reframing detector improvement as a learning-process problem rather than an architecture problem.

Real-world applications:

• UAV-based environmental monitoring and litter detection, the paper's primary domain, where small object sizes, complex backgrounds and high scene variability make detection hard.
• Traffic analysis systems requiring fast, reliable detection.
• Surveillance systems where both speed and precision are critical.
• Lightweight deployment on resource-constrained platforms such as edge or embedded devices, enabled by compact models such as SSDLite (9.42 MB, 2.47 M parameters).

Industry relevance: The method is model-agnostic and drops into existing pre-trained detectors, so organisations can improve accuracy on an existing deployment without retraining new architectures, hiring for more annotation, or accepting slower inference. The only added cost is a longer training phase, which is typically paid once while inference is run far more frequently.

Future Directions

• Integrating the framework with newer detection architectures such as YOLOv12 and RF-DETR.
• Exploring richer and more diverse privileged information, including semantic maps and attention-based cues.
• Adapting the framework to related tasks such as object segmentation.
• Addressing the identified limitations of mask-based privileged information: overlapping objects of the same category, occlusion from larger bounding boxes, and limited colour differentiation within mask representations.

Target Audience

Researchers and graduate students working on object detection, knowledge distillation, or learning using privileged information; computer-vision engineers looking for a training-time-only technique that improves accuracy without changing inference cost; and practitioners in UAV monitoring, environmental monitoring, traffic analysis or surveillance who deploy lightweight detectors on resource-constrained hardware.

Authors’ abstract

This paper investigates the integration of the Learning Using Privileged Information (LUPI) paradigm in object detection to exploit fine-grained, descriptive information available during training but not at inference. We introduce a general, model-agnostic methodology for injecting privileged information-such as bounding box masks, saliency maps, and depth cues-into deep learning-based object detectors through a teacher-student architecture. Experiments are conducted across five state-of-the-art object detection models and multiple public benchmarks, including UAV-based litter detection datasets and Pascal VOC 2012, to assess the impact on accuracy, generalization, and computational efficiency. Our results demonstrate that LUPI-trained students consistently outperform their baseline counterparts, achieving significant boosts in detection accuracy with no increase in inference complexity or model size. Performance improvements are especially marked for medium and large objects, while ablation studies reveal that intermediate weighting of teacher guidance optimally balances learning from privileged and standard inputs. The findings affirm that the LUPI framework provides an effective and practical strategy for advancing object detection systems in both resource-constrained and real-world settings.