Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design

AI summary

Overview

• Research area: Real-time object detection, specifically Detection Transformers (DETR) that must run at low latency with high accuracy, plus efficient attention mechanisms (Neighborhood Attention) and backbone architecture design.
• Technical level: Intermediate. A reader needs familiarity with object detection metrics (COCO mAP), CNN backbones, and transformer attention, but the paper's central argument — that heavy backbone pre-training is compensating for weak architecture — is stated in plain terms.
• Scope: The paper proposes Le-DETR (Low-cost and Efficient DEtection TRansformer), a real-time DETR family built on a new EfficientNAT backbone and a Neighborhood-Attention encoder, claiming state-of-the-art real-time accuracy while using only ImageNet-1K and COCO2017 and saving roughly 80% of the pre-training images used by prior real-time DETR models.
• Paper metadata: arXiv:2602.21010v1 [cs.CV], published 24 Feb 2026, licensed CC BY 4.0. Authors are Jiannan Huang, Aditya Kane, Fengzhe Zhou, Yunchao Wei, and Humphrey Shi; affiliations are SHI Labs @ Georgia Tech and Beijing Jiaotong University. Code and weights are stated to be released later ("will be open-sourced").

What This Paper Is About

Real-time DETR detectors reach strong accuracy, but they are hard to reproduce from scratch because their backbones depend on expensive pre-training: roughly four million extra filtered unlabeled images plus long knowledge-distillation schedules. The authors ask whether that pre-training burden is truly necessary, or whether it has been masking sub-optimal architecture choices. Their goal is a real-time detector that reaches state-of-the-art accuracy using only the standard ImageNet-1K pre-training set (about one million images) plus COCO2017.

Key Contributions

1. Diagnosis of a reproducibility bottleneck. The authors identify that real-time DETR models rely on heavy backbone pre-training (extra images plus knowledge distillation), which locks the community into specific costly pipelines and discourages new backbone research. They show that state-of-the-art results are reachable with ImageNet-1K pre-training alone, cutting roughly 80% of pre-training images.
2. A new backbone, EfficientNAT. Built from a detailed architecture study, EfficientNAT combines efficient convolution (depthwise separable convolutions, Fused Mobile Convolution, Mobile Convolution) in early stages with a Neighborhood Attention block plus an MBConv feed-forward network in the final stage.
3. A redesigned encoder using local attention, NAIFI. The Neighborhood Attention-based Improved Feature Inference module replaces the original AIFI self-attention design in the hybrid encoder, reported to improve both accuracy and latency (local attention lowers latency while raising mAP in the reported ablation).
4. An empirical backbone-scaling study. The paper maps three block-distribution patterns — balanced (P_A), late-stage heavy (P_B), and early-stage heavy (P_C) — and finds the best pattern differs by model scale (P_A for the L scale, P_C for the X scale).

Main Findings

• Reported accuracy and latency (abstract and Table 2): Le-DETR-M/L/X achieve 52.9/54.3/55.1 mAP on COCO Val2017 at 4.45/5.01/6.68 ms on an RTX 4090 (PyTorch profiler, FP16, 640x640 input).
• Internal inconsistency in the reported numbers: Section 4.2's prose states Le-DETR-M at 52.4 mAP, Le-DETR-L at 53.6 mAP, and Le-DETR-X at 54.7 mAP, which disagrees with the abstract, Table 2, and Table 4 (52.9/54.3/55.1). The Introduction also lists the M latency as 4.73 ms where the abstract and tables list 4.45 ms.
• Pre-training savings: The method uses about 1M pre-training images versus 5M for RT-DETRv2-L (which uses knowledge distillation and additional training images), described as saving 3M images, roughly 80%.
• Comparison with RT-DETRv2-L (Table 1): RT-DETRv2-L with KD and ATI: 5.46 ms, 53.4 AP. Without KD (& with variant): 5.46 ms, 51.6 AP (↓1.8). Without KD or variant: 4.91 ms, 51.6 AP. Le-DETR-L: 5.01 ms, 54.3 AP — a reported +0.9 mAP vs RT-DETRv2-L, widening to +2.7 mAP under identical pre-training settings.
• Comparison with YOLOv12: Le-DETR surpasses YOLOv12-L/X by +0.6 / -0.1 mAP at similar speed and with a reported +20% speedup on the X model. Against YOLO11, Le-DETR-M is reported 1.4 mAP better, Le-DETR-L 1.0 mAP better, and Le-DETR-X 0.5 mAP better while being 14% faster.
• Comparison with DEIM-D-FINE: Le-DETR-M is +0.2 mAP with slightly faster inference; Le-DETR achieves +0.4 mAP over DEIM-D-FINE-L with only 0.4 ms additional latency.
• Backbone ablation (Table 3, reported on the L-scale configuration): Full Le-DETR: 54.3 mAP / 5.01 ms. Replacing EfficientNAT with ResNet50_vd_ssld: 53.6 mAP / 5.80 ms. Replacing the backbone with EfficientViT-L1: 53.8 mAP / 6.60 ms. Replacing NAIFI with the original AIFI: 54.1 mAP / 5.18 ms.
• Backbone scaling study (Table 5): For the L scale, P_A-1 (1, 1, 4, 4 blocks) gives 81.994 Top-1 and 1798.75 throughput, and P_A-2 (1, 1, 6, 6) gives 82.135 Top-1 at 1607.68 throughput; P_A-1 was chosen because on COCO the two match in mAP while P_A-1 is faster (4.53 ms vs 4.96 ms). For the X scale, P_C-2 (2, 7, 15, 2) gives the best reported Top-1 of 82.902 at 1006.83 throughput.
• Decoder-layer ablation (Table 4, models trained with 6 decoder layers): Le-DETR-M at 4/5/6 inference layers: 52.5 / 52.9 / 52.9 mAP at 4.19 / 4.45 / 4.73 ms. Le-DETR-L: 53.7 / 54.2 / 54.3 mAP at 4.53 / 4.79 / 5.01 ms. Le-DETR-X: 54.6 / 54.9 / 55.1 mAP at 6.10 / 6.42 / 6.68 ms. The text describes the 5-layer choice for Le-DETR-M as a 0.26 ms speedup without harming performance.
• Training setup: Backbones are supervised-pre-trained on ImageNet-1K (300 epochs, batch size 128, base LR 1e-3, cosine schedule, weight decay 5e-2), then trained on COCO train2017 (80 epochs, total batch size 64, base LR 1.25e-4, backbone LR 5e-5, EMA decay 0.9999, AdamW). FDR and GO-LSD from D-FINE and the Matchability-Aware Loss from DEIM are used as training enhancements.
• Not reported: No absolute wall-clock training time for the full pipeline, no ONNX/TensorRT benchmark numbers (the authors note Neighborhood Attention is unsupported in those export paths), and no full COCO results for the efficient convolution-only ablation beyond the rows shown.

Methodology in Plain English

The authors start from a simple experiment on RT-DETRv2-L: strip away the extra pre-training images and knowledge distillation and watch accuracy fall from 53.4 to 51.6 AP. That tells them the backbone, not the detector head, is carrying the accuracy. So instead of buying accuracy with more pre-training data, they redesign the backbone.

They start from EfficientViT, which balances speed and accuracy on ImageNet-1K, and keep efficient convolutions (depthwise separable convolutions in the stem, Fused Mobile Convolution early, Mobile Convolution deeper) for the first three stages where local features dominate. In the fourth and final stage they insert Neighborhood Attention — attention restricted to a fixed local window around each token, which avoids the quadratic cost of full self-attention — paired with an MBConv feed-forward network. This block is the EfficientNATBlock, and the resulting backbone is EfficientNAT.

Because block counts control the speed/accuracy trade-off, they run a design study keeping the first two stages fixed and varying how blocks are distributed between stages three and four, testing balanced, late-heavy, and early-heavy layouts at the L and X scales on ImageNet classification (Top-1, Top-5, throughput) and then confirming the choice on COCO.

For the encoder they replace the AIFI module used in RT-DETR with NAIFI, a single-layer neighborhood-attention transformer with a small kernel size, and they use Flash Attention in the decoder. The overall detector keeps the standard backbone/encoder/decoder layout with DINO-style training heads on each decoder layer, so layers can be dropped at inference for speed.

Why This Matters

• Research impact: By showing that state-of-the-art real-time DETR accuracy does not require the undisclosed 4M-image pre-training pipeline, the paper removes a barrier the authors argue has confined the community to PResNet/PP-HGNet-style backbones. It also argues local attention is a viable, faster alternative to self-attention in encoders and backbones for this task.
• Reproducibility: The whole pipeline relies on ImageNet-1K and COCO2017 only, and the paper reports it can be trained on 12GB GPUs such as 8 x 2080Ti with the stated batch setup.
• Real-world applications (plausible implications, not enumerated in the paper):
  • Industrial and manufacturing inspection, where latency and accuracy both matter on constrained hardware.
  • Autonomous vehicles and robotics perception, where NMS-free end-to-end detection simplifies the inference pipeline.
  • Video surveillance and traffic monitoring, where many frames must be processed per second.
  • Edge and embedded deployments such as drones or retail analytics, where pre-training budget and inference latency limit what can be deployed.
• Industry relevance: Lower pre-training cost translates directly into lower compute spend and faster iteration cycles for teams that cannot afford multi-million-image distillation pipelines.

Future Directions

• Removing pre-training entirely: The authors state they expect new transformer-based models requiring no pre-training at all, since YOLO-series models need only COCO train2017 images; they note Le-DETR converges in fewer epochs than YOLO models so overall overhead is similar.
• Export and deployment support: Neighborhood Attention is not supported in ONNX and TensorRT exports, which the authors flag as a limitation for practical use in some cases, and they look forward to progress there.
• Extending the backbone design study: The reported scaling rule (P_A for L, P_C for X) is empirical and scale-specific; a principled rule for choosing block distributions across scales remains open.
• Resolving the reported number discrepancies: The mismatch between abstract/table values (52.9/54.3/55.1 mAP) and Section 4.2's prose (52.4/53.6/54.7 mAP), and between 4.45 ms and 4.73 ms for Le-DETR-M, would need clarification, and the promised code and weights would allow independent verification.

Target Audience

Researchers and engineers working on real-time object detection, end-to-end DETR training pipelines, or efficient transformer architectures will get the most from this paper. It is also relevant to practitioners who need state-of-the-art detection accuracy under strict latency and pre-training budgets, and to anyone studying how backbone design choices interact with pre-training data scale. Readers should already be comfortable with COCO mAP reporting, mAP variants (AP50, AP75, AP_S, AP_M, AP_L), and the basics of attention mechanisms.

Authors’ abstract

Real-time object detection is crucial for real-world applications as it requires high accuracy with low latency. While Detection Transformers (DETR) have demonstrated significant performance improvements, current real-time DETR models are challenging to reproduce from scratch due to excessive pre-training overheads on the backbone, constraining research advancements by hindering the exploration of novel backbone architectures. In this paper, we want to show that by using general good design, it is possible to have \textbf{high performance} with \textbf{low pre-training cost}. After a thorough study of the backbone architecture, we propose EfficientNAT at various scales, which incorporates modern efficient convolution and local attention mechanisms. Moreover, we re-design the hybrid encoder with local attention, significantly enhancing both performance and inference speed. Based on these advancements, we present Le-DETR (\textbf{L}ow-cost and \textbf{E}fficient \textbf{DE}tection \textbf{TR}ansformer), which achieves a new \textbf{SOTA} in real-time detection using only ImageNet1K and COCO2017 training datasets, saving about 80\% images in pre-training stage compared with previous methods. We demonstrate that with well-designed, real-time DETR models can achieve strong performance without the need for complex and computationally expensive pretraining. Extensive experiments show that Le-DETR-M/L/X achieves \textbf{52.9/54.3/55.1 mAP} on COCO Val2017 with \textbf{4.45/5.01/6.68 ms} on an RTX4090. It surpasses YOLOv12-L/X by \textbf{+0.6/-0.1 mAP} while achieving similar speed and \textbf{+20\%} speedup. Compared with DEIM-D-FINE, Le-DETR-M achieves \textbf{+0.2 mAP} with slightly faster inference, and surpasses DEIM-D-FINE-L by \textbf{+0.4 mAP} with only \textbf{0.4 ms} additional latency. Code and weights will be open-sourced.