DETR — Detection Transformer

What DETR Changed#

Before DETR, every competitive detector was a stack of hand-crafted components: anchor boxes, region proposals (Faster R-CNN) or grid cells (YOLO), per-anchor classification and regression heads, and a non-maximum suppression pass to deduplicate overlapping predictions. Each piece had hyperparameters and failure modes; together they made detection pipelines hard to port across datasets.

DETR collapsed the entire pipeline into a single transformer that emits a fixed set of N predictions (typically 100) in one forward pass. Training matches predictions to ground-truth boxes one-to-one with the Hungarian algorithm and supervises both class and box. There is no NMS — duplicate suppression is learned implicitly because the bipartite matcher will only reward one prediction per ground-truth box.

Architecture#

DETR is conceptually simple: a CNN backbone (ResNet-50 or ResNet-101 in the original paper) produces a feature map, which is flattened and fed into a transformer encoder. A transformer decoder then attends over the encoded features using N learned object queries — slots that each emit one prediction. Final feed-forward heads turn each query's decoded vector into a class label and a normalised box.

• Backbone — ResNet (frozen-BN), producing a 2D feature map.
• Encoder — standard transformer encoder over the flattened spatial tokens with sinusoidal 2D positional encoding.
• Decoder — transformer decoder with N learned object queries cross-attending to the encoder output.
• Heads — shared MLP per query producing (class logits, normalised bbox).
• Loss — Hungarian-matched class cross-entropy + L1 + Generalised IoU on boxes.

The Convergence Problem#

Vanilla DETR needed roughly 500 epochs of COCO training to match Faster R-CNN, against the 12 to 36 epochs typical for CNN detectors. The slow convergence had two root causes: the decoder's cross-attention was dense over the full feature map, making early-training query-to-feature alignment noisy; and the object queries themselves were content-only with no spatial prior, so they had to learn 'where to look' from scratch.

Every subsequent transformer detector tackles one of those problems. Deformable DETR (Zhu et al., 2020) replaced dense attention with sparse deformable sampling around reference points, cutting training to 50 epochs. DAB-DETR re-parameterised queries as anchor boxes. DINO combined denoising training and mixed query selection. RT-DETR (Lv et al., 2023) carried this lineage into a real-time-capable detector.

Object Queries Explained#

The object queries are the conceptual heart of DETR. Each is a learned embedding that, after training, develops a specialty — one query reliably emits 'large object near the centre', another 'small object on the left edge'. Visualising attention maps shows queries acting like learned spatial templates.

Setting N too low caps the number of detections per image; the COCO default of 100 is comfortably above the maximum object count in the dataset. Modern variants (DINO, Group DETR) push N higher and add denoising or grouped queries to stabilise training.

Why DETR Mattered#

DETR is the architectural inflection point that brought detection into the transformer era. It proved that hand-crafted post-processing — NMS, anchor design, proposal generation — was not necessary. It set the template for set prediction in dense vision tasks (segmentation, tracking, pose). And it opened the door to multi-modal detection, where the query embeddings can be conditioned on text (OWL-ViT, Grounding DINO).

For Yobitel deployments, the practical successor is RT-DETR — real-time-capable, Apache-licensed, and inherits the DETR design lineage without the convergence pain.