Multi-Scale Hierarchies and Feature Pyramids

A road-scene model must find lane markings, cars, and large construction zones. The spread is not an exotic case. On the COCO detection benchmark roughly 41% of annotated objects have an area below 32² pixels, while 24% exceed 96². Most instances sit at the small end of the range one network has to cover.

Early feature maps retain many positions. They often represent local patterns. Deeper maps cover larger regions and combine evidence into more abstract channels.

These maps are not interchangeable copies at different resolutions. Fusing them means lining up coordinates, channels, normalization, and semantic level. The default fusion rule is to add them together, which ignores all of that. EfficientDet's authors put the objection plainly in 2020: “While fusing different input features, most previous works simply sum them up without distinction; however, since these different input features are at different resolutions, we observe they usually contribute to the fused output feature unequally.”

Their COCO validation ablation prices the distinction. A repeated top-down FPN scores 42.29 AP. The same bidirectional architecture without weighting reaches 43.94 AP. With learned per-input weights it reaches 44.39 AP.

A separate group at Beihang University found the same defect. Their ASFF work calls it “the inconsistency across different feature scales”, and names it a primary limitation of feature-pyramid single-shot detectors. Their fix filters the levels spatially rather than weighting them, and reports 43.9% AP at 29 FPS on MS COCO. Two labs, one diagnosis. Adding levels together is a modelling choice, and usually the wrong one.

A feature pyramid creates semantically stronger maps at several resolutions. A bottom-up backbone produces the hierarchy. A top-down pathway carries deep semantics toward finer grids. Lateral connections re-inject preserved backbone detail. Each head then consumes the level matched to the structures it must find.

One arrow is missing from that picture. PANet added it in 2018, and its argument is about path length, which is countable: “This process yields a "shortcut" (dashed green line in Figure 1), which consists of less than 10 layers, across these levels. In comparison, the CNN trunk in FPN gives a long path (dashed red line in Figure 1) passing through 100+ layers from low layers to the topmost one.” Fine localisation evidence in the early maps reaches the deep semantic levels only after traversing that trunk.

The added path is short and it pays. On COCO val-2017 the bottom-up augmentation consistently improved mask AP by more than 0.6 and box AP by more than 0.9, with or without adaptive feature pooling. The system took 1st place in the COCO 2017 Challenge Instance Segmentation task and 2nd in Object Detection. EfficientDet's authors later re-measured the same comparison in another lab: repeated FPN plus PANet at 44.08 AP, against 42.29 AP for repeated top-down FPN alone. The extra path is under ten layers long. That is the point of it.

They differ in cost, alignment, and where scale variation is handled. An input image pyramid buys direct scale coverage by re-running the model on resized copies. A feature pyramid reuses one backbone hierarchy and fuses levels. The other two options carry measured trade-offs worth stating in numbers.

Dilation keeps a fine grid while widening context. Yu and Koltun measured what that is worth in 2016. Mean IoU on the PASCAL VOC-2012 validation set rose from 69.8% for the front-end alone, to 71.3% with the Basic context module, to 72.1% with the Large one.

The same operator has a characteristic failure. The same two researchers named it the following year, in a paper with one more author: “Gridding artifacts occur when a feature map has higher-frequency content than the sampling rate of the dilated convolution.” Two of those three authors wrote the original dilation paper, so that is not independent confirmation. A genuinely unrelated team of seven authors found the defect concurrently. They called it the “gridding issue”, attacked it with hybrid dilated convolution, and reported 80.1% mIoU on Cityscapes test.

Parallel multi-branch designs take the opposite route. HRNet, published in 2019, keeps four resolution subnetworks running side by side rather than in series: “Instead, our proposed network maintains high-resolution representations through the whole process.” Its COCO keypoint ablation prices the exchange between those branches, and the series is monotone. One final fusion gives 70.8 AP. Three across-stage fusions give 71.9 AP. Eight across- and within-stage fusions give 73.4 AP. HRNet-W48 reached 75.5 AP on COCO test-dev, and 77.0 AP with extra training data. The exchange is the architecture, not a finishing step.

Aggregate performance can conceal a pyramid that fails at one end of the scale range. Use the slice a real benchmark defines rather than an informal one. COCO's detection protocol splits average precision by object area: APsmall below 32² pixels, APmedium between 32² and 96², APlarge above 96².

That split is not cosmetic, because the distribution is lopsided. COCO's own evaluation page states it: “In COCO, there are more small objects than large objects. Specifically: approximately 41% of objects are small (area < 32²), 34% are medium (32² < area < 96²), and 24% are large (area > 96²). Area is measured as the number of pixels in the segmentation mask.” Singh and Davis measured the same skew independently in 2018. The median object scale relative to the image is 0.106 in COCO against 0.554 in ImageNet, with the smallest and largest 10% of COCO instances at 0.024 and 0.472.

Ablating the fusion paths is the other half, and the FPN authors ran it themselves, on COCO minival with a ResNet-50. A single-scale RPN on the C4 map reaches AR¹ᵏ 48.3, with AR¹ᵏ on small objects at 32.0. The full feature pyramid reaches 56.3 and 44.9. That is 8.0 points overall and 12.9 on small objects. The small slice moves roughly half again as much as the aggregate that would otherwise report the result.

Each path costs something to remove. Deleting the top-down pathway drops AR¹ᵏ to 49.5. Deleting the 1×1 lateral connections drops it to 46.1, about 10 points below the full FPN. The authors explain why the laterals are not redundant with the top-down stream: “This top-down pyramid has strong semantic features and fine resolutions. But we argue that the locations of these features are not precise, because these maps have been downsampled and upsampled several times.”

In practice: partition by structure size against declared thresholds, separate recognition from localisation and contour accuracy, test controlled scale shifts, visualise which level or branch handled each case, and remove one fusion path at a time to see what it was contributing.

Hard rules that send each target size to one level simplify training, and FPN's is written down: a region of interest goes to level k = ⌊k₀ + log₂(√(wh)/224)⌋, where 224 is the canonical ImageNet pre-training size and k₀ is set to 4. The rule is a convention about image sizes. It is not a measurement of the instance in front of it. Three researchers at Carnegie Mellon stated the objection in 2019: “In other words, the anchor matching mechanism is inherently heuristic-guided. This leads to a major flaw that the selected feature level to train each instance may not be optimal.”

Then they priced it. On COCO minival with a ResNet-50 backbone, anchor-free branches using heuristic feature selection scored 34.7 AP. Content-based online feature selection raised that to 35.9 AP, a gain of 1.2. The joint anchor-based plus anchor-free detector reached 37.2 AP, against RetinaNet's 35.7.

Borders between levels can create discontinuities. Learned or soft assignment is less rigid, and more complicated. In one careful measurement, 1.2 AP is what that complication was worth.

Choose assignment using the target distribution and deployment geometry. A pyramid tuned on one camera setup may fail when focal length or resolution changes.