EAGLE: Entropy-Aware Geometric Learning Enhancer for Partially Polygon-Supervised Object Localization

AI summary

Overview

Research area: Computer vision, specifically object detection and localization, combining information theory with statistical shape analysis (Kendall shape spaces) and partial polygon supervision.

Technical level: Advanced. The paper develops formal entropy derivations, a geodesic-distance formulation on Kendall shape space, and a Fano-type inequality, alongside a lightweight neural refinement head.

Scope (one sentence): The paper builds a mathematical framework for quantifying per-class shape uncertainty from bounding-box annotations alone, validates that measure empirically, and uses it to drive a post-training refinement head called EAGLE that improves detector localization using only partial polygon supervision.

What This Paper Is About

Detectors trained only on bounding boxes lack the geometric precision to do well at high IoU thresholds, and adding polygon masks during training is expensive to annotate. Partial polygon supervision (masks on a fraction of instances) reduces that cost, but the authors note that for low-complexity object classes the added segmentation loss can actually obscure box localization and hurt performance. EAGLE addresses this by first measuring, per class, how much shape uncertainty remains given only a bounding box, then using that entropy signal to drive a small add-on head that refines box edges and rescoring without touching the frozen detector.

Key Contributions

1. An entropy framework for shape uncertainty under partial polygon supervision. The authors derive an upper bound on continuous conditional shape entropy h(S|A_box) via tangent-space covariance in Kendall shape space, an explicit closed form for the discrete entropy H(S|A_box) using aspect-ratio bins (Knuth's Rule) and shape clusters (Tibshirani's Gap Statistic), and an upper bound on expected box IoU that is strictly decreasing in post-inference shape uncertainty.

2. Empirical validation of H(S|A_box) as a heuristic. The measure is tested as a per-class predictor of shape reconstruction error on MS COCO 2017 (with K = 24 vertices) and as a predictor of the localization advantage of mask-derived boxes over native detection boxes across YOLO26 variants on the 20 Pascal SBD classes.

3. The EAGLE head itself. A lightweight, post-training, entropy-conditioned geometric refiner with a proposal selector and a set of postprocessing transforms, trainable with a budget of polygon instances rather than full polygon supervision — demonstrated with only 40% of objects polygon-annotated.

4. Cross-framework evaluation. EAGLE is trained and measured on CNN detectors (YOLO26n, YOLO26m) and a transformer detector (RT-DETRv2-S with ResNet-18) on MS COCO 2017, reporting accuracy gains at minimal measured latency overhead.

Main Findings

• Entropy strongly predicts modal shape error: across MS COCO 2017 classes, the correlation between H(S|A_box) and modal-cluster reconstruction loss was r = 0.921, ρ = 0.886, with permutation p = 7.56·10⁻³³ and R² = 0.848.

• Entropy also predicts mask reconstruction error, but more weakly: r = 0.732, ρ = 0.803, permutation p = 2.66·10⁻¹⁴, R² = 0.536.

• Positive but modest correlation with the mask-box AP gap on Pascal SBD: all YOLO26 variants showed positive Spearman correlation between per-class entropy and Δ_c = AP_mask − AP_box, with ρ = 0.553, 0.256, 0.461, 0.186 and 0.319 for n, s, m, l and x respectively. Averaging Δ_c across models gave ρ = 0.298 with no outliers; excluding the lowest-correlation model (YOLO26l) from the average produced one outlying class and raised ρ to 0.540. The authors describe this as a promising but outlier-dependent result.

• Consistent mAP gains on MS COCO 2017 with a 40% polygon budget: the enhancer raised mAP50-95 by roughly 0.64 points on YOLO26n, 0.27 on YOLO26m and 0.06 on RT-DETRv2-S; with the third postprocessing phase these rose to 0.9, 0.45 and 0.16 respectively.

• Larger gains at mAP50: the optimal bounded combination produced mAP50 gains of 1.4, 0.8 and 0.29 points on the same three frameworks.

• Headline table numbers (YOLO26n): baseline AP 40.25, mAP50 55.80, mAP75 43.65, 2.4M params, 2.3 ms latency; enhancer 40.89 / 56.77 / 44.17 at 2.7M params and 2.3 ms; duplicate damping 41.02 / 57.02 / 44.30 at 2.7M and 2.4 ms; bounded combination 41.11 / 57.19 / 44.25 at 2.7M and 2.3 ms.

• Headline table numbers (YOLO26m): baseline 52.49 / 69.69 / 57.25 at 20.4M params and 6.4 ms; enhancer 52.76 / 70.07 / 57.50; duplicate damping 52.88 / 70.28 / 57.64; bounded combination 52.94 / 70.48 / 57.56, all at 20.7M params and 6.4 ms.

• Headline table numbers (RT-DETR): baseline 48.10 / 65.07 / 52.09 at 20.0M params and 6.8 ms; enhancer 48.16 / 65.17 / 52.17 at 20.3M and 6.9 ms; duplicate damping 48.25 / 65.36 / 52.24 at 20.3M and 6.8 ms; bounded combination 48.26 / 65.36 / 52.26 at 20.3M and 6.9 ms.

• Small, bounded cost: a parameter increase of 300,000 is reported uniformly across all methods and frameworks, translating to a latency increase bounded by 0.1 milliseconds across all models.

• Recovering the segmentation penalty: the paper reports that native YOLO26n instance segmentation box AP is 0.5 points below standard object detection, while EAGLE provides a 0.9 point increase, surpassing the prior polygon-supervised model by roughly 1.4 mAP50-95 points after postprocessing.

• No degradation observed: every other method combined with the EAGLE enhancer produced scores exceeding the baseline, according to the authors.

Methodology in Plain English

Step 1 — Measure how ambiguous a class's shape is from boxes alone. The authors convert each polygon mask into a shape descriptor that ignores position, size and rotation, so only the intrinsic outline remains (a planar Kendall shape space with K landmarks). They compute the average shape, project every instance into a flat tangent space around that average, and build a local covariance matrix of shape variation, weighted by how similar each instance's aspect ratio is to a query ratio. The log-determinant of that covariance gives an upper bound on how many bits of shape information remain unknown when only a box is seen. A discrete version bins aspect ratios with Knuth's Rule and clusters shapes with the Gap Statistic (B = 100 reference sets), then computes a conditional entropy over bins and clusters.

Step 2 — Connect that ambiguity to a performance ceiling. Using a neighborhood-based Fano-type inequality, the authors show that expected box IoU is capped by a strictly decreasing function of the remaining shape uncertainty given the model's prediction. High residual ambiguity implies a lower achievable ceiling.

Step 3 — Turn entropy into a refinement signal. A frozen detector produces boxes and feature maps. For each prediction, entropy is computed as a weighted mix of the class-and-aspect-ratio-bin entropy and the per-class entropy (weights 0.8 and 0.2). For each of 128 box candidates, a 9×9 feature grid is extracted, mapped to 48 channels, and an MLP predicts attention over nine positions along each of the four edges; the four edge vectors plus confidence descriptors form a 246-dimensional input to a 192-dimensional channel. Entropy conditions this representation additively, and four linear outputs predict distributions over 21 displacement bins in [−1, 1] whose expectations define the edge shifts, scaled by the box size and an entropy-dependent gate. The final projection is zero-initialized, so an untrained head leaves accurate predictions untouched. Training weights are entropy-proportional, clipped to [0.65, 1.35], and the loss combines localization, displacement-bin cross-entropy, an IoU-improvement margin and polygon-boundary attention supervision.

Step 4 — Select and postprocess. A separate selector chooses between the native box and refinements at 0.5, 0.75, 1 or 1.25 times the proposed displacement, switching from the soft default only when predicted utility minus predicted harm exceeds a calibrated threshold. Optional postprocessing applies entropy penalties, residual score calibration, geometric bias correction, duplicate damping and refinement-strength adjustment, ending with a bounded combination of these.

Step 5 — Evaluate. Entropy is correlated against reconstruction error per MS COCO class, against mask-derived versus native box AP per Pascal SBD class, and finally EAGLE is trained and benchmarked on three detector frameworks.

Why This Matters

Research impact. The paper offers a quantifiable, dataset-level alternative to manual intuition about which classes benefit from mask supervision. If a class's shape entropy is low, the framework predicts that joint mask learning buys little and may interfere; if high, geometric refinement should pay off. This reframes the mask-versus-box tradeoff as a measurable information problem and gives a principled selection criterion for partial annotation budgets. It also extends annotation-level embeddings into Kendall shape space, a formulation the authors state does not previously exist for boxes and polygons.

Real-world applications (drawn from the paper's framing of detection as a first-stage component):

• Industrial real-time detection pipelines where bounding-box-only training is standard and annotation budgets are tight.
• Robotics systems that consume detector outputs as an initial localization step.
• Region proposal generation for Vision Language Models, where tighter boxes improve downstream grounding.
• Object tracking and initial passes for 3D computer vision pipelines, which inherit localization error from the 2D stage.

Industry relevance. The cost story is concrete: mask annotation is described as increasing manual effort by a near tenfold factor, and EAGLE is trained with 40% of objects polygon-annotated, with only 10% of all instances used to compute the discrete entropy. The deployment story is also concrete: 300,000 extra parameters and at most 0.1 ms added latency, with the head operating as a post-training add-on to a frozen model, so no retraining of the base detector is required.

Future Directions

• Strengthen the box-level predictive claim. The mask-derived-versus-native-box AP correlation is the weakest result in the paper (ρ = 0.298 averaged across all five YOLO26 variants, requiring outlier exclusion and rising to ρ = 0.540 only when YOLO26l is dropped). A tighter link between entropy and expected box gains would make the framework a more reliable budget-allocation tool.

• Broaden the framework and dataset coverage. EAGLE is trained on three detectors (YOLO26n, YOLO26m, RT-DETRv2-S with ResNet-18) on MS COCO 2017, and the mask-box analysis covers YOLO26 variants on 20 Pascal SBD classes. Generality to other detector families, to non-COCO distributions and to classes with unusual shapes is not established.

• Push the transformer result further. Gains on RT-DETRv2-S (0.06 mAP50-95 for the enhancer, 0.16 with the third postprocessing phase, 0.29 at mAP50) are much smaller than on YOLO26n, so understanding why the head helps some architectures far more than others is an open question.

• Reduce reliance on postprocessing. The largest numbers come from the bounded combination of five postprocessing transforms whose coefficients and operating points are fixed before evaluation. Whether the geometric head alone can approach those gains, and how sensitive the pipeline is to those fixed operating points, is not reported.

• Explore sensitivity to the supervision budget. Only the 40% polygon budget and the 10%-of-instances entropy estimate are reported; how performance degrades below or scales above that budget is not reported.

Target Audience

Researchers and engineers working on object detection, weakly and partially supervised localization, and instance segmentation who want to reduce mask annotation cost without losing localization precision. It also suits readers interested in information-theoretic analyses of deep models or in statistical shape analysis applied to vision problems, since the entropy derivations stand somewhat independently of the EAGLE architecture. Practitioners deploying frozen detectors who want a small, low-latency accuracy add-on will find the engineering details (zero-initialized refinement, entropy-weighted training, selector, postprocessing) directly useful. A background in detection metrics (mAP, IoU) and comfort with entropy and Riemannian geometry is needed to follow the theory sections.

Authors’ abstract

Object detection is a critical application of computer vision, yet models trained uniformly on bounding boxes lack the geometric precision to be fully effective in inference settings, especially at high IoU thresholds. One common approach is the introduction of polygon-supervised localization, where a model is trained explicitly for localization but with access to polygon masks during training. Among such methods, partial polygon supervision is viable due to the higher annotation costs of developing fully supervised training datasets. However, this introduces a further challenge: in object classes of relatively low shape complexity, the addition of training masks and subsequent segmentation loss may obscure box localization, often worsening model performance despite access to deeper geometric information. To address the practical limitations of both full and partial polygon supervision, we introduce a mathematical framework for quantifying per-class geometric complexity by combining information theory and statistical shape analysis. With this framework, we develop Entropy-Aware Geometric Learning Enhancer (EAGLE), a lightweight, post-training model head that learns geometric information for the localization model, improving accuracy via edge refinement and localization rescoring using entropy-informed updates. Empirical analysis across MS COCO and Pascal SBD demonstrates the entropy framework meaningfully predicts shape reconstruction error and accuracy differences among mask-derived and standard box predictions, while the implemented EAGLE framework achieves consistent improvement on already SOTA detection frameworks (+0.9 mAP50-95 points on MS COCO) at minimal observed inference overhead.