Skip to content
AI.info

Computer vision

Instance and Panoptic Segmentation

Learn how instance masks, panoptic scene maps, matching, mask quality, occlusion policy, and crowded-scene evaluation work together.

By the end you can

Comparison

Semantic, instance, and panoptic scene representations

The three tasks encode different notions of identity and completeness. COCO fixes the distinction in its own ontology. The panoptic task runs over 133 categories: 80 of them things carried over from the detection task, 53 of them stuff.

Semantic segmentation would put a label on every pixel of that list and stop. Instance segmentation would count only the 80. Panoptic segmentation has to do both at once, and then decide, pixel by pixel, which of two competing owners wins.

FigureComparison · 3 columns

Semantic segmentation

Every pixel receives a category, but same-class objects merge.

  • Good for material or land-cover maps
  • No per-object identity
  • Simple class field
  • Example: road versus sidewalk

Instance segmentation

Each countable object receives its own mask.

  • Supports count and shape
  • Usually omits amorphous background identity
  • Requires instance matching
  • Example: every vehicle

Panoptic segmentation

Every pixel receives a class and relevant instance ID.

  • Combines things and stuff
  • Produces a complete scene map
  • Needs conflict resolution
  • Example: urban scene parsing

Instance segmentation inherits detection errors

Many instance systems first localize candidate objects and then predict a mask for each candidate. A missed box cannot receive a correct mask. A duplicate box can create duplicate masks.

The size of that dependency has been measured twice, independently. Hand one Mask R-CNN R50 the ground-truth boxes and it scores 53.6 mask AP. Make it predict its own boxes and it scores 37.2. That is a 16.4-point drop with the mask head untouched, and the Boundary IoU paper ran both halves of it in 2021.

TIDE reached the same conclusion from the error side in 2020. It decomposes mask results for Mask R-CNN R-101-FPN on COCO into six error types — Classification, Localization, Both, Duplicate, Background and Missed GT — and charges 9.3 AP50 to localization and 7.5 to missed ground truth, against only 3.1 to classification. False negatives alone cost 17.8 AP50.

Repairing the scores at the end does not undo that. TIDE's mask-scoring ablation moves AP50 by just +0.2, while localization error falls 1.5 and background error rises 0.7. Its table caption says why: “Mask scoring better calibrates localization, leading to decrease in localization error. However, by scoring based on localization, the calibration of other error types suffer.”

End-to-end designs may organize the pipeline differently. But they still solve instance assignment, classification, and spatial extent. Evaluate those components separately, and expect the earlier ones to dominate.

Mask quality cannot compensate for an object that was never proposed or matched: 53.6 mask AP with the boxes given, 37.2 without.

Visual

Building a panoptic scene map

A complete scene output must reconcile object masks and background regions. The reconciliation is not cosmetic. The non-overlapping property of the finished map is what makes panoptic matching unique in the first place. Hand a pixel to the wrong owner at the overlap-resolution step, and you have moved the score at the scoring step.

FigureProcess · 5 steps

  1. 1. Predict thing instances

    Identify countable objects such as people, cars, or packages.

  2. 2. Predict stuff regions

    Label amorphous areas such as sky, road, grass, or wall.

  3. 3. Resolve overlaps

    Choose ownership where masks compete for the same pixels.

  4. 4. Apply confidence and size rules

    Remove unsupported fragments and preserve valid small objects.

  5. 5. Produce one coherent map

    Assign every included pixel a semantic label and applicable instance ID.

Example

Mask policies that must be written before annotation

Different policies produce visibly different masks for the same scene. Serious datasets answer these questions in writing, before anyone opens the annotation tool. Cityscapes settled the holes and transparency questions by fiat: “Labeled foreground objects must never have holes, i.e. if there is some background visible ‘through’ some foreground object, it is considered to be part of the foreground. This also applies to regions that are highly mixed with two or more classes: they are labeled with the foreground class. Examples: tree leaves in front of house or sky (everything tree), transparent car windows (everything car).” The same page settles crowds and touching objects: “if the boundary between such instances cannot be clearly seen, the whole crowd/group is labeled together and annotated as group, e.g. car group”, and “This label is not included in any evaluation and treated as void”.

Writing the policy down is the cheap part. The Cityscapes team reports that “Annotation and quality control required more than 1.5 h on average for a single image” for the 5,000 fine images, against under 7 minutes for the 20,000 coarse ones. Thirty images annotated twice by different annotators agreed on 96% of pixels, 98% ignoring void. Google's Fluid Annotation team cites the identical cost from outside the project: “Fully annotating one image of the Cityscapes dataset took 1.5 hours.”

The occlusion question, the one that looks least decidable, has also been measured. Multiple annotators labeled all 500 BSDS images with amodal masks plus a partial depth order. What came back: “The region consistency of our amodal regions is substantially higher than the consistency of the original modal regions: median of 0.723 versus 0.425.” Annotators agreed more about the hidden extent of an object than the earlier project's annotators had agreed about its visible one.

That team went on to annotate 5,000 COCO images. KINS applied the policy to KITTI at scale in 2019 — 7,474 training and 7,518 test images — under a comparable protocol: “Each image is labeled by three experienced annotators. The final annotation of each instance is determined by crowd-sourcing to deal with the ambiguity”. The hidden part of an object is a decidable question. It just has to be decided in advance.

  • Occluded object: mark only visible pixels, or infer the full hidden silhouette? The inferred version is not the vaguer one — amodal regions reached a median pairwise consistency of 0.723 against 0.425 for the modal BSDS labels.
  • Transparent object: include the visible boundary, the contents, reflections, or only opaque material? Cityscapes chose the widest option and wrote it into the policy as “everything car”.
  • Loose material: treat a pile of grain as one instance, many particles, or a stuff region? No benchmark cited here answers this one for you; the project has to.
  • Connected objects: separate touching products using visual boundaries or known inventory metadata? Where the boundary cannot be clearly seen, Cityscapes stops trying and emits one group label, e.g. car group.
  • Crowd: annotate individuals, one crowd region, or an ignore area? COCO ignores it, Open Images V7 pays exactly one true positive for it, LVIS forgives the precision but still charges the recall.
  • Holes: preserve openings in handles and rings, or fill them as part of the object mask? Cityscapes forbids holes outright, so background seen through a foreground object is annotated as foreground.

Case

One flag in the COCO schema decides what counts as a miss

COCO wrote its crowd policy directly into the annotation file format. The data-format page states that “The segmentation format depends on whether the instance represents a single object (iscrowd=0 in which case polygons are used) or a collection of objects (iscrowd=1 in which case RLE is used)”, and that “Crowd annotations (iscrowd=1) are used to label large groups of objects (e.g. a crowd of people)”.

The reference implementation reads that flag and acts on it. In cocoeval.py, _prepare() sets the ground-truth ignore field from iscrowd, the iscrowd vector is passed into maskUtils.iou(), and evaluateImg() lets one crowd absorb more than one detection — “if this gt already matched, and not a crowd, continue”. A detection landing inside a crowd is therefore neither rewarded nor punished.

That is one answer, and reading it as the answer is the mistake. Open Images V7 draws a box instead. Google's documentation says “we drew a single box around groups of objects (e.g., a bed of flowers or a crowd of people) if they had more than 5 instances which were heavily occluding each other and were physically touching”, and its evaluation protocol states “If at least one detection is inside group-of box a single True Positive is scored”. A crowd there is worth one hit, no more and no less.

LVIS refuses both settlements. It is a federated dataset: it judges a category only on images where that category was exhaustively annotated, recording a per-image exhaustiveness flag that is true in 91% of cases. On the rest it suppresses false positives while still measuring recall, because “Missing annotations often occur in ‘crowd’ cases in which there are a large number of instances and delineating them is difficult”. Its authors name what they are departing from: “Our strategy differs from other datasets that use a small maximum number of instances per image, per category (10-15) together with ‘crowd regions’ (COCO) or use a special ‘group of c’ label to represent 5 or more instances (Open Images).”

Three benchmarks, three verdicts on the same photograph: ignored, worth one true positive, or excused from precision but still owed recall. None of that is a bookkeeping detail of a schema. It decides what counts as a miss and what is quietly excused. A model tuned against one policy is being graded on a different exam under the next.

Analogy

Cutting layered paper shapes from a collage

An artist traces and cuts every foreground object from a collage, then colors the remaining background materials. Overlaps require deciding which layer owns each visible piece. Cityscapes was annotated in exactly this way. Its fine images were drawn as layered polygons from back to front, so the order in which the shapes were laid down is itself an implicit depth ordering of the scene.

Cut paper cannot represent hidden geometry or uncertain transparency. Recovering the hidden part took a separate annotation protocol and a separate agreement study before anyone could train on it. Identity, mask ownership, and full-scene coverage are what the collage does show.

Instance masks describe visible assigned regions under an explicit ownership policy.

Key idea

A high class score does not guarantee a high-quality mask

A detector may be confident that an object exists while its mask misses thin parts or bleeds into neighbors. Ranking masks by class confidence therefore prioritizes the wrong property. The Mask Scoring R-CNN paper put the objection in one sentence: “It is inappropriate to use classification confidence to measure the mask quality since it only serves for distinguishing the semantic categories of proposals, and is not aware of the actual quality and completeness of the instance mask.”

Then it measured the gap. Running Mask R-CNN with a ResNet-18 FPN backbone on COCO 2017 val, the authors found classification score and MaskIoU “not well correlated”. A learned MaskIoU head that predicts mask quality directly reaches a correlation of about 0.74 with ground-truth MaskIoU, and adding it lifts AP “by about 1.5%” across backbones.

The controlled version of that result comes from someone else's code. In OpenMMLab's mmdetection model zoo, Mask R-CNN R-50-FPN 1x scores 38.2 box AP and 34.7 mask AP. MS R-CNN R-50-FPN 1x scores the identical 38.2 box AP and 36.0 mask AP. Detection did not improve at all. The +1.3 mask AP was bought purely by rescoring masks the model had already drawn.

Inspect mask-specific quality estimates, overlap distributions, and boundary errors. The score used for decision-making should correspond to the property the product needs.

Rescoring the same masks moved mask AP from 34.7 to 36.0 while box AP stayed at 38.2 — evidence that the two scores are measuring different things.

Steps

Analyze instance segmentation as a matched set

Use one-to-one matching to reveal where scene decomposition failed: match predicted and reference instances under a declared protocol, separate missing from duplicate instances, measure mask fidelity, review conflict resolution, and evaluate scene completeness.

The mask-fidelity step is the one that most often lies to you. The reason is arithmetic, not modeling: “Mask IoU divides the intersection area of two masks by the area of their union. This measure values all pixels equally and, therefore, is less sensitive to boundary quality in larger objects: the number of interior pixels grows quadratically in object size and can far exceed the number of boundary pixels, which only grows linearly.” That is the Boundary IoU paper, stating the defect it was built to expose.

One mask is enough to see it. For a Mask R-CNN horse mask the Mask IoU is 89% while the Boundary IoU is 69%. PointRend's visibly cleaner mask on the same object moves Mask IoU by +8% and Boundary IoU by +22%. The metric that changes least is the one most people report.

At dataset scale the two metrics disagree about which objects are the problem. On COCO val with ground-truth boxes Mask R-CNN scores Mask AP 52.5 against Boundary AP 36.1, and the size breakdown inverts: mask AP climbs from AP_S 44.9 to AP_L 66.0 while boundary AP falls from AP_S 44.8 to AP_L 25.9. Large objects are where the model looks best and where its outlines are worst.

Mask Transfiner reports the same divergence from the improvement side. On Cityscapes it lifts boundary AP from 11.4 to 18.0 while mask AP reaches 37.6, and its abstract keeps the two apart on purpose, quoting “+3.0 mask AP” separately from “+6.6 boundary AP”. Whichever single number a team reports, the other one is moving too.

FigureProcess · 5 steps

  1. 1. Match predicted and reference instances

    Use class and mask overlap under a declared protocol.

  2. 2. Separate missing and duplicate instances

    Distinguish no prediction from repeated predictions.

  3. 3. Measure mask fidelity

    Inspect overlap, boundary, holes, and thin-structure errors.

  4. 4. Review conflict resolution

    Find pixels lost or reassigned when masks overlap.

  5. 5. Evaluate scene completeness

    Check stuff coverage, void regions, and panoptic consistency.

Panoptic quality combines recognition and segmentation

Panoptic evaluation typically links correctly matched segments with their overlap quality, while penalizing unmatched predictions and references. The summary remains sensitive to class balance and matching rules.

The arithmetic was fixed in the definition of panoptic quality: “a predicted segment and a ground truth segment can match only if their intersection over union (IoU) is strictly greater than 0.5. This requirement, together with the non-overlapping property of a panoptic segmentation, gives a unique matching”. That is Kirillov and colleagues, defining the metric. PQ then factors cleanly — “PQ = SQ × RQ”, where SQ “is simply the average IoU of matched segments” and RQ, written |TP| / (|TP| + ½|FP| + ½|FN|), “is the familiar F1 score”. COCO runs the task over 133 categories, 80 things and 53 stuff. The same paper put people through the metric, using “30 doubly annotated images for Cityscapes, 64 for ADE20k, and 46 for Vistas”, and reported that “As expected, humans are not perfect at this task”.

The headline number still hides its own parts. Mask2Former with a Swin-L backbone reports 57.8 PQ on COCO panoptic val2017. Inside that one figure sit 64.2 PQ on things and 48.1 PQ on stuff — a 16.1-point spread a team tracking only PQ would never see. The same model reports 50.1 AP for instance segmentation and 57.7 mIoU on ADE20K. OneFormer reproduces the identical 57.8 / 64.2 / 48.1 and 50.1 in its own comparison table, so the gap belongs to the model rather than to one evaluation run.

Mask2Former's shipped post-processing also declines to collapse the two scores this lesson keeps separating: “We multiply class confidence and mask confidence (i.e., averaged foreground per-pixel binary mask probability) for a final confidence.”

Report thing and stuff behavior separately, plus important classes and region sizes. Scene-level maps should also be inspected for seams, voids, and implausible ownership.

One panoptic score cannot explain whether the problem is recognition, segmentation, or scene reconciliation — 57.8 PQ was 64.2 on things and 48.1 on stuff.

Key takeaways