Skip to content
AI.info

Computer vision

Semantic Segmentation: Pixelwise Prediction

Design semantic segmentation systems with coherent class maps, dense losses, overlap metrics, boundary checks, and deployment-aware post-processing.

By the end you can

Visual

A label field over the image

Semantic segmentation predicts a class for each spatial location without separating instances of the same class.

FigureLayers · 5 layers
  1. 01

    Input image

    Provides color, intensity, texture, and context on a spatial grid.

  2. 02

    Dense representation

    Preserves enough resolution to support per-location decisions.

  3. 03

    Class logits per pixel

    Assigns evidence for each semantic category at each location.

  4. 04

    Spatial refinement

    May restore detail, enforce constraints, or remove small regions.

  5. 05

    Decision mask

    Feeds measurement, visualization, planning, or downstream rules.

Comparison

Classification, detection, and semantic segmentation

Choosing the wrong task changes both annotation and evaluation.

FigureComparison · 3 columns

Image classification

Answers which classes describe the image.

  • No object location
  • Low annotation cost
  • Can use global context
  • Cannot measure region area

Object detection

Returns a box for each instance.

  • Separates object count
  • Approximate shape
  • Useful for localization
  • Includes background inside boxes

Semantic segmentation

Labels pixels by category.

  • Precise class regions
  • Does not separate same-class instances
  • Supports area measurement
  • Expensive boundary annotation

Four radiologists, the same scans, and no forced consensus

A road may blend into gravel, a lesion margin may be uncertain, and tree canopies overlap; annotators can disagree even when they inspect the same high-quality image.

The most-used public reference database for lung CT was built by people who expected that disagreement. They recorded it instead of averaging it away. Four experienced thoracic radiologists annotated 1,018 cases in two phases, and no consensus round was imposed at the end. LIDC-IDRI said so when it published the design in 2011: “The goal of this process was to identify as completely as possible all lung nodules in each CT scan without requiring forced consensus.”

The counts are the argument. The database holds 7,371 lesions marked 'nodule' by at least one radiologist. Of the 2,669 lesions marked 'nodule >=3 mm' by at least one radiologist, only 928 — 34.7% — received that mark from all four. Before any question of where a border runs, four experts reading the same images agreed on the existence of a lesion of that size in 928 cases out of 2,669.

The Cancer Imaging Archive hosts the collection: 1,018 cases, 1,010 subjects, 244,527 images. Its own description of the annotation matches, two phases and four radiologists with no forced consensus.

Specify boundary tolerance, ignore zones, and uncertainty handling. Forcing crisp labels can make the model appear wrong where the reference itself is unstable. In LIDC-IDRI's own numbers, the single crisp reference may never have existed.

Dense labels should represent annotation confidence as honestly as possible.

Case

Two annotators agreed on 96% of Cityscapes pixels

Cityscapes measured annotator disagreement instead of assuming it away. Its 5,000 finely annotated images were not cheap to make. “Annotation and quality control required more than 1.5 h on average for a single image”, the 2016 dataset paper reports.

Then the team ran the experiment most datasets skip. Thirty images went through the pipeline twice, in different hands: “30 images were finely annotated twice by different annotators and passed the same quality control. It turned out that 96 % of all pixels were assigned to the same label.” The remaining pixels are not carelessness. They are what two trained people, each spending over an hour and a half on an image, still disagree about.

The dataset also gave annotators a way to say they did not know. They “were instructed to choose a void label if unclear (such that the region is ignored in training and evaluation)”. Exclude the pixels either annotator marked void and the same experiment yields “98 % agreement”. A ground truth is a measured artifact with a known error bar. It is not a fixed point to score against.

Figure

What a pixel-level ground truth costs to make, and how much of it two careful people still disagree about.

Example

Why pixel counts distort the learning problem

Large background regions can dominate dense losses even when small classes matter most. A regulator's file shows how far the gap runs in a deployed product. GE Medical Systems' Auto Segmentation was cleared by the FDA on 4 May 2023, as 510(k) K230082, and tested against per-organ Dice acceptance criteria for 40 organs at risk. “These 2552 contours were generated from the 302 unique patient exams in the bench testing dataset,” the performance-testing section of the submission states.

Both the scores and the pass marks fall steeply with structure size. Body averaged 99.50% against a 98.1% criterion. Whole Brain averaged 98.53% against 93.0%. Chiasma averaged 43.81% — against a criterion of just 11.7%. Superior pharyngeal constrictor averaged 59.57% against 50.0%. The acceptance threshold itself had to be written per class, because no single bar could be both meaningful for the body outline and reachable for the chiasm.

An independent evaluation of two other commercial products found the same structure hardest. One institution, 40 patients, 22 organs, published in 2024: “the chiasm performed the lowest (0.28 ± 0.14)” for AccuContour, and “the chiasm performed the lowest (0.30 ± 0.14)” for ProtegeAI. The reason it gives is “small volumes of these structures as well as low contrast to the surrounding brain tissue in CT”.

  • Road scene: Sky and pavement occupy most pixels while pedestrians carry higher safety cost.
  • Medical scan: Healthy tissue dominates while a small lesion drives diagnosis — in K230082, Chiasma averaged 43.81% Dice where Body averaged 99.50%.
  • Satellite tile: Common vegetation overwhelms rare flood or fire regions.
  • Industrial surface: Normal material covers nearly everything while defects are tiny.
  • Document page: White background dwarfs characters and table lines.

Analogy

Coloring a map with imperfect borders

Every location on a map is assigned to water, forest, road, or building. Large regions are easy. Coastlines, mixed land, and narrow paths create disputed borders.

A map's borders are recorded decisions. Images contain projected appearance rather than official geographic boundaries. Region labeling and boundary uncertainty are what the two share.

Segmentation quality depends on both regional agreement and the treatment of boundaries.

Key idea

Pixel accuracy can reward a model that ignores rare classes

A model predicting only the dominant background may achieve high pixel accuracy when foreground is scarce. The number can look reassuring while the system misses every important region.

Report class-specific overlap, macro summaries, boundary quality, and performance by region size. Use task costs to decide which errors matter.

The Cityscapes benchmark says as much about its own headline number, in its own documentation. “The global IoU measure is biased toward object instances that cover a large image area”, it warns, which is why the benchmark additionally evaluates “using an instance-level intersection-over-union metric”, and why “pixels labeled as void do not contribute to the score”. Rarity is handled earlier still. The dataset defines “30 visual classes for annotation”, and those “that are too rare are excluded from our benchmark, leaving 19 classes for evaluation”. That decision ships in code: cityscapesScripts marks the rest ignoreInEval, “Whether pixels having this class as ground truth label are ignored during evaluations or not”. Mean IoU is the summary of record. The fully convolutional network paper headlined a model that “achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow”.

Overlap has a second blind spot, and it survives even after every class is scored separately. The Boundary IoU paper opens with the geometry of it: “This measure values all pixels equally and, therefore, is less sensitive to boundary quality in larger objects: the number of interior pixels grows quadratically in object size and can far exceed the number of boundary pixels, which only grows linearly.” The demonstration is a single horse. Mask R-CNN, BMask R-CNN and PointRend score Mask IoU 89%, 92% and 97% on it. Measured at the boundary, the same three masks score 69%, 78% and 91%. Improvements plainly visible in the contours, worth +9 and +22 Boundary IoU points, register in the Mask IoU column as +3 and +8.

A 78-author consortium reached the same conclusion by a different route. Working from a multi-stage Delphi process, and writing in Nature Methods in 2024, they state that “overlap-based metrics such as the DSC do not take the correctness of an object's boundaries into account”.

Large regions should not silence small critical structures.

Steps

Evaluate a segmentation model beyond one overlap score

Use complementary measurements and visual diagnostics. Step 2 — report per-class overlap — is the one most often published without a reference point. A Dice or IoU number means little until you know what two humans score against each other on the same task.

BRATS supplies that number for brain tumours. Twenty tumour-segmentation algorithms were applied to 65 multi-contrast MR scans annotated by up to four raters, and the raters were measured against each other first. Menze and colleagues put the result in the 2015 abstract: “Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%–85%), illustrating the difficulty of this task.” A per-class score on that task has to be read against a 74%–85% band, not against a perfect reference.

An independent group in Bern found the ordering can invert. Two expert raters segmented 25 glioblastoma cases in 2014, and for contrast-enhancing tumour volume the automatic method “had a better overlap with the ground-truth than Dice overlap between manual inter-rater segmentations”. Steps 3 to 5 exist because the same argument applies again inside each class. An overlap number, human or machine, does not say where the disagreement sits.

FigureProcess · 5 steps

  1. 1. Verify label policy

    Check class definitions, ignore regions, and boundary conventions.

  2. 2. Report per-class overlap

    Compute IoU or Dice for every relevant category.

  3. 3. Measure boundary behavior

    Evaluate contours or tolerance bands when edge precision matters.

  4. 4. Slice by region geometry

    Separate tiny, thin, fragmented, and large connected regions.

  5. 5. Inspect spatial errors

    Review holes, bleeding, missed islands, and systematic border shifts.

Loss functions change which pixels receive attention

Cross-entropy treats pixels through class likelihood. Overlap-oriented losses emphasize region agreement. Reweighting or focal variants can focus on rare or difficult pixels.

For thin structures the choice decides whether the structure survives at all. The clDice paper begins from what a vessel actually is: “For such structures, the topology is their most important characteristic; particularly preserving connectedness: in the case of vascular networks, missing a connected vessel entirely alters the blood-flow dynamics,” reads its abstract. Its centerlineDice measure is computed on the intersection of the masks with their morphological skeleta, and the authors prove it guarantees topology preservation up to homotopy equivalence for binary 2D and 3D segmentation. The differentiable soft-clDice loss was benchmarked on five public datasets of vessels, roads and neurons.

A region-overlap signal and a connectivity-aware one can therefore rank the same two predictions differently. The Nature Methods consortium analysed clDice against standard overlap measures in 2024 and found that it “uncovers the fact that Predictions 1 misses the fine-granular branches” — the branches the overlap scores pass over, because a missing capillary costs almost no pixels.

No loss automatically fixes missing labels, noisy boundaries, or domain shift. Compare optimization behavior and final task metrics rather than picking whichever loss is popular.

A loss is a training signal, not the definition of successful segmentation.

Example

Post-processing rules that require validation

Cleaning a mask can make it easier to use, or it can erase real evidence. The size of the effect is measurable rather than a matter of taste. DeepLab reported PASCAL VOC 2012 val mean IoU rising from 59.80% to 63.74% when a fully connected CRF is applied on top of the network, and from 64.21% to 68.70% for its multi-scale large-field-of-view variant. Its abstract puts the headline this way: “Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 71.6% IOU accuracy in the test set.”

A second group measured the same idea independently. On the reduced VOC 2012 validation set they put “Plain FCN-8s” at 61.3 mean IU against “FCN-8s and CRF disconnected” at 63.7, and 68.3 against 69.5 with COCO pre-training. 59.80% to 63.74% in one study, 61.3 to 63.7 in the other. A post-processing step is large enough to be worth having, which is exactly why an unvalidated one is large enough to take a comparable amount away. It takes it from the thin and tiny regions first.

  • Remove connected regions below a minimum area
  • Fill small holes within predicted objects
  • Enforce mutually exclusive class priorities
  • Smooth jagged boundaries with morphology
  • Snap regions to known geometry or map boundaries
  • Merge temporally stable masks across adjacent frames

Key takeaways