Encoder–Decoder and U-Net Architectures

The two arms solve different problems. They meet through explicit interfaces.

One sentence in the 2015 U-Net paper holds the whole shape: the network “consists of a contracting path to capture context and a symmetric expanding path that enables precise localization”. Encoder, bottleneck, decoder and skip fusion are the named parts of that sentence. The dense head turns the recovered grid into the quantity a particular task needs.

A compressed representation can recognize that an object exists and still lose the boundary a mask, a depth map, or a restored signal needs. The decoder translates context back into an output grid. U-Net-style skips help because they keep fine features reachable. They do not remove the need for the bottleneck, whose broader context disambiguates local textures.

Fully convolutional networks made that trade explicit in 2015. They are “fully convolutional”, and they “take input of arbitrary size and produce correspondingly-sized output”. The component that recovers detail is named after the two things it joins: “a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations”. What that bought was “state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012)”. The deep, coarse layer supplied the what. The shallow, fine layer supplied the where. The 20 per cent is what happens when a design stops choosing between them.

Each changes artifact risk, flexibility, and how much work it takes to build — and one of the three carries a defect you can predict with arithmetic before training anything.

That arithmetic was published in Distill in 2016, and the condition is exact: “In particular, deconvolution has uneven overlap when the kernel size (the output window size) is not divisible by the stride (the spacing between points on the top).” A stride-2, size-3 transposed convolution therefore gives some outputs twice the input contributions of others in one dimension. On a grid the unevenness compounds, to a factor of four in two dimensions. A network could in principle learn weights that cancel it. Distill's finding is that in practice networks struggle to avoid it completely. The checkerboard is a property of the geometry you chose, not a mystery of the optimizer.

Rearrangement methods attack the same problem from the other side. The sub-pixel layer arrived in 2016: “In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output.” Features are extracted in low-resolution space. Channels are then rearranged into pixels. That is the exact channel–resolution contract. It bought +0.15 dB on images and +0.39 dB on videos. It ran an order of magnitude faster than the prior CNN methods, and was the first CNN able to super-resolve 1080p video in real time on a single K2 GPU. Distill, assessing it independently, notes that this amounts to choosing a kernel size divisible by the stride, and that “while this approach helps, it is still easy for deconvolution to fall into creating artifacts”. Interpolation followed by convolution sidesteps the divisibility question entirely. It separates resizing from feature learning, at the cost of the follow-up convolution.

The decoder and head must match the physical meaning of each target. Where the target is a patient, the contract may be written by a regulator rather than by the modeller. Limbus Contour was cleared by the FDA on 13 August 2020 for automatic contouring of CT and MR images. The Indications for Use draw the line in a single sentence: “Limbus Contour is not intended to automatically contour tumors or tumor clinical target volumes.” Healthy anatomical structures, yes. Tumour volumes, no.

The product runs a separate U-Net per structure. An independent hospital evaluation in 2022 measured what that delivers. Average Dice across structures was 0.72 — 0.99 for lungs, 0.39 for the penile bulb. Contouring time in head and neck cases fell by 65%. One architecture, one clearance, and a per-structure quality spread wide enough that the output contract has to be written structure by structure rather than model by model.

Start with deterministic tensor checks before blaming the optimizer. Two of these steps are arithmetic rather than experiment. Whether kernel size divides stride can be settled on paper. A synthetic impulse makes the resulting checkerboard visible in one forward pass, instead of after a training run.

U-Net's contracting and expanding paths are widely reusable. Architecture names still do not settle data efficiency, calibration, robustness, or annotation quality. The KiTS19 challenge in 2019 put numbers on the last of those, over 210 public training CT scans and 90 private test scans. The winning entry came from Isensee and Maier-Hein, under the title An Attempt at Beating the 3D U-Net. The organisers recorded what it did: “The winning team achieved a Dice of 0.974 for kidney and 0.851 for tumor, approaching the inter-annotator performance on kidney (0.983) but falling short on tumor (0.923).” One entry, one set of scans, one architecture. It sat at the annotation ceiling for kidney and a long way beneath it for tumour. On kidney the binding constraint was no longer the network.

What won is equally instructive. nnU-Net configures preprocessing, architecture, training and post-processing automatically. Its 2020 paper in Nature Methods states the result: “Without manual intervention, nnU-Net surpasses most existing approaches, including highly specialized solutions on 23 public datasets used in international biomedical segmentation competitions.” The KiTS19 organisers saw the same lesson from the other side, writing that “one might have expected KiTS19 to be won by the team which finds the right architectural garnish with which to augment the nnU-Net baseline, but as we discuss in Section 4, the 'vanilla' nnU-Net demonstrated top performance despite attempts by several teams to enhance it”. Three-dimensional variants can also become prohibitively memory intensive.

Treat depth, skip placement, receptive field, and decoder capacity as hypotheses. Evaluate them against the actual output geometry and acquisition process.