Sampling, Resizing, Filtering, and Morphology

Interpolation methods trade sharpness, smoothness, speed, and aliasing behavior. Nearest neighbor copies the closest sample and preserves discrete label values. Bilinear and bicubic estimate new samples from a neighborhood. Area or antialiased reduction aggregates the source support before producing fewer samples.

The name of the method is not the whole choice. Take one identical set of 1024x1024 FFHQ face images, resize them down to 299x299 with different library implementations, and compare each result against PIL-bicubic. The images are the same images, so FID should come out near 0. PIL-bilinear adapts its prefilter width to the downsampling factor, and scores FID 0.64. The fixed-width prefilters used by PyTorch and TensorFlow score 4.3 — close to naive nearest-neighbour subsampling at 7.4, which does no filtering at all. Parmar and colleagues ran that comparison in 2021. Nothing changed between the runs except which library performed the resize.

The paper names the cause in one sentence: “However, commonly-used implementations use a fixed-width prefilter, resulting in aliasing artifacts.” The four method names in this map are a first-order description. Underneath them sits a second choice: whether the prefilter adapts to the downsampling factor. The library makes that choice. It usually does not record it.

When high-frequency image structure is sampled too coarsely, different patterns can produce the same reduced samples. Fine stripes may become false waves. And a one-pixel defect may vanish.

Low-pass filtering before you downsample limits aliasing. But it also removes detail. The correct trade-off depends on the smallest evidence the task must preserve.

Deep networks inherited the problem rather than solving it. Richard Zhang put it flatly in 2019: “commonly used downsampling methods, such as max-pooling, strided-convolution, and average-pooling, ignore the sampling theorem”. The consequence is that “small input shifts or translations can cause drastic changes in the output”. The remedy is the classical one. Low-pass filtering before downsampling is “the well-known signal processing fix”. But inserting it naively does not work: the preprint says it “degrades performance”, the published version says it “leads to performance degradation”, and both mean the same thing. Placed carefully instead, the same filter raised “accuracy in ImageNet classification, across several commonly-used architectures”. The signal processing had been known for decades. Where to put it had not.

The same sampling failure has a visible face, one that people looked at for years without naming it. StyleGAN2 produced “texture sticking” — detail glued to screen coordinates rather than to the depicted surface. Tero Karras and six co-authors at NVIDIA and Aalto University found where it came from in 2021: “We trace the root cause to careless signal processing that causes aliasing in the generator network.” They reinterpreted every internal signal as continuous and filtered accordingly. The result matches StyleGAN2's FID and is fully equivariant to translation and rotation at subpixel scales. The fix was signal processing, not more capacity.

This map separates common operations by the information they emphasize: smoothing filters reduce local variation to suppress noise or small texture, edge and derivative filters respond to directional intensity change, sharpening filters increase local contrast by amplifying high-frequency components, adaptive or edge-preserving filters use local structure to limit boundary blur, and learned restoration estimates the transformation from data and inherits the training distribution.

What none of these families guarantees is that information removed from the picture has been removed from the model. Standard CNNs can predict a patient's self-reported race from medical images, at AUC 0.91-0.99 on x-rays, 0.87-0.96 on chest CT and 0.81 on mammography. Gichoya and colleagues reported that in the Lancet Digital Health in 2022. The obvious proxies did not account for it: body-mass index reached AUC 0.55, disease distribution 0.61, breast density 0.61. So they filtered the images in the frequency domain. Low-pass filtering only degraded performance around diameter 10, by which point the images were already visibly destroyed. High-pass filtering left performance high up to diameter 100, where the radiologist co-authors could not tell the image was an x-ray. The Results section says it plainly: “A high performance (up to diameter 100) in the absence of discernible anatomical features was maintained with the addition of a high-pass filter (ie, model performance was maintained despite extreme degradation of the image visually).”

A filter is chosen to remove something. What counts as removed is defined against the human eye, and the model is not the human eye. “The image looks unusable now” is not evidence that the evidence is gone.

Binary and grayscale morphology can be useful when the structuring element matches a real geometric assumption. The way to find out whether it matches is to measure it.

The most standard morphological assumption is that only the largest connected component is real. nnU-Net treats it as a hypothesis rather than a cleanup step. Isensee and colleagues built the system, described in Nature Methods in 2020, to configure preprocessing and post-processing automatically across 23 public biomedical segmentation datasets. It keeps non-largest-component suppression only if that raises the mean foreground Dice on cross-validation without lowering the Dice of any individual class. The rule is one line: “Whether to apply this postprocessing is determined by monitoring validation performance after cross-validation.” Standard practice has to earn its place on each dataset. Otherwise it does not get applied.

Use controlled experiments, not which output looks better, to decide whether a transformation belongs in the pipeline. State the intended effect and name the artifact or invariance the operation should address. Define the vulnerable evidence — the small objects, thin boundaries, colors or textures the operation might damage. Compare matched pipelines with data splits, model, seed plan and evaluation protocol held constant. Inspect slice changes on both the target conditions and unaffected reference conditions. Save the exact parameters: kernels, interpolation modes, thresholds, and library behavior.

nnU-Net is what that procedure looks like once it is automated. Across 23 public biomedical segmentation datasets, the largest-connected-component step is not applied because it is standard practice. It survives on a given dataset only when cross-validation shows it raises the mean foreground Dice and lowers no individual class Dice. The same test is available for a resize mode, a blur radius or a threshold. And the fifth step is not bureaucracy: FID 0.64 against 4.3 on identical images is the size of the difference an unrecorded library choice can make.

A global threshold assumes comparable illumination and measurement scale across the image; adaptive thresholds use local neighborhoods, which can help under uneven lighting but may amplify local noise.

Thresholding should be evaluated as a classifier with false positives, false negatives, and uncertain boundary cases. A clean-looking mask is not enough.

There is a public benchmark that does exactly this. The 2018 Data Science Bowl set is 841 two-dimensional microscopy images with 37,333 manually annotated nuclei, from 31 experiments and 22 cell types, released as Broad Bioimage Benchmark Collection accession BBBC038. Caicedo and colleagues, writing in Nature Methods in 2019, scored 3,891 teams against a reference pipeline built in CellProfiler from classical thresholding and watershed. Precision, recall and F1 were computed from true positives, false positives and false negatives at stated coverage thresholds. At an IoU threshold of 0.5 the best model reached F1 0.889 against 0.819 for the classical reference, and 85 challenge algorithms beat that reference outright. The classical pipeline was not worthless. It was measured, and its distance from the field was a number rather than an impression. The paper gives the reason a fixed parameter cannot carry the task: “And even under controlled experimental conditions, no single parameter choice can segment all images correctly, because classical algorithms can fail to adapt to the heterogeneity of biological samples or can be sensitive to technical artifacts”.

Small deterministic tests can catch library or configuration changes before they alter a dataset. Take the FFHQ resize again: PIL-bilinear at FID 0.64, the fixed-width prefilters of PyTorch and TensorFlow at FID 4.3, on identical images resized to 299x299. That gap is what an unpinned resize call costs when nothing in the repository tests for it. And it appears in the evaluation numbers, not in the code review.