Pixels, Color, and Dynamic Range

A pixel triplet does not carry its own meaning. What the three numbers stand for is fixed by the profile attached to the file. In a browser that is not a convention but a requirement. Take a JPEG tagged ITU-R BT.2020, shown on a Display P3 screen. CSS Color Module Level 4 rules on that exact case: “It must not treat the ITU Rec BT.2020 values as if they were Display P3 values, which would produce incorrect colors.” The image has to be converted into the display's space first.

That is the rule a pipeline breaks when it drops the profile on load. Nothing errors, because the fallback is specified too: “For compatibility, colors specified in HTML, and untagged images must be treated as being in the sRGB color space ([SRGB]) unless otherwise specified”. The loader keeps working. It simply reads wide-gamut numbers as sRGB numbers, and the two spaces are not close — the W3C gamut-volume table puts rec2020 at 2.042 million Lab units against display-p3's 1.233 and sRGB's 0.820. Nothing was lost in transit. The numbers were read in the wrong language.

Four properties have to be tracked together when interpreting image values. Each one is something a real specification writes down. None of them can be read off the array.

Spatial grid: rows and columns sample the projected scene at finite locations, and the spacing can be mandated — the FBI's Electronic Biometric Transmission Specification fixes fingerprint scanning at 500 or 1,000 ppi within ±1%.

Channels: values may represent red, green, blue, alpha, depth, infrared or another measured band. Their count says nothing about their resolution. Sentinel-2's Multi-Spectral Instrument carries 13 bands sampled at three different ground resolutions.

Numeric encoding: integers or floating-point values use a bit depth, scale, offset and valid range, and the container need not match the measurement. ESA acquires at 12 bits and ships 16-bit integers.

Colour interpretation: transfer functions, primaries, white points and profiles determine perceived colour, and CSS Color Module Level 4 makes the profile binding on the renderer.

Drop any one of the four and the array still loads.

An image with 4000 by 3000 pixels contains twelve million samples. It does not contain twelve million independent pieces of scene information. Lens blur, motion, demosaicing, compression and sensor noise all reduce effective detail. A smaller image can preserve the task signal instead, as long as the objects stay large enough after resizing.

Where the stakes are high, the choice is made explicitly and written into a specification. The FBI's Electronic Biometric Transmission Specification states the requirement for a fingerprint scanner in a single sentence: “The scanner's final output resolution in both sensor detector row and column directions shall be in the range: (R–0.01R) to (R+0.01R) and shall be gray-level quantized to eight bits per pixel (256 gray-levels).” R is 500 or 1,000 ppi, so the tolerance is ±1%. Field-level rules narrow Type-4 records further, to 495–505 ppi.

NIST's compression guidance for 1000 ppi friction ridge imagery records the installed base: “The criminal justice community has traditionally exchanged and stored fingerprint imagery data at 500 pixels per inch (ppi) or 19.7 pixels per millimeter (ppmm)”, with WSQ as the compression standard at that resolution. For 1000 ppi imagery it specifies JPEG 2000 at “Bit depth Ssiz = 7 (corresponding to 8-bit Gray Scale)”, with a final ratio of 10:1 for rolled, flat, slap and palm impressions. Latent imagery is compressed losslessly instead, at 1:1. Two separate axes, two separate numbers, both fixed on purpose.

The sampling question does not stop at the file. Shift an image down by one pixel and a trained network can change its answer. Azulay and Weiss measured how often, testing six ImageNet networks — VGG16, ResNet50 and InceptionResNetV2 from Keras, VGG16, ResNet50 and DenseNet121 from PyTorch — on 1,000 images under four one-pixel perturbation protocols. Their 2019 paper reports it plainly: “We show that the chance that a CNN output on a randomly chosen image will change after translating downward by a single pixel can be as high as 30%.” That 30% ceiling belongs to the Keras models; the PyTorch DenseNet121 tops out at 0.19.

Their diagnosis is a sampling one: “The convolutional architecture ignores the classical sampling theorem, so that aliasing effects make the output not invariant”. Richard Zhang named the same culprit the same year — “Commonly used downsampling methods, such as max-pooling, strided-convolution, and average-pooling, ignore the sampling theorem” — and showed that the repair pays. Inserting a low-pass filter before those stages cut ResNet-50's mean flip rate on ImageNet-P from 7.92 to 6.90, and its mean corruption error on ImageNet-C from 60.6 to 58.1, with a Bin-5 filter. ImageNet accuracy improved as well. Neither paper is describing a failure of the model. Both are describing a failure of the grid underneath it.

Channel count alone does not reveal how values should be processed. An RGB photograph is typically H × W × 3 and encodes a display-oriented view of visible light; its characteristic failure is a channel-order mismatch, and its usual transform is colour normalisation. A grayscale image is H × W or H × W × 1. It records intensity under a specific acquisition process — an X-ray, a document scan — with the discarded colour signal as the risk and contrast adjustment as the usual transform. A depth map estimates distance or disparity per pixel, in metres or inverse depth. It carries invalid and missing values, so it is range-masked before use. A multispectral image samples distinct wavelength ranges in several bands, at possibly different resolutions. Its risk is precisely that it looks like display RGB. What it needs is radiometric calibration, not colour normalisation.

Sentinel-2 shows how far the last case is from a three-channel photograph. ESA describes “the optical Multi-Spectral Instrument (MSI) that samples 13 spectral bands: four bands at 10 m, six bands at 20 m and three bands at 60 m spatial resolution”. NASA Earthdata publishes the identical split: “Sentinel-2 MSI samples 13 spectral bands at different spatial resolutions: four bands at 10 meter resolution, six bands at 20 meter resolution, and three bands at 60 meter resolution.” There is no single H and W for the scene. Three of them coexist inside one product.

The numeric axis is separate again, and the container lies about it. ESA states that “The radiometric resolution of the MSI instrument is 12 bit (1), enabling the image to be acquired over a range of 0 to 4095 potential light intensity values”, and the footnote spells out what happens next: “The Sentinel-2 instrument acquires measurements at 12 bits. These measurements are converted to reflectance and stored as 16-bit integers in the S2 product.” The instrument measures 12 bits. The container is 16. A pipeline that reads the dtype and trusts it has invented precision the instrument never measured.

Record these decisions for every image source before training. The source encoding: file format, profile, channel order, bit depth, compression. The physical meaning of each channel, and its units. The invalid ranges: missing values, saturation, clipping, padding, sensor sentinels. The transformations in order, from resize and crop through colour conversion, scaling and normalisation. And a verification step using known images and exact tensor checks, so that a silent pipeline change is detected rather than absorbed.

One published dataset shows what a finished ledger looks like. MIMIC-CXR distributes chest radiographs in DICOM — “A total of 377,110 images are available in the dataset”, from 227,835 imaging studies of 65,379 patients. The derived 8-bit release, MIMIC-CXR-JPG v2.1.0, reduces exactly those images to JPG, and writes down how: “Pixel values were normalized to the range [0, 255] by subtracting the lowest value in the image, dividing by the highest value in the shifted image, truncating values, and converting the result to an unsigned integer.” The remaining steps are equally explicit. The image is inverted according to the DICOM PhotometricInterpretation field, histogram-equalised with OpenCV, then encoded as JPG at quality factor 95.

Every step in that chain is lossy. None of them is a secret. That is the difference a ledger makes. Two groups training on MIMIC-CXR-JPG are training on the same numbers, and anyone comparing a result on the JPG release against a result on the DICOM originals knows exactly which four operations separate them.

Subtracting a mean and dividing by a scale can place channels in a numerically convenient range. It does not correct mislabeled colour profiles, saturation or lost spatial detail. And the constants are not universal. Two mainstream libraries expect incompatible encodings of the same H × W × 3 array.

torchvision documents its pretrained ResNet-50 weights this way: “Finally the values are first rescaled to [0.0, 1.0] and then normalized using mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225]”. It lists acc@1 76.13 for IMAGENET1K_V1 and 80.858 for the V2 recipe, on the identical 25,557,032-parameter architecture. Keras' default preprocessing, the mode named “caffe”, takes “values in the range [0, 255]” and does something else entirely: “The images are converted from RGB to BGR, then each color channel is zero-centered with respect to the ImageNet dataset, without scaling.” The constants it subtracts are [103.939, 116.779, 123.68]. Its other two modes differ again — “torch” divides by 255 and applies the mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225], and “tf” divides by 127.5 and subtracts 1.

Feed a [0, 1] tensor to the caffe path and nothing raises. The network receives an image roughly 255 times too dark, with its channels reversed, and reports a confident wrong class. So statistics must be computed on the appropriate training boundary, and that boundary belongs to the weights as much as to the dataset. Reusing values from an unrelated domain may be acceptable for a pretrained backbone, but the decision should be tested rather than assumed.