Cameras, Optics, Sensors, and Compression

A vision system inherits every transformation applied before the first learned layer.

The last stage of that chain is the one most often treated as neutral, and it has been written down since 1992. The baseline JPEG standard is CCITT Recommendation T.81, and its foreword gives the date: “The text of CCITT Recommendation T.81 was approved on 18th September 1992. The identical text is also published as ISO/IEC International Standard 10918-1.” Clause 4.3 then describes exactly what the encoder does to the pixels a model will later read: “In the encoding process the input component's samples are grouped into 8 × 8 blocks, and each block is transformed by the forward DCT (FDCT) into a set of 64 values referred to as DCT coefficients” — one DC coefficient and 63 AC coefficients per block, each of them then quantised.

Step 5 is therefore not a wrapper around the image. It is a lossy re-description of it on a fixed 8 × 8 grid. The block boundaries a model can learn to key on are the grid the standard specifies, not the symptom of a badly saved file.

A longer exposure gathers more light but increases motion blur. Higher gain brightens weak signals while amplifying noise and reducing usable dynamic range.

A wider aperture admits more light but changes depth of field and optical behavior; the best setting depends on object speed, lighting, required detail, and the tolerated failure mode.

At low light the trade stops being a preference and becomes the whole problem. Short exposures are noisy. Long ones blur, and are often impractical anyway. In 2018 Chen and three co-authors stopped restating that dilemma and built the measurement instead, contributing “a dataset of raw short-exposure low-light images, with corresponding long-exposure reference images”. Every noisy frame has its own long-exposure ground truth to be scored against. What they trained on it does not sit downstream of the camera pipeline. It stands in place of most of it: “the network operates directly on raw sensor data and replaces much of the traditional image processing pipeline, which tends to perform poorly on such data”. Capture settings were not a nuisance parameter in that work. They were the experiment.

Similar-looking image degradation can require different fixes. Three of the four columns below are not adjectives but separately graded corruptions in a public benchmark. The fourth has been timed in milliseconds.

ImageNet-C is the benchmark that turns those adjectives into a graded test set. Hendrycks and Dietterich published it at ICLR 2019. It applies 15 corruption types to the ImageNet validation images at graded intensity: “Each corruption type has five levels of severity since corruptions can manifest themselves at varying intensities.” The arithmetic is the point — “Overall, the ImageNet-C dataset consists of 75 corruptions, all applied to ImageNet validation images for testing a pre-existing network.” Defocus blur and motion blur travel in blur.tar alongside glass blur and zoom blur. JPEG compression travels in digital.tar alongside contrast, elastic transform and pixelate. The release runs to roughly 66 GB across five archives — noise 22.6 GB, extra 15.8 GB, weather 12.8 GB, digital 7.8 GB, blur 7.1 GB. It also ships the AlexNet top-1 error rates used to normalise scores: 82.0 percent under defocus blur, 88.6 percent under Gaussian noise. The “model risk” line in the defocus column has a published number behind it.

Rolling shutter has been measured on the hardware. Readout on the DJI Phantom 2 Vision+ FC200 was estimated at a median 74 ms, and DJI confirmed 73 ms. The DJI Inspire 1 FC300X came out at 30 ms, confirmed as 33 ms. Consumer cameras in general run 30-40 ms. Vautherin and six co-authors published those figures in 2016, and also put them where a model sees them: “For a DJI Phantom 2 flying at 8 m/s and 70 m above ground, the rolling shutter causes a displacement of about 10 pixels.” Skew is not a cosmetic complaint. It is a geometric error with a size.

Compression is the column most often blamed on settings alone. The standard itself declines to set them: “No default values for quantization tables are specified in this Specification; applications may specify values which customize picture quality for their particular image characteristics, display devices, and viewing conditions.” That is clause 4.3 of T.81, the JPEG standard describing itself. Two files stamped with the same nominal quality by two different encoders are therefore not the same evidence. Nothing in the pixels announces which encoder wrote them.

These cases illustrate why data collection and camera engineering belong inside the ML system boundary. Two of them were flown, measured and published — once, and then again by a different group.

Qualify the camera system before collecting a large training set. A regulator has already written this protocol once, in public, with numbers in it.

The law came first and set a clock running. Public Law 110-189, the Cameron Gulbransen Kids Transportation Safety Act of 2007, was enacted on 28 February 2008: “Not later than 12 months after the date of the enactment of this Act, the Secretary shall initiate a rulemaking to revise Federal Motor Vehicle Safety Standard 111 (FMVSS 111) to expand the required field of view to enable the driver of a motor vehicle to detect areas behind the motor vehicle to reduce death and injury resulting from backing incidents”. Final standards were due within 36 months, and full compliance within 48 months of the final rule. What came out of it is 49 CFR 571.111. Every vehicle at or under 4,536 kg GVWR built on or after 1 May 2018 must display a rearview image — passenger cars under S5.5.1 through S5.5.7, and multipurpose passenger vehicles, low-speed vehicles, trucks, buses and school buses under the identically worded S6.2.1 through S6.2.7.

Read against the five steps above, the standard is this same protocol with the blanks filled in. The operating envelope is stated: ambient illumination between 7,000 and 10,000 lux for the field-of-view and image-size test, and 15 to 25 °C inside the vehicle for the response-time test. The controlled targets are physical objects at fixed places: seven cylinders, 0.8 m high and 0.3 m in external diameter, placed 0.3 m, 3.05 m and 6.1 m behind a plane tangent to the rear bumper, and 1.52 m either side of the centreline. The acceptance threshold is a quantity a device can fail: test objects A, B and C must average not less than 5 minutes of arc in the displayed image, and none may fall below 3. Latency is a number rather than an impression — “The rearview image meeting the requirements of S6.2.1 and S6.2.2, when tested in accordance with S14.2, shall be displayed within 2.0 seconds of the start of a backing event.” And S6.2.7 sends the qualified system back through corrosion, humidity and temperature exposure to be re-tested. That is steps 3 and 4 of the protocol, written as law.

Exposure time, gain, focus distance, camera identifier, firmware version, and codec settings can reveal why performance changed; they may also create privacy, leakage, or serving dependencies.

Almost none of this has to be invented as a schema. Exif already numbers the fields. In Version 2.32, published in May 2019, ExposureTime is tag 33434 (0x829A), FNumber 33437 (0x829D), PhotographicSensitivity 34855 (0x8827), SubjectDistance 37382 (0x9206), FocalLength 37386 (0x920A), Model 272 (0x0110), Software 305 (0x0131), BodySerialNumber 42033 (0xA431) and LensSerialNumber 42037 (0xA435). A capture log that stores exposure in a column of its own devising is re-deriving a standard that already ships in the file.

The privacy caution becomes concrete in the same block. The Library of Congress summarises what the format carries in one sentence: “Exif metadata tags include descriptive metadata, copyright details, camera settings, technical image data, date and time information, geographic data, and thumbnail storage.” Geographic data travels beside exposure time, and GPS support entered Exif at Version 2.0. So the record that explains a drift can also disclose where each training image was taken, and the thumbnail can survive a crop.

Firmware is the awkward case, because the field this section asks you to log was standardised late. A dedicated CameraFirmware tag appears only in Exif Version 3.0, published in May 2023. Before that, the version string a fleet writes into Software 305 is whatever the vendor chose to put there. It has to be captured and versioned by the collection system rather than assumed.

Store metadata deliberately, check how far it can be relied on, and decide whether the model, monitoring system, or neither should use it.