Data Engineering for Text, Images, Audio, and Sensors

An image does not tell you which way up it is in its pixels. It tells you in a tag beside them. Exif — the exchangeable image file format for digital still cameras, managed jointly by CIPA and JEITA since 2009, in version 3 since 2023 — calls that tag Orientation. Tag 274 (112.H), type SHORT, count 1, default 1, eight defined values covering every rotation and mirror. Value 1 is the ordinary one: “The 0th row is at the visual top of the image, and the 0th column is the visual left-hand side.”

That is the whole contract for orientation: one SHORT, eight legal values, sitting outside the pixel array. Two files can carry identical pixels and be two different pictures, because one of them carries tag 274 and the other lost it in transit. The raw object, its metadata and its derived representation have to travel together, and a single tag is enough to measure how far apart they have drifted.

Text, images and video, audio and speech, sensors and telemetry, and multimodal records each have their own version of that tag — the small piece of context, held beside the bytes rather than inside them, without which the bytes cannot be read correctly. Encoding, language, layout and document version do it for text. Capture device, compression, crop and annotation geometry for images and video. Sample rate, channel, codec and noise conditions for audio. Clock synchronization, calibration, units and device placement for sensors. Alignment and temporal tolerance for multimodal records.

Unstructured pipelines often preserve files. They lose the context required to interpret them, and the loss is quiet, because decoding still succeeds and nothing raises an error.

A resized image, an OCR transcript, an audio embedding or a windowed sensor tensor is not the same artifact as its source. Each derived object depends on code, parameters, model versions, and sometimes on a step that does not run the same way twice. Store a stable source identity, content hash, provenance, transformation version, and the relationship between raw and derived forms. That is what lets you reprocess when a decoder, tokenizer or representation changes.

For sensitive data, “preserve raw” can conflict with minimization and retention duties. The policy then has to say which evidence is kept, which is protected, and which is irreversibly transformed. Medical imaging shows what stating it looks like in machine-readable form. The Cancer Imaging Archive applies the DICOM Basic Application Level Confidentiality Profile with five named options: Clean Pixel Data, Clean Descriptors, Retain Longitudinal With Modified Dates, Retain Patient Characteristics, Retain Safe Private. It then records every rule it applied inside the derived object itself, in Method Code Sequence (0012,0063). The derived image travels with a declaration of what was removed from it. The alternative is a promise held in a separate document that the next team will never read.

The correct unit depends on the decision, not on what the file format happens to make easy. A whole document or recording preserves broad context but localizes badly. A chunk, frame or window supports detection and retrieval, but it creates correlated examples, and its boundaries change the labels. An event or region aligned to a detected span, box, speaker or sensor event gives precise supervision, but it depends on upstream detection and on geometry or time metadata being present.

The cost of getting the grouping wrong is not hypothetical. It has been measured on a benchmark everybody trusts. In 2020 Barz and Denzler checked CIFAR for near-duplicates, and their abstract says what they found: “We find that 3.3% and 10% of the images from the CIFAR-10 and CIFAR-100 test sets, respectively, have duplicates in the training set.” On that duplicated subset their ResNet-110 models made an error rate of 0% on CIFAR-10 and 2.9% on CIFAR-100. On the full test sets the same models made 5.3% and 26.1%. Nothing was memorized in a mysterious way: the model had seen those examples. Re-running on the duplicate-free ciFAIR test sets raised average error by 0.41 percentage points on CIFAR-10 and 2.73 points on CIFAR-100, a relative difference the paper puts as high as 12%.

That is what “unrealistically easy evaluation” costs when the duplicates are accidental copies of whole images. Chunks of one document, frames of one video, windows from one device, or utterances from one speaker are duplicates by construction. They are also far more numerous.

Adapt the checklist to the modality rather than reducing it to generic row validation. Identify the source object first: content hash, producer, acquisition device, consent state, original timestamps. Everything else attaches to that stable identity.

Step 2, validate decoding, is where the small print lives — codec, shape, duration, channel, orientation, language, corrupted-object handling. “Orientation” here means one specific thing that can be checked mechanically. Exif defines it as tag 274 (112.H), type SHORT, count 1, default 1, with eight defined values. A validator can assert that the tag is present and legal. A pipeline that silently assumes the default is making a claim about the data it never tested.

Step 3 preserves the transformations: crop, normalization, segmentation, resampling, OCR and embedding logic, each versioned. As Parmar and colleagues measured, the resize function alone moved FID from 4.82 ± 0.09 to 6.21 ± 0.23 on one dataset.

Step 4 groups related samples, so that frames, chunks, speakers, patients or devices do not leak across splits. Omitting it is what Barz and Denzler quantified: 3.3% of CIFAR-10 and 10% of CIFAR-100 test images had duplicates in the training set, and on those the error rate was 0% and 2.9%. Step 5 inspects real artifacts across devices, conditions, languages and failure modes, because no aggregate statistic will show you a rotated page or a mis-resampled channel. Step 6 publishes the limitations: missing modalities, coverage, rights, and known acquisition shifts.

Embedding drift can reveal a change. It rarely explains whether the cause is device, codec, language mix, lighting, noise, content, or model behavior. How much of a representation is capture rather than content has been measured directly.

A pneumonia model trained on 158,323 chest radiographs from three institutions scored AUC 0.931 (95% CI 0.927–0.936) on the Mount Sinai and NIH data it was pooled from, and 0.815 (0.745–0.885) at Indiana University. Zech and colleagues published that in PLOS Medicine on 6 November 2018. Then they trained a network to do nothing but name the hospital a radiograph came from. It identified 22,050 of 22,062 NIH radiographs (99.95%) and 8,386 of 8,388 Mount Sinai radiographs (99.98%). Their author summary draws the consequence: “The performance of CNNs in diagnosing diseases on X-rays may reflect not only their ability to identify disease-specific imaging findings on X-rays but also their ability to exploit confounding information.”

The effect is not particular to radiographs. Biondetti and colleagues classified the scanner manufacturer of 782 head CTs at 98.11% ± 0.65 and 98.86% ± 1.25 test accuracy, with AUROC 0.99 ± 0.01. A network that reads the site off the pixels at 99.95% and the manufacturer at 98.11% is telling you that site and scanner are inside every embedding computed from those pixels. Change either one and the embedding distribution moves, before anything a monitor is watching does.

So preserve metadata about how the item was captured, and raw quality indicators. Monitor missing modalities, decode failures, duration and resolution distributions, device and firmware mix, language coverage, and alignment error. These signals support faster root-cause analysis. When privacy prevents retaining raw examples, maintain approved diagnostic samples or irreversible summaries that still expose system changes.