ML data engineering
Dataset Versioning, Lineage, Provenance, and Reproducibility
Track dataset, job, run, recipe, snapshot, dependency, and retention identities across releases.
By the end you can
- Distinguish path, snapshot, content, recipe, and release identities
- Model lineage across datasets, jobs, runs, and external dependencies
- Create manifests that support impact analysis and clean rebuilds
- Explain when lineage can diagnose history but cannot reproduce expired evidence
Key idea
Lineage without retained evidence can explain but not reproduce
A lineage graph may show that a model used table snapshot 421. The rebuild fails if that snapshot was expired. Metadata and data retention need coordinated policies. Not every dataset can be kept forever. Sensitive records may require deletion, and storage cost limits history. Preserve approved summaries, hashes, code, and audit evidence when full data retention is impossible. Document the reproducibility level honestly: exact rebuild, statistically equivalent rebuild, explainable but not rebuildable, or intentionally non-reproducible after deletion.
That last level is not hypothetical. It is also not always the team's own choice. Sometimes a regulator makes it. The US Federal Trade Commission announced a settlement with Everalbum, Inc. on 11 January 2021, and the published analysis of the proposed consent order sets out what had to go: “Part III of the proposed order requires Respondent to delete (A) photos and videos of Ever app Users who requested deactivation of their accounts, (B) face recognition data that it created without obtaining Users' affirmative express consent, and (C) models and algorithms it developed in whole or in part using images from Users' photos.” The order will remain in effect for 20 years.
Read clause (C) as a reproducibility statement. The deletion reached past the raw images. It reached past the derived face-recognition data. It took the models built on them too. A team under that order can still hold everything lineage is good at holding: which datasets fed which jobs, which runs produced which outputs, who approved them, what validation said. What it can never do again is rebuild the release. The evidence and the artifact derived from it were both compelled out of existence. The honest promise to write into the manifest, before anyone asks for a rebuild, is the fourth level.
Reproducibility has retention and governance costs; the promised level should be explicit.
Case
Delta Lake stops time travelling after thirty days, Iceberg after five
A snapshot ID recorded in lineage can easily outlive the files it points to. Delta Lake puts numbers on that window. Its logRetentionDuration property defaults to interval 30 days, and deletedFileRetentionDuration defaults to interval 7 days. The documentation states the consequence plainly: “By default you can time travel to a Delta table up to 30 days old”.
Thirty days is not the industry number. It is one vendor's number, and the next one is much tighter. Apache Iceberg's own configuration reference gives history.expire.max-snapshot-age-ms a default of “432000000 (5 days)”, described as the “Default max age of snapshots to keep on the table and all of its branches while expiring snapshots”, and gives history.expire.min-snapshots-to-keep a default of “1”. A managed service then applies those defaults on the team's behalf. The AWS Glue console instructions for the Iceberg snapshot retention optimizer say so: “In the absence of this configuration, AWS Glue retains one snapshot for five days, and deletes files associated with the expired snapshots.” The API reference repeats both numbers field by field — snapshotRetentionPeriodInDays, “The default value is 5”, and numberOfSnapshotsToRetain, “The default value is 1”.
So the same recorded identity, a snapshot ID in a lineage graph, becomes unresolvable roughly six times faster on Iceberg than on Delta Lake. Neither figure was chosen by the person who wrote the lineage record. Note also what the Glue sentence adds to expiry. It does not merely forget the snapshot. It deletes the files behind it. Snapshot-addressed identity is a pointer whose lifetime is set by the table format's defaults, and a manifest that cites the ID without citing the lifetime is promising a rebuild it does not control.
A dataset path is not a dataset identity
Paths can be overwritten. Queries can read mutable sources. Transformation environments can change. Reproducibility requires a stable identity for the evidence a run used. A dataset version may be content-addressed, snapshot-addressed, or recipe-addressed. Whatever the scheme, it should resolve to exact source versions, code, parameters, schemas, and split assignments.
Lineage connects datasets, jobs, and runs into a graph. It helps answer which upstream change affected a model, which consumers depend on a table, and which outputs must be rebuilt once a source is corrected.
Provenance is broader than lineage edges. It includes ownership, execution environment, validation results, label policy, access conditions, and known limitations.
A model is reproducible only when its dataset identity resolves to stable evidence and an executable, versioned protocol.
Visual
Datasets, jobs, and runs form the core graph
Lineage systems record both how a pipeline was designed and what each run actually did.
The OpenLineage specification names the same three entities and defines each one. A Job is “a process that consumes or produces Datasets”. A Run “is an instance of a Job that represents one of its occurrences in time”. A Dataset is “an abstract representation of data”. Every run carries a runId generated as a UUID.
Read the three definitions as a division of labour. The Job carries intent, and it survives every deployment. The Run carries one occurrence in time, and it is the only place where parameters, environment and status are pinned to something that actually happened. The Dataset is the abstraction both attach to. A client-generated UUID per run is what lets a downstream artifact name the specific execution that produced it, rather than the pipeline that might have.
Dataset
A named input or output with schema, version, location, and semantic metadata.
Job
A logical transformation, validation, labeling, or publication process.
Run
One execution of a job with timestamps, parameters, code, environment, and status.
Facet or metadata
Additional details such as column lineage, quality results, ownership, and documentation.
Consumer link
The model, report, service, or downstream dataset that uses the output.
Run-level lineage shows what actually happened, while design lineage shows what is intended to happen.
Comparison
Content, snapshot, and recipe identities
Different systems can combine these strategies to balance cost and reproducibility. Each one is a different bet about what will still exist when the rebuild is attempted. Content hashing bets on the digest. Snapshot addressing bets on the catalog's retention window. Recipe addressing bets on the inputs and the runtime all surviving together.
Content hashing offers the strongest evidence that two datasets are byte-identical, and the strength of that evidence is measurable. On 23 February 2017 CWI Amsterdam and Google announced the first practical SHA-1 collision: two PDF files with distinct visible contents and the same SHA-1 fingerprint. Their press release gave the price of producing it. Elie Bursztein (Google) put it this way: “It required over 9,223,372,036,854,775,808 SHA1 computations that took 6,500 years of CPU computation and 100 years of GPU computations.”
That number cuts both ways, and the second edge is the one a data team should record. A hash is not a permanent guarantee. It is a dated one. NIST had already deprecated SHA-1 in 2011 and disallowed it for digital signatures at the end of 2013, and on 15 December 2022 it announced that it will transition away from SHA-1 for all applications by 31 December 2030. A content-addressed dataset identity inherits the expiry of whichever digest it was built on. That is why the algorithm belongs in the manifest beside the hash value. A bare hex string does not say which guarantee it is making, or until when.
Snapshot identity is efficient for large analytical tables. It supports time travel and rollback while retained, with the retention windows of the previous section as its real boundary. Recipe identity records exact inputs and deterministic build logic, and it supports feature variants and flexible rebuilding. It fails whenever an input, a dependency or an environment has moved.
Content hash
Identity derives from the exact bytes or canonical records.
- Strong evidence that content is identical
- Can be expensive for very large datasets
- Needs canonical ordering or manifest hashing
- Useful for immutable artifacts and files
Table snapshot ID
Identity points to a transactional table state.
- Efficient for large analytical tables
- Supports time travel and rollback while retained
- Depends on catalog and snapshot retention
- Useful for lakehouse-style storage
Recipe plus source versions
Identity records exact inputs and deterministic build logic.
- Supports flexible rebuilding and feature variants
- Fails if inputs or environment disappear
- Needs versioned parameters and dependencies
- Useful for pipeline-generated datasets
Steps
Create a dataset release manifest
The manifest should let another engineer identify, inspect, rebuild, and assess the dataset without oral history. Two published instruments show what that looks like when it stops being a house convention. One names a minimum field list by regulation. The other ships as machine-readable metadata at scale.
Software got its field list by law. Executive Order 14028 required a minimum, and a Department of Commerce / NTIA report of 12 July 2021 fixed it as seven baseline data fields — Supplier, Component Name, Version of the Component, Other Unique Identifiers, Dependency Relationship, Author of SBOM Data, and Timestamp — plus automation support and defined practices and processes. Read that list against a dataset release. Supplier is the owning team. Component and version are the dataset identity. Unique identifiers are the hashes and snapshot IDs. Dependency relationship is the lineage edge. Author is who recorded the manifest, and timestamp is when. The report names SPDX, CycloneDX and SWID tags as the formats used to generate and consume SBOMs, because a bill of materials that only a human can read does not support impact analysis.
Datasets already have their own version of this. Croissant is an MLCommons metadata format built as an extension of schema.org/Dataset, released as a 1.0 specification on 6 March 2024. It records, in the words of its peer-reviewed paper, “datasets' attributes, the resources they contain, and their structure and semantics”. It is in production rather than in proposal. That paper, presented at NeurIPS in 2024, reports that “Croissant has been integrated into three major dataset repositories: Hugging Face Datasets, Kaggle Datasets, and OpenML, which together describe over 400,000 datasets in the Croissant format”. One of those repositories says the same thing from its own side: “The dataset viewer automatically generates the metadata in Croissant format (JSON-LD) for every dataset on the Hugging Face Hub”, for every dataset that can be converted to Parquet. The manifest is generated rather than remembered.
The five steps below are the local work that a format like Croissant then carries. Identify content and sources with snapshot IDs, manifests, hashes, retention and authoritative locations. Record the build with code commit, job version, parameters, environment, dependencies and run ID. Record semantics including grain, keys, time rules, label policy, schema, contracts and split version. Attach evidence in the form of validation results, dataset diff, coverage, known issues and approvals. Then link the models, evaluations, reports and services that consume the release.
1. Identify content and sources
Record snapshot IDs, manifests, hashes, retention, and authoritative locations.
2. Record the build
Capture code commit, job version, parameters, environment, dependencies, and run ID.
3. Record semantics
Include grain, keys, time rules, label policy, schema, contracts, and split version.
4. Attach evidence
Store validation results, dataset diff, coverage, known issues, and approvals.
5. Link consumers
Associate models, evaluations, reports, and services that depend on the release.
A manifest should answer “what, how, when, why, and who uses it” for one dataset version.
Analogy
Lineage is the bill of materials for a model
The bill of materials is not a borrowed metaphor. Software has one by law, with a definition, a citation and a date. Executive Order 14028 of 12 May 2021 defines the Software Bill of Materials, and it requires vendors to hand one over — “providing a purchaser a Software Bill of Materials (SBOM) for each product directly or by publishing it on a public website”. The Department of Commerce / NTIA report that followed on 12 July 2021 states the object in a single sentence: “An SBOM is a formal record containing the details and supply chain relationships of various components used in building software.”
Now read that sentence with datasets in the place of components. Source datasets are the components. Jobs are the assembly steps. Runs are the specific executions. Validation results are the inspections. A model version is one finished product tied to its data bill of materials. Two words in the definition are load-bearing: “formal record” and “relationships”. A formal record is one a purchaser or an auditor can demand, not one a team keeps if it has time. Relationships are what make the artifact a graph.
That second word is the reason the analogy has to stretch. Data products are derived repeatedly and shared by many consumers. Lineage must therefore support a graph, not only one linear chain.
Lineage makes upstream cause and downstream impact navigable across a shared graph of data products and runs.
Use lineage before and after change
Before changing a field or pipeline, traverse downstream to identify the models, datasets, reports, and services at risk. During an incident, traverse upstream to locate the first changed source or run. Column-level lineage can narrow impact. It is difficult for dynamic code, and it does not capture semantic dependencies automatically. A model may depend on category meaning even when the physical column remains unchanged.
There is a measured case of what it costs when neither traversal happens, because nobody can see that anything broke. The UK Department of Health and Social Care disclosed it in the weekly NHS Test and Trace statistics for 1 October to 7 October 2020: “A technical issue was identified overnight on Friday 2 October in the automated process that transfers pillar 2 COVID-19 positive lab results into reporting dashboards and contact tracing. After rapid investigation, 15,841 cases were identified between 25 September and 2 October that were not included in the reported daily COVID-19 cases”. The same release adds that “All affected cases were immediately transferred to contact tracing on 3 October”. The upstream job kept running. The downstream dashboard kept publishing a plausible number. The defect was silent in both directions from 25 September until 3 October.
The gap then became a natural experiment, and it has a price attached. Fetzer and Graeber studied it in PNAS in 2021. Their abstract states: “Because of a coding error involving spreadsheet data used by the health authorities, a total of 15,841 COVID-19 cases (around 20% of all cases) failed to have timely contact tracing.” Their conservative causal estimates were that, relative to the cases the system initially missed, cases subject to proper contact tracing were associated with 63% fewer subsequent infections and 66% fewer subsequent COVID-19-related deaths across the six weeks following the glitch. One data-load job, one unnoticed break, 15,841 records. The consumer was a public-health intervention, and those two percentages are what the break cost it.
Combine lineage with contracts, ownership, usage telemetry, and dataset diffs. The graph shows what is connected. These other artifacts explain why it matters.
Review undeclared consumers by comparing query logs and feature usage with registered dependencies. Hidden consumers are a major source of unsafe retirement and surprise breakage.
Lineage is most valuable when it supports proactive impact analysis, rapid diagnosis, and accountable change management.
Key takeaways
- Dataset identity should resolve to immutable content, a retained table snapshot, or an exact recipe with stable source versions — and each option carries a published expiry, from SHA-1, which NIST will transition away from for all applications by 31 December 2030, to an Iceberg snapshot, retained five days by default.
- Lineage connects datasets, jobs, runs, metadata, and consumers so teams can trace cause and impact. OpenLineage makes the distinction concrete by defining a Run as “an instance of a Job that represents one of its occurrences in time”, identified by a runId generated as a UUID.
- Provenance also includes semantics, owners, environments, label policies, validation evidence, and known limitations — the things that make a graph interpretable rather than merely connected.
- Hidden dependencies such as current reference tables, external APIs, mutable functions, and manual files often break reproduction. Unchanged Willoughby–Hoye scripts returned 173.2 on macOS Mavericks and Windows 10 but 172.4 on Ubuntu 16, because the operating system changed the file ordering.
- Lineage metadata and data retention policies must be coordinated to support the promised reproducibility level. The FTC's Everalbum order of 11 January 2021 compelled deletion of the photos, the face-recognition data and the models developed from them, and it runs for 20 years.
- A release manifest should capture what the dataset contains, how it was built, why it exists, what evidence supports it, and who depends on it. The NTIA fixed seven minimum SBOM fields on 12 July 2021, and Croissant already describes over 400,000 datasets across Hugging Face Datasets, Kaggle Datasets and OpenML.