Table Snapshots, Compaction, and Retention

Modern table formats expose snapshots, transaction logs, schema metadata and file manifests. That makes concurrent reads and updates far safer than treating a directory of files as one mutable table. It does not make the past permanent. The vendors publish the clock themselves.

Delta Lake sets VACUUM's default retention threshold for data files at 7 days. Transaction log files are deleted automatically under a default retention period of 30 days, configurable through delta.logRetentionDuration. The project warns against shortening the first of those windows: “It is recommended that you set a retention interval to be at least 7 days, because old snapshots and uncommitted files can still be in use by concurrent readers or writers to the table.”

Microsoft's Azure Databricks documentation states the same 7-day data-file default. It adds that in Databricks Runtime 18.0 and above, time travel queries are blocked when they request a version older than delta.deletedFileRetentionDuration, which defaults to 7 days. Then it says the rest in one sentence: “Don't use table history as a long-term backup solution for data archival.”

Snapshot history, deleted files, change feeds and source data all have retention boundaries. Those boundaries are measured in days. A training release therefore has to record two things: its logical identity, and the immutable versions needed to rebuild its rows. The storage defaults were never designed to cover that horizon.

Reproducibility improves when each identity is recorded separately.

A table name is a stable logical address. Its current contents may change with every overwrite and merge. Convenient for discovery, weak as historical identity.

A snapshot or version is a committed state under a transaction log or manifest. It is good for consistent reads, audit and rollback. But it depends on retention, and it may reference files that expiry is entitled to remove — on the 7-day Delta default or the 5-day Iceberg one.

A dataset release is a governed ML artifact: cohort, labels, transformations, splits and provenance. It names all input snapshots, carries the build code and configuration, and documents exclusions and limitations.

Only the third survives the maintenance cycle on its own terms. Only the third is obliged to say what evidence it needs, and for how long.

Compaction and cleanup solve operational problems. Writes create files and a committed table state. Compaction rewrites small files into larger ones for efficient reads. Clustering, sorting or partition changes alter physical access patterns.

Then history expires, and the defaults are published, not mysterious. Apache Iceberg's table properties set history.expire.max-snapshot-age-ms to 432000000, which is 5 days, with history.expire.min-snapshots-to-keep at 1 and gc.enabled true. The expire_snapshots procedure defaults older_than to 5 days ago and retain_last to 1. AWS Glue's table optimizer documents the same effective default from the other side: “In the absence of this configuration, AWS Glue retains one snapshot for five days, and deletes files associated with the expired snapshots.”

The procedure is careful about what it touches: “This procedure will remove old snapshots and data files which are uniquely required by those old snapshots. This means the expire_snapshots procedure will never remove files which are still required by a non-expired snapshot.”

That care is precisely why compaction and expiry are dangerous together. Once compaction has rewritten the rows into new files, the old files are uniquely required by the old snapshots. And the old snapshots are five days from being unprotected.

The final step, archiving evidence before cleanup, has a documented cost when it is skipped. In a 2023 deployment of a Google Cloud VMware Engine Private Cloud for the Australian superannuation fund UniSuper, Google operators left one input parameter blank in an internal tool. The system assigned an unknown default fixed one-year term. At the end of that period the Private Cloud was deleted, with no customer notification.

What made restoration possible was evidence held outside the thing that expired. Google's incident review, published on 24 May 2024, says so: “Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and, along with third party backup software, were instrumental in aiding the rapid restoration.” UniSuper's CEO Peter Chun and Google Cloud CEO Thomas Kurian issued a joint statement on 8 May 2024; UniSuper also had backups in place with an additional service provider.

An unattended retention default destroyed the primary copy. Only an independently held archive answered for it.

The release has to survive routine optimization and retention work.

Pin the inputs: record table versions, object manifests or content hashes for every source. Preserve the recipe: version the query logic, parameters, code, environment and referenced lookup data. Materialize critical evidence as immutable outputs whenever upstream retention is shorter than audit needs — against a 7-day data-file default and a 5-day snapshot default, that is most of the time. Test a clean rebuild outside the original workspace and compare row-level fingerprints.

Then align retention, and note that the window may not be yours to choose. 17 CFR 240.17a-4 requires broker-dealers to preserve certain records for not less than 6 years, the first two years in an easily accessible place, and others for not less than three years.

The format was prescribed too. Until 2022 electronic records had to be kept exclusively in a non-rewriteable, non-erasable form. The SEC's amendments, adopted on 12 October 2022, opened a second route: “The SEC’s broker-dealer electronic recordkeeping rule currently requires firms to preserve electronic records exclusively in a non-rewriteable, non-erasable format, known as the write once, read many format. The amendments add an audit-trail alternative under which electronic records can be preserved in a manner that permits the recreation of an original record if it is altered, over-written, or erased.”

That alternative requires a complete time-stamped audit trail of all modifications and deletions, the date and time of each action, and the identity of the person responsible. That is a fair description of what a transaction log is. It is a precise description of what expiring one destroys.

Keeping every historical file forever is expensive, and it can conflict with privacy or deletion duties. Removing history too aggressively destroys rollback, audit and reproducibility. The second failure has a price, and the price has been assessed.

On 27 September 2022 the SEC charged 15 broker-dealers and one affiliated investment adviser over off-channel electronic communications from January 2018 through September 2021. Combined penalties came to more than $1.1 billion: eight firms at $125 million each, two at $50 million, Cantor Fitzgerald at $10 million. The CFTC settled the same conduct on the same day with affiliates of 11 financial institutions, for over $710 million, for recordkeeping and supervision failures.

The SEC's Director of Enforcement, Gurbir S. Grewal, put the reasoning plainly: “Today’s actions – both in terms of the firms involved and the size of the penalties ordered – underscore the importance of recordkeeping requirements: they’re sacrosanct. If there are allegations of wrongdoing or misconduct, we must be able to examine a firm’s books and records to determine what happened,”

Nothing there was a storage bug. The records simply were not kept.

Classify dataset releases by what goes wrong when one of them is bad. A disposable experiment, a customer-facing model and a regulated decision system need different evidence windows. Document what can be reconstructed after each retention boundary. If exact rebuild is impossible, preserve enough evidence to explain behavior and quantify the remaining uncertainty.

A team tries to reproduce a model trained eight months earlier. The table name still exists. Several things behind it have changed.

One of those changes deserves attention, because it looks like the format's business and is not. Iceberg's schema evolution is deliberately safe for the changes it supports: “Iceberg uses unique IDs to track each column in a table. When you add a column, it is assigned a new ID so existing data is never used by mistake. Formats that track columns by name can inadvertently un-delete a column if a name is reused, which violates #1.” Adds, drops, renames and reorders are independent and free of side-effects. Amazon Athena's documentation confirms the mechanism costs nothing physically: “Iceberg schema updates are metadata-only changes. No data files are changed when you perform a schema update.”

But the supported type promotions are a short, widening list: integer to big integer, float to double, and increasing the precision of a decimal. A nullable integer turned into a string code is not on it. The change this team is looking at was never a sanctioned evolution. It was a rewrite that something else had to permit.