Clocks: Event, Ingestion, Availability, and Label Time

A record becomes safe for different uses at different moments. One card transaction carries five defensible timestamps. Each answers a different operational question: when the action occurred, when the producing system durably recorded it, when the data platform accepted it, when a decision path could actually retrieve it, and when its outcome became final enough to train or evaluate on.

None of these is a modelling convention a team is free to adopt or ignore. Each is set somewhere concrete, by someone who wrote the number down. In an Apache Kafka topic configuration. In Google's published measurement of how wrong its own clocks can be. In a Public Health England data-load failure. In a Visa dispute rule. In an EU delegated regulation with articles and annex tables. The rest of this lesson works through those cases, because a clock nobody can point at is a clock nobody owns.

Consider a card transaction scored at 10:00 on Monday and disputed later. How much later is not a matter of taste. Two published rulebooks bound it.

Regulation Z gives the consumer a window and the creditor a deadline. A billing-error notice is timely if the creditor receives it no later than 60 days after transmitting the first periodic statement reflecting the error. The creditor must then complete resolution within two complete billing cycles, and in no event later than 90 days after receiving that notice.

Visa's own windows run far longer. Its client bulletin of 14 January 2021 states: “Currently, issuers can process a dispute for a transaction where merchandise or services were not rendered due to merchant insolvency or bankruptcy no later than 120 calendar days from the last date the cardholder expected to receive the merchandise or services, not to exceed 540 calendar days from the transaction processing date.”

A training set that treats a transaction as non-fraudulent because no dispute has arrived is asserting that the 60-day, 90-day, 120-day and 540-day clocks have all run out. Usually they have not.

Event time answers when something happened. A prediction can only use a value once the serving system could have accessed it under the production path.

This gap matters when databases are backfilled, corrected, or replicated with delay. A row may describe an earlier event and still become available after the decision.

Point-in-time datasets therefore need more than a business timestamp. They need a defensible availability model for every feature source.

The two clocks were named and separated in 2015, in the paper on the Dataflow model that Akidau and ten colleagues published. Event time is “the time at which the event itself actually occurred”. Processing time is “the time at which an event is observed at any given point during processing within the pipeline”.

What the paper refuses to do is treat the distance between them as a defect awaiting a fix: “During processing, the realities of the systems in use (communication delays, scheduling algorithms, time spent processing, pipeline serialization, etc.) result in an inherent and dynamically changing amount of skew between the two domains.” Inherent, and dynamically changing. A single subtracted constant contradicts both words at once.

This is not one research group's vocabulary. Apache Flink's engine documentation states the same split independently, and draws the consequence for anyone building on it: in event time “the progress of time depends on the data, not on any wall clocks”. An availability model is nothing more than an honest account of that moving gap, source by source.

The tempting timestamp is often not the one that governed the production decision. Partition date is convenient for storage pruning, but it is usually derived from ingestion or the batch schedule. It may lag the event, may change after a replay, and is generally too coarse for scoring. A `last_updated` column shows the most recent mutation of a row rather than when each value was known. It overwrites earlier states, may record correction time, and destroys the intermediate versions a historical join needs. Only an availability timestamp states when the production decision path could retrieve the value — defined per source, tested against logs, supporting as-of retrieval.

There is a third substitution, and it never appears in a modelling document, because it happens at the broker. Apache Kafka has a topic-level setting called message.timestamp.type. The 4.0 configuration reference gives its purpose in one line: “Define whether the timestamp in the message is message create time or log append time.” Its valid values are CreateTime and LogAppendTime. Its default is CreateTime. Flip that one word to LogAppendTime and every record in the topic carries ingestion time in the field a downstream job will read as event time. No schema change, no migration, no line in a feature definition.

The default is not passive either. Under CreateTime the broker rejects any record whose timestamp is further ahead of the broker clock than message.timestamp.after.max.ms, default 3600000 ms — one hour. Under LogAppendTime that rule is ignored entirely. So a producer whose clock runs fast produces a rejected record in one configuration and, in the other, a record silently stamped by the broker instead. Redpanda, an independent implementation of the Kafka API, exposes the same switch as log_message_timestamp_type, with the same CreateTime default. Confluent Platform documents the setting downstream. The choice is a property of the interface, not one vendor's quirk, and it is one config key away from rewriting your entire event-time column.

Choose a high-impact feature and reconstruct exactly when it became available.

First, name the source event: the real-world action and its native timestamp. Second, trace the durable writes — commit, replication, ingestion, correction — and record each as a separate time rather than collapsing them into one. Third, inspect the serving path and establish when the online or batch consumer could first retrieve the value. For anything arriving over a log, that includes reading the topic configuration, because CreateTime and LogAppendTime put different clocks into the same field. Fourth, compare historical builds against production, and check whether backfills expose earlier values than the serving path ever did. Fifth, encode the policy: store or derive the chosen availability boundary, and add temporal tests that fail when it is violated.

RTS 25 demands one step from regulated firms that most feature audits skip entirely. It requires them to identify the exact point at which a timestamp is applied. Most feature definitions cannot answer that question about their own sources. The audit is finished when yours can.

Every feature definition should name its event time, availability rule, maximum tolerated staleness, correction behavior, and behavior when the time is missing. Dataset manifests should preserve the build cutoff and source snapshot. Monitoring should report latency distributions and the fraction of records assigned by fallback logic.

This is not an exotic standard invented for machine learning. The EU delegated regulation known as RTS 25 has been in force since 3 January 2018, and it writes a temporal contract by number rather than by convention. Divergence from UTC is capped at 1 millisecond for trading venues whose gateway-to-gateway latency exceeds 1 ms. It is capped at 100 microseconds, timestamped to a granularity of 1 microsecond or better, where that latency is 1 ms or less and for firms using high-frequency algorithmic trading techniques. The same regulation requires firms to identify the exact point at which a timestamp is applied. ESMA's guidelines say which latency figure decides which cap a venue falls under: the 99th percentile, not the typical one.

So the contract already exists somewhere, with an accuracy bound, a granularity, a stamping point, and a percentile that fixes the classification. A team that cannot say, for one feature, which clock it uses, how far that clock may drift, and where in the path the value is stamped is holding a weaker temporal contract than European trading venues have been legally obliged to hold since 3 January 2018.

These choices let reviewers distinguish legitimate historical evidence from reconstructed knowledge. They also make outages and backfills explainable.