Windows, Watermarks, and Late Data

In an unbounded pipeline, records arrive out of event-time order. Devices disconnect, networks retry, upstream systems replay data. The system still has to decide when to emit a result.

Windowing groups events by a chosen notion of time. A watermark estimates how far event-time processing has progressed. Triggers decide when a window produces early, on-time, or late updates.

None of these mechanisms makes late data disappear. They turn what is still missing into an explicit policy covering latency, corrections, state retention, and cost. Every term of that policy is a number someone has to choose, a default someone has to inspect, or a boundary someone has to publish.

A watermark has two failure modes, and the 2015 paper that defined the Dataflow model is candid about both. Watermarks are “sometimes too fast”, so data can still arrive behind them. They are also “sometimes too slow”, because a watermark is a global progress metric. One record can hold everything up: “the watermark can be held back for the entire pipeline by a single slow datum”. Even in healthy pipelines, the paper warns, “the baseline level of skew may still be multiple minutes or more”.

An early pane is a partial estimate emitted before the watermark reaches the window end. An on-time pane is emitted when the watermark crosses that boundary. A late pane updates the window afterwards, while state is still retained. A finalization boundary is where the system stops accepting revisions. Which of those four a pipeline actually produces depends on trigger, accumulation mode, lateness allowance, and sink semantics. The lateness allowance is the term that decides whether the third one exists at all.

A Flink window is created and destroyed by the clock. The documentation states the lifecycle without metaphor: “a window is created as soon as the first element that should belong to this window arrives, and the window is completely removed when the time (event or processing time) passes its end timestamp plus the user-specified allowed lateness”. That trailing term is the finalization boundary, and it is a number a developer types. Its shipped value is zero. The Allowed Lateness section says so outright: “Allowed lateness specifies by how much time elements can be late before they are dropped, and its default value is 0.” The same page repeats it plainly — “By default, the allowed lateness is set to 0.” So in an unconfigured Flink job the window is torn down at its end timestamp. The late pane a lifecycle diagram draws does not happen. The delayed element has nowhere to land.

Google's Dataflow runs the same regime on a different engine. “A watermark is a threshold that indicates when Dataflow expects all of the data in a window to have arrived.” “By default, results are emitted when the watermark passes the end of the window.” After that, “If the watermark has progressed past the end of the window and new data arrives with a timestamp within the window, the data is considered late data.” Correction is available, but as an opt-in: “You can allow late data with the Apache Beam SDK.”

Two engines, two vendors, one shape. Revision is a capability you switch on and pay for in retained state, not a property the window has by nature.

Event time is the timestamp associated with when the represented event occurred. It supports domain windows and historical semantics. It can arrive out of order, and it is only as good as the source clock. Processing time is the clock of the machine currently executing the operation. It is useful for timeout and latency triggers, it varies with backlog and retries, it cannot reveal late events by itself, and it is a weak basis for business ordering. A watermark is neither. It is an estimate of event-time completeness at a point in the pipeline. It advances window progress, it may be heuristic, it can be delayed by slow or silent sources, and it does not prove that no earlier event will appear. Confusing the three can make a low-latency pipeline look temporally correct when it is not.

“May be heuristic” is not a caution invented for a course. It is what the engines document about themselves. Apache Spark's Structured Streaming sets its watermark at the maximum event time seen by the engine minus a late threshold, and states the state-retention rule exactly: “For a specific window ending at time T, the engine will maintain state and allow late data to update the state until (max event time seen by the engine - late threshold > T).” The guarantee that follows is deliberately one-sided. A 2-hour watermark delay guarantees that no data less than 2 hours delayed is ever dropped. Past that line, the Structured Streaming Programming Guide says only: “Data delayed by more than 2 hours is not guaranteed to be dropped; it may or may not get aggregated.” May or may not. The engine will not tell you which, and the number in your dashboard does not carry the answer.

A peer-reviewed reading of the same algorithm goes further. Spark's construction, records a 2021 PVLDB paper comparing watermarks in Flink and Dataflow, “uses a non-conformant watermark algorithm for garbage collecting intermediate state that is identical to the grace period Kafka Streams uses for its final results feature: track a high watermark of the max event time ever seen within a stream, then offset that by a static allowed-lateness delta”. The construction three different products ship under three different names is one construction. The literature classifies it as non-conformant.

A fleet dashboard reports five-minute temperature windows and must decide how to treat delayed telemetry. The size of the gap is not a matter of taste: for vaccine storage the sampling interval is a published recommendation. The CDC's temperature-monitoring guidance for health care providers lists the features of a recommended digital data logger. A detachable buffered probe. An out-of-range alarm. A recommended uncertainty of ±0.5°C (±1°F). And a “Logging interval (or reading rate) that can be programmed to measure and record temperatures at least every 30 minutes”. The CDC defines such a logger as having “the capability for continuous monitoring” and reporting “how long a unit has been operating outside the recommended temperature range”. Immunize.org gives providers the same number: “A DDL provides a log of the temperature recorded at preset intervals (at least every 30 minutes is recommended).”

Hold the interval at that recommended maximum of 30 minutes and the six-hour disconnection stops being a vague gap. A reading every half hour for six hours was taken and withheld. They will arrive together, all stamped with event times inside windows the dashboard already closed and already reported. Whether those twelve readings reach the compliance record, and whether anyone downstream learns that the record changed, is decided by the lateness allowance — which, as the previous section showed, is zero until someone raises it.

The consumer decision comes first, ahead of any framework default. Five steps, in order.

1. Define the event clock: specify timestamp source, expected error, timezone, and clock-reset behavior. 2. Choose window semantics: fixed, sliding, session, or domain-specific, selected from the decision the window serves. 3. Set emission needs: decide whether consumers need early estimates, on-time values, or only corrected results. 4. Price revisions: measure the operational and storage cost of retaining state and rewriting outputs. 5. Publish finality: expose version, pane, and correction status so downstream users can interpret the value.

Steps 1 and 2 usually collapse into guessing a constant. The heuristic the engines actually ship is a fixed lateness bound. Flink's WatermarkStrategy.forBoundedOutOfOrderness lags the watermark behind the maximum event-time timestamp seen, by a constant maxOutOfOrderness. The documentation describes it — “For these cases, Flink provides the BoundedOutOfOrdernessWatermarks generator which takes as an argument the maxOutOfOrderness” — and states the consequence for anything slower: “If lateness > 0 then the element is considered late and is, by default, ignored when computing the result of the job for its corresponding window.” The 2021 PVLDB watermarks paper assesses that heuristic and the timeout heuristic together. Several of its authors built these systems. Their verdict: “We have found both of these heuristics to be problematic in practice, as they introduce unnecessary delays when the system is running well and not enough delay when problems arise, yielding large amounts of late data.” Failing in both directions is the point. The constant is too large when the pipeline is healthy and too small exactly when it is not. Their alternative gives step 1 a method instead of a guess: fit a statistical model, for example a Gamma distribution, to a rolling histogram of element lag, then delay the watermark by a chosen quantile such as 0.999. The authors found that substantially improved watermark latency while yielding less late data.

Steps 3 to 5 have a documented cautionary case at production scale. Kafka Streams shipped a default grace period of 24 hours on windowed operations. KIP-633, on the Apache Kafka wiki in 2021, is blunt about the result: “The current default value is 24hours, which has caused continuous problems and confusion for users of suppression since it means results won’t show up for 24 hours.” A default nobody chose became a day-long delay nobody expected. The same page notes the other edge of the same parameter: “Records coming in after the grace period has elapsed will be dropped from those windows.” The remedy a project of that size chose was not a better default. It was to force the number into the open. The old constructors were replaced with paired APIs, where “In constructing a windowed operation, users must choose between one of these two APIs and make a conscious decision whether to select a grace period or ignore that parameter for the time being.” Confluent's documentation records the outcome: “Since KIP-633, there is no default grace period for Kafka Streams.” And then, one line later, “However, in ksqlDB, the default is 24 hours.” One organisation, two products, two different answers to the same question. That is precisely why step 5 exists. The PVLDB authors put the obligation in one line: “the next best thing to having accurate results is knowing their inaccuracy”.

A streaming test suite should advance event time and processing time independently. Include records before the watermark, after the watermark, and beyond allowed lateness. Also test idle sources, duplicated offsets, source restarts, clock skew, and corrections that replace earlier values. Verify both emitted data and retained state. The expected result should specify pane status, version, and downstream side effect. Testing only the final aggregate misses the operational behavior that users actually observe.

“Test idle sources” names a defaulted mechanism, not a vague worry. When a partition goes quiet, the generator has nothing left to work from. Flink's documentation puts it plainly: “If one of the input splits/partitions/shards does not carry events for a while this means that the WatermarkGenerator also does not get any new information on which to base a watermark.” The condition has a name. “We call this an idle input or an idle source.” Then the consequence: “In that case, the watermark will be held back, because it is computed as the minimum over all the different parallel watermarks.” A minimum, not an average. One silent partition freezes event-time progress for everything downstream, and the escape hatch has to be requested, through an explicit WatermarkStrategy.withIdleness(Duration) that excludes the quiet channel. Confluent documents the same rule for a different vendor — “The watermark of an operator is the minimum of received watermarks over all partitions of all inputs.” — and publishes the values a test has to drive past: “The default maximum watermark drift is 5 minutes. This value matches the default maximum idleness detection timeout, which is also 5 minutes.” The PVLDB watermarks paper records the mechanism independently: “To mitigate the problem of a source task without input data, Flink source nodes can declare themselves as idle, which means that their output channels are temporarily excluded when subsequent nodes update their watermark.”

“Clock skew” has a date and a blast radius. At midnight UTC on 1 January 2017 the leap second made an interval computed inside Cloudflare's Go-based RRDNS software go negative, causing panics. John Graham-Cumming's post-mortem on the Cloudflare blog that day names the assumption: “The root cause of the bug that affected our DNS service was the belief that time cannot go backwards.” The mechanism was arithmetic: “At midnight UTC on New Year’s Day, deep inside Cloudflare’s custom RRDNS software, a number went negative when it should always have been, at worst, zero.” The blast radius was measured, not estimated: “At peak approximately 0.2% of DNS queries to Cloudflare were affected and less than 1% of all HTTP requests to Cloudflare encountered an error.” And recovery was dated: “The most affected machines were patched in 90 minutes and the fix was rolled out worldwide by 0645 UTC.” The Register's Thomas Claburn reported the same incident independently on 4 January 2017: “Cloudflare's RRDNS software is written in Go and uses Go's time.Now() function to fetch the current time.” and “The incident affected only a few machines across the company's 102 data centers, about 0.2 per cent of DNS queries, and less than 1 per cent of HTTP requests.” The repair was one character, broadening a check so that values less than zero were caught rather than recorded. A test suite that only shuffles the order of records would never have produced that value. One that drives the clock — forward past a 5-minute idleness timeout, and backwards across a second — would.