Batch, Change Data Capture, and Stream Ingestion

Volume checks alone can miss these correctness problems. The clearest public case is a load job whose output stayed plausible on every dashboard that watched it while thousands of records were simply absent.

Some lab result files were larger than the maximum file size the loader accepted, and what did not fit was never loaded. Public Health England published what that cost on 4 October 2020, in a statement by its interim chief executive, Michael Brodie: “A technical issue was identified overnight on Friday 2 October in the data load process that transfers COVID-19 positive lab results into reporting dashboards. After rapid investigation, we have identified that 15,841 cases between 25 September and 2 October were not included in the reported daily COVID-19 cases.” Over 75% of the missing cases — 11,968 of the 15,841 — were dated 30 September to 2 October alone.

The cost of the missing rows was afterwards measured rather than estimated in the abstract. A peer-reviewed natural experiment built on the failure, published in PNAS in 2021 by Fetzer and Graeber, found that one case referred late to contact tracing was associated with 18.6 additional infections and about 0.24 additional COVID-19 deaths over the following six weeks. The share of contacts reached within 24 hours fell from around 80% to just above 60%. The pipeline reported success throughout. The failure was in the loader, and the damage was in the six weeks that followed.

Teams often frame ingestion as a choice between slow batch jobs and fast streams. That framing misses the harder questions about replay, updates, retries, and meaningful ordering.

A daily snapshot can be correct and sufficient for monthly forecasting. A stream can be low-latency and still unusable, if events lack stable IDs or arrive with inconsistent clocks.

The ingestion contract should describe records, delivery, ordering, correction, retention, and recovery. Latency matters, but historical reproducibility and failure behavior matter just as much.

ML workflows also need two views of time. They must know when the real event happened and when the platform learned about it, and knowing both shapes feature windows, labels, monitoring, and backfills. The distinction is not folklore. It was defined in 2015, in the paper that introduced the Dataflow Model, and the first of its two time domains reads: “Event Time, which is the time at which the event itself actually occurred, i.e. a record of system clock time (for whatever system generated the event) at the time of occurrence.” Processing time is when the pipeline observed it. The gap between the two is a dynamically changing skew, and a watermark tracks it. The same paper concedes the limit that makes a late-data policy necessary at all: for most real-world distributed data sets the system lacks sufficient knowledge to establish a 100% correct watermark, so most watermarks are heuristic. Apache Flink's documentation adopts the same two domains and says outright that it implements many techniques from the Dataflow Model.

Kafka’s own design document names the three delivery guarantees plainly and separately. At most once may lose messages but never redelivers. At least once never loses but may redeliver. Exactly once delivers each message once. Since version 0.11.0.0 the producer has an idempotent option. It works because “the broker assigns each producer an ID and deduplicates messages using a sequence number”. That is a property of the transport, not a promise about your sink.

Each pattern can be reliable when its assumptions match the source and consumer. Change data capture carries the hardest coordination problem of the four: reconciling a full-state read with a live log. That problem has a published solution rather than a warning.

Netflix hit it and wrote up the fix. DBLog, published in 2020 by Andreakis and Papapanagiotou, states the idea in its abstract: “DBLog utilizes a watermark based approach that allows us to interleave transaction log events with rows that we directly select from tables to capture the full state.” Low and high watermarks go into the transaction log itself. A chunk of the table is selected between them. No locks are taken on the source.

Debezium adopted the same algorithm in version 1.6, and described the snapshot window and its buffer on the Debezium blog on 7 October 2021. In the MySQL connector the default incremental-snapshot chunk size is 1,024 rows. That write-up names the three problems the approach removes: snapshots that must run all-or-nothing, snapshots that cannot be resumed after a restart, and change streaming blocked until the snapshot finishes. Two independent systems now ship the mechanism. That is the difference between telling a team to coordinate snapshot and log, and pointing them at how it is done.

End-to-end behavior depends on producer IDs, transport, consumer state, and sink writes together. Steps 3 and 4 are where the guarantee is actually bought — making transformations repeatable across retries, and committing output without duplicate logical results. The bill for it has been published.

Google's MillWheel implements exactly-once by journaling a unique record ID in the same atomic write as the state modification, with a Bloom filter of record fingerprints as the fast path. Ten authors published it in 2013, and they measured what the machinery costs. On a single-stage pipeline running over 200 CPUs, the median record delay was 3.6 milliseconds and the 95th-percentile delay 30 milliseconds. Then comes the other half of the comparison: “This test was performed with strong productions and exactly-once disabled. With both of these features enabled, median latency jumps up to 33.7 milliseconds and 95th-percentile latency to 93.8 milliseconds.”

That is roughly a ninefold increase at the median, for a property most architecture diagrams draw as a single arrow between two boxes. The chain above is not free at any link. Steps 3 and 4 are where the cost lands.

A good design starts with source behavior and downstream consequences. Step 4 is the one teams write in the abstract and discover concretely, because the window you are allowed to replay belongs to systems the pipeline does not own.

Kafka's per-topic retention.ms defaults to 604800000 ms — seven days — and the documentation does not present it as a storage knob: “This represents an SLA on how soon consumers must read their data.” The delete markers that carry removals expire sooner still, after delete.retention.ms, 86400000 ms — one day — by default. On the source side, MySQL is typically configured to purge its binary logs after a set period. That is exactly why the Debezium MySQL connector must take an initial consistent snapshot before it can stream from the log at all.

So the safe recomputation boundary in step 4 is not a number the team chooses. It is the shortest of the retention windows the team inherits: seven days of transport by default, one day for deletes, and whatever is left on the binlog. Any backfill plan that reaches further back has to name the historical source it will read instead.