Skip to content
AI.info

MLOps

Streaming and Event-Driven Inference

Design event-driven prediction systems with explicit event time, state, replay, deduplication, and side-effect semantics.

By the end you can

Events arrive in a different order from the world

A payment event reaches the stream processor before the account-update event that preceded it in reality. A mobile client reconnects and sends three hours of buffered activity. One message is delivered twice after a broker retry.

Streaming systems have to define time and identity explicitly. Arrival order is not event order. And processing an event once does not mean every downstream side effect happens once.

Example

One transfer, four Horizon receipts, a £24,000 shortfall

A subpostmaster moved £8,000 in cash from her core branch into her outreach branch. Once. Horizon recorded it four times.

The mechanism was a re-run. An incomplete Pouch Delivery script was executed again, and the re-run issued duplicate pouch IDs. The trial called it the Dalmellington bug. Mr Justice Fraser set out what it cost in the Bates v Post Office judgment of 16 December 2019.

“The transfer replicated four times. She had Horizon receipts in her outreach branch of £32,000 (4 times £8,000) and therefore a discrepancy of £24,000, as she had only transferred £8,000 in cash out of her core branch into it.” — Mr Justice Fraser, Bates v Post Office, 16 December 2019.

The replay was not exotic. Fujitsu's own investigation found 112 occurrences affecting 88 different branches over the previous five years, and the judge held that there was “a significant and material risk” of branch accounts being affected by such bugs. The vendor's own problem report, published by the statutory Post Office Horizon IT Inquiry, shows what was proposed in the meantime. An October 2015 email chain records the mitigation: “advising them not to press enter multiple times”, pending a code fix in counter Release 13.05.

The code fix was months away. The advice was for the humans.

  • Duplicate at the source: An incomplete Pouch Delivery script was re-run and issued duplicate pouch IDs for a single real remittance.
  • State update: Each duplicate was accepted as a distinct transfer, so £8,000 of cash produced £32,000 of Horizon receipts in the outreach branch.
  • Side effect: The £24,000 gap did not surface as a flagged replay of a known identifier. It surfaced as a shortfall attributed to the branch.
  • Operational harm: Fujitsu's investigation found 112 occurrences affecting 88 different branches over the previous five years, and Fraser J found “a significant and material risk” of branch accounts being affected by such bugs.
  • Lesson: The interim fix was “advising them not to press enter multiple times” until counter Release 13.05 — a human-behaviour patch standing in for a missing idempotency boundary.

Visual

Four clocks in an event-driven decision

Each timestamp supports a different claim.

FigureTimeline · 4 stops

  1. Event time

    When the source says the real-world activity occurred.

  2. Ingestion time

    When the platform first accepted the event.

  3. Processing time

    When a worker handled the event or updated state.

  4. Decision time

    When the prediction or action became effective.

Comparison

Watermarks, triggers, and allowed lateness do different jobs

Treating them as one “late data setting” hides behavior. Two published runners show how far apart the three knobs actually sit.

A watermark is an estimate of completeness, and nothing more. Beam's programming guide defines it: “Beam tracks a watermark, which is the system's notion of when all data in a certain window can be expected to have arrived in the pipeline. Once the watermark progresses past the end of a window, any further element that arrives with a timestamp in that window is considered late data.” Google Cloud Dataflow's documentation states the same threshold semantics — “A watermark is a threshold that indicates when Dataflow expects all of the data in a window to have arrived.” — and defines late data the same way, as arrival after the watermark has passed the end of the window.

Neither sentence says how long a late element may still revise a result. Beam does not quietly assume an answer. Its default windowing configuration advances the watermark past the end of the window, and then the guide is blunt: “This default configuration does not allow late data.” Lateness is a separate, explicit request: withAllowedLateness.

So there are three decisions here, not one. A watermark estimates completeness. A trigger decides when to emit. A lateness budget decides how long revisions are accepted. Only the third one keeps a late event alive.

FigureComparison · 3 columns

Watermark

Estimates progress in event time.

  • Guides when a window may be considered mostly complete
  • Can advance despite future late arrivals
  • Depends on source behavior
  • Is not proof that older events are impossible

Trigger

Decides when intermediate or final results are emitted.

  • Can use processing time or counts
  • May produce multiple revisions
  • Affects latency and downstream load
  • Does not change event-time membership

Allowed lateness

Defines how long late events can revise state or results.

  • Balances correction and retention cost
  • Requires update semantics downstream
  • Can discard extremely late events
  • Must match product tolerance

State is part of the model input

A streaming prediction usually depends on rolling counts, recency, sessions, or sequence summaries. Those features are maintained state. Their correctness depends on windowing, late-data policy, deduplication, and checkpoint recovery.

The model can sit perfectly still while feature state drifts underneath it, moved by operational semantics alone. Monitor state size, lag, watermark progress, correction rate, and replay behavior alongside the scores.

Case

A watermark is sometimes too fast and sometimes too slow

A watermark fails in two opposite directions, and the Dataflow paper named both in 2015. It is “sometimes too fast”, so late data arrives behind it. It is also “sometimes too slow”, because it is a global progress metric: “the watermark can be held back for the entire pipeline by a single slow datum”.

Both are true of the same number at the same time. A trigger decides when to emit. A watermark only estimates when the input is done.

Analogy

The newsroom publishes early and corrects in public

Reports reach a newsroom from correspondents in different time zones and over unreliable networks. Editors publish an early version, revise it as late reports arrive, and preserve which edition readers saw.

A printed correction reaches a reader who has already formed an opinion, and nothing else has happened in between. In a streaming pipeline the side effect fires automatically before the correction arrives. So the design has to state whether a revised event changes only analytics, or also reverses actions already taken.

Streaming outputs are editions of evolving state, not timeless facts.

Steps

Design event-driven inference semantics

Write these choices before selecting window and broker settings. Two of the five steps already have published contracts you can adopt instead of inventing.

Step 1 is event identity, and there is a normative rule for it. CloudEvents, a CNCF graduated project, makes identity the deduplication contract in the definition of its REQUIRED id attribute: “Identifies the event. Producers MUST ensure that source + id is unique for each distinct event. If a duplicate event is re-sent (e.g. due to a network error) it MAY have the same id. Consumers MAY assume that Events with identical source and id are duplicates.” That is version 1.0.2, released on 5 February 2022.

The rule exists because the transport does not keep the promise for you. Azure Event Grid carries CloudEvents 1.0 and documents at-least-once delivery in its own words: “It tries to deliver each message at least once for each matching subscription immediately”. The caveat follows: “If the endpoint responds within 3 minutes, Event Grid attempts to remove the event from the retry queue on a best effort basis, but duplicates might still be received”. On ordering it is equally plain: “Event Grid doesn't guarantee order for event delivery, so subscribers might receive events out of order.” The producer promises identity. The pipe promises neither once nor in order.

Step 4 is protecting side effects, and it has a header field and a definition to cite. An IETF working-group draft of 15 October 2025 states it: “An idempotency key is a unique value generated by the client which the resource uses to recognize subsequent retries of the same request.” The server behaviours are specified rather than implied. A retried request is answered with “the result of the previously completed operation”. A concurrent one gets a conflict error.

Stripe's API implements exactly this: “Stripe's idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.” It also puts a number on the deduplication horizon that step 1 asks you to choose: “You can remove keys from the system automatically after they're at least 24 hours old.” A day is how long that boundary remembers.

FigureProcess · 5 steps

  1. 1. Define event identity

    Choose stable keys, source sequence, and deduplication horizon.

  2. 2. Define all clocks

    Record event, ingestion, processing, and decision time.

  3. 3. Specify state and windows

    Name aggregation, retention, watermark, trigger, and lateness policy.

  4. 4. Protect side effects

    Use idempotency, transactions, outbox patterns, or reconciliation.

  5. 5. Test replay and disorder

    Inject duplicates, late events, partition movement, and checkpoint recovery.

Key idea

“Exactly once” must name the boundary

A stream engine may guarantee one state update per event inside its own checkpointed topology. An external database, notification service, or payment API has a different transaction boundary.

State the guarantee precisely: source-to-state, state-to-sink, or end-to-end side effect. Use idempotency keys, transactional sinks, or reconciliation where the guarantee stops.

Flink draws that boundary in its own documentation rather than leaving it to the reader. Its checkpointing is a variant of the Chandy–Lamport algorithm, known as asynchronous barrier snapshotting. Carbone and colleagues defined it in 2015: “ABS persists only operator states on acyclic execution topologies while keeping a minimal record log on cyclic dataflows”. Chandy and Lamport published the original in 1985. Having named the machinery, the page then says what the guarantee is not: “Given that Flink recovers from faults by rewinding and replaying the source data streams, when the ideal situation is described as exactly once this does not mean that every event will be processed exactly once. Instead, it means that every event will affect the state being managed by Flink exactly once.” End to end, the same page adds, needs two things the engine cannot supply on its own: replayable sources, and transactional or idempotent sinks.

Kafka's own default now favours the stronger guarantee. In the 4.0 producer configuration reference, enable.idempotence defaults to true and acks defaults to all. The rule is stated directly: “Idempotence is enabled by default if no conflicting configurations are set.” Confluent's reference documents the same default. That covers the producer-to-broker hop. It says nothing about the ticket your consumer creates.

The boundary is a latency decision as well as a correctness one, and one system measured it. MillWheel reported a median record delay of 3.6 ms and a 95th-percentile latency of 30 ms on a single-stage pipeline running over 200 CPUs, in 2013. Then it named the configuration those numbers came from: “This test was performed with strong productions and exactly-once disabled. With both of these features enabled, median latency jumps up to 33.7 milliseconds and 95th-percentile latency to 93.8 milliseconds.” Roughly ninefold on the median. Paid inside the engine, and still stopping at the engine's own state.

Exactly-once claims are meaningful only when the covered boundary and failure assumptions are explicit.

Replay is part of normal operation

Recovery, new logic, and source correction all require replay. A stream that cannot be replayed under known semantics is hard to debug and unsafe to evolve. Flink's end-to-end requirement begins with exactly this property: a replayable source, without which rewinding is not available at all.

Keep raw events or a defensible reconstruction source. Version the state logic. Test how downstream consumers handle a correction.

Key takeaways