ML Workload Architecture: Batch, Online, Streaming, and Edge

A fraud team asks for real-time scoring. One stakeholder means a decision within 150 milliseconds. Another means the alert should arrive before the daily review meeting. The phrase conceals two architectures with different cost and failure behavior.

European law has already run that separation in public, and written both clocks into one statute. Regulation (EU) 2024/886, the Instant Payments Regulation, was adopted on 13 March 2024. Article 5a(4)(c) gives the payee's payment service provider 10 seconds from the time of receipt of the payment order to make the funds available and confirm completion. Article 5a(5) says what happens if that confirmation does not arrive inside the 10 seconds: the payer's PSP restores the payer's account. A deadline, and a named compensating action for missing it. De Nederlandsche Bank, the Dutch supervisor, states the user-visible half plainly: “With instant credit transfers, the funds are made available on the payee's account within 10 seconds.”

The same regulation then takes one check off that path entirely. Article 5d moves screening for targeted financial sanctions onto a daily customer-level check, and expressly forbids performing it during execution of the transfer: “PSPs shall carry out such verifications immediately after the entry into force of any new targeted financial restrictive measures, and immediately after the entry into force of any amendments to such targeted financial restrictive measures, and at least once every calendar day.” The European Central Bank's summary of the regulation records the same obligation: PSPs offering instant credit transfers verify periodically, and at least daily, whether any of their payment service users are subject to targeted financial restrictive measures. PSPs had to comply with Article 5d by 9 January 2025. One product, two clocks. The law says which work goes on which.

The strict promise has a measured price. Take a server that usually answers in 10ms but has a 99th-percentile latency of one second. Send it one request and one request in 100 comes back slow. Now fan the same request out. Dean and Barroso ran that arithmetic in Communications of the ACM in February 2013: “If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second”. Nothing about the model changed between those two sentences. Tail latency is a property of the topology.

An online endpoint can serve a model trained once per month. A nightly batch can use features updated every few seconds. Streaming describes how events are processed, not whether parameters update continuously.

Conflating serving with learning leads to unnecessary complexity. Name the prediction clock, feature clock, label clock, and retraining clock independently, and give each one a number. A regulator writing for an entire market found it necessary to do exactly that.

MiFID II RTS 25 sets how accurate a trading firm's business clocks have to be. It fixes the reference before it fixes any tolerance. Article 1: “Operators of trading venues and their members or participants shall synchronise the business clocks they use to record the date and time of any reportable event with the Coordinated Universal Time (UTC) issued and maintained by the timing centres listed in the latest Bureau international des poids et mesures Annual Report on Time Activities.” The instrument is Commission Delegated Regulation (EU) 2017/574, adopted in 2016 under the MiFID II directive.

Only then does it split the requirement, and it splits it by activity rather than by system. Activity using the high-frequency algorithmic trading technique gets a maximum divergence from UTC of 100 microseconds, timestamped to 1 microsecond. Other trading activity gets 1 millisecond, timestamped to 1 millisecond. Voice systems, request-for-quote systems requiring human intervention, and negotiated transactions get 1 second. Trading-venue operators face 100 microseconds where gateway-to-gateway latency is 1 millisecond or less, and 1 millisecond above that. The same firm, on the same venue, on the same day, runs clocks that differ by four orders of magnitude in tolerance. Four different kinds of event are being recorded. The regulation does not ask whether the venue is fast. It asks which event is being timestamped.

Teams often overpay for synchronous serving because they ignore three other dimensions. Each of the three has been measured, and the measurements are public.

State and ordering. A watermark is how a streaming system says how far event time has progressed. The canonical account of why that is hard is Google's Dataflow Model paper, published in 2015, and it gives watermarks two failure modes rather than one. They are sometimes too fast, so late data arrives behind them. And, in Section 2.3: “They are sometimes too slow. Because they are a global progress metric, the watermark can be held back for the entire pipeline by a single slow datum.” The paper concludes that watermarks alone are insufficient, and that triggers and accumulation modes are needed as well. Apache Flink ships the same mechanism and inherits the same consequence. Its release 1.20 documentation defines the marker — “A Watermark(t) declares that event time has reached time t in that stream, meaning that there should be no more elements from the stream with a timestamp t' <= t” — and defines elements arriving after the watermark has passed their timestamp as late, to be handled explicitly through allowed lateness. “Late events can revise state” is not a caveat. It is a configured budget with a named knob.

Failure recovery. The recovery unit differs by pattern. Batch favors reruns — a partition rather than a whole month. Online favors failover. Edge favors graceful local fallback. Replay is what makes these different. Re-running work that already emitted a side effect can duplicate that side effect, so event identity and idempotent handling are part of the pattern choice, not an afterthought.

Economics. Latency has a measured price, and two companies measured it independently. Google slowed its own search results page on purpose and reported the result in June 2009: “Our experiments demonstrate that slowing down the search results page by 100 to 400 milliseconds has a measurable impact on the number of searches per user of -0.2% to -0.6% (averaged over four or six weeks depending on the experiment).” The effect deepened with exposure. A 200 ms delay cost 0.22% of searches in the first three weeks and 0.36% in the second three, and it persisted after the delay was removed. Microsoft ran the mirror-image experiment at Bing, slowing 10% of users by 100 ms and another 10% by 250 ms for two weeks. Kohavi and colleagues reported it at KDD in 2014: “The results of this controlled experiment showed that every 100msec speedup improves revenue by 0.6%.” Those numbers are the argument on both sides. They tell a team what a millisecond is worth here, so that always-on capacity, cold starts, and accelerator choice can be weighed against something instead of asserted against nothing.

A worked hypothetical, to fix the vocabulary: suppose one demand model supports planning, website ranking, and shelf alerts. The model is the same object in all four paths below. Only the clock, the state, and the recovery unit change.

The four patterns are not a taxonomy invented for teaching. The industry benchmark that vendors submit hardware to is built on the same split, and it attaches published millisecond budgets to it. MLPerf Inference, whose paper was written by 47 authors and published in 2020, says where the scenarios came from: “On the basis of that feedback, we identified four scenarios that represent many critical inference applications: single-stream, multistream, server, and offline.”

Each scenario carries its own metric and its own budget. In the v0.5 rules, server-scenario QoS constraints were 15 ms for heavy image classification, 10 ms for light image classification, 100 ms for heavy object detection, 10 ms for light object detection, and 250 ms for machine translation. No more than 1% of vision queries and 3% of translation queries could exceed the bound. The offline scenario is measured as raw throughput in samples per second, with latency unconstrained. Red Hat, an independent submitter, describes the same semantics from the other side of the benchmark: “In offline mode, we assess the raw throughput of the system-under-test by batch processing the entire dataset of queries. In server mode, the benchmark simulates an interactive environment, where the load generator sends requests to the system one at a time, following a Poisson distribution, achieving a specified average requests-per-second rate.” The first call for submissions produced more than 600 reproducible measurements from 14 organisations. Batch and online are not two moods of the same system. They are scored on different axes.

Edge or on-device is the fourth pattern, and its distinguishing cost is recovery. CrowdStrike's own root cause analysis of 6 August 2024 opens its account of what happened with the design: “The CrowdStrike Falcon sensor delivers powerful on-sensor AI and machine learning models to protect customer systems by identifying and remediating the latest advanced threats.” Those models ship as Sensor Content, compiled into the sensor release. Cloud-delivered Rapid Response Content is a different thing: behavioural heuristics, “separate and distinct from CrowdStrike's on-sensor AI prevention and detection capabilities”, interpreted on the device.

The defect was a shape mismatch between the two: “The new IPC Template Type defined 21 input parameter fields, but the integration code that invoked the Content Interpreter with Channel File 291's Template Instances supplied only 20 input values to match against.” On 19 July 2024 two additional Template Instances were deployed. One of them introduced a non-wildcard matching criterion for the 21st input. Reading a value that was not there produced an out-of-bounds memory read beyond the end of the input data array, and the system crashed.

Microsoft's David Weston put the blast radius in writing the next day: “We currently estimate that CrowdStrike's update affected 8.5 million Windows devices, or less than one percent of all Windows machines.” The Cybersecurity and Infrastructure Security Agency (CISA) issued a public alert on 19 July 2024 recording that the outage impacted Windows 10 and later systems, did not impact Mac and Linux hosts, and was due to the content update rather than to malicious cyber activity. CrowdStrike reported that as of 29 July at 5 p.m. PT, roughly 99% of Windows sensors were online compared with before the content update. A push measured in minutes, a restoration measured in days. That interval is the whole of what “edge favors graceful local fallback” means.

Work through these questions before drawing the serving topology. Set the decision deadline: measure how late the output can arrive before it loses value. Article 5a(4)(c) says 10 seconds, RTS 25 says 100 microseconds, and both are answers to this question, not to a question about technology. Set the freshness requirement for features and model parameters. Define state and replay: ordering, windows, deduplication, allowed lateness, and side-effect recovery. Model load and failure, including peaks, outages, disconnected operation, and backlogs.

Then compare total cost, and compare it against production traces rather than intuition. The shape of real demand is more skewed than anyone designs for. Researchers at Microsoft and Brown University characterised every function invocation across Azure's entire infrastructure between 15 and 28 July 2019. Invocations per day vary by more than 8 orders of magnitude. Their paper, at USENIX ATC 2020, reports: “45% of the applications are invoked once per hour or less on average, and 81% of the applications are invoked once per minute or less on average.” Meanwhile the 18.6% most popular applications account for 99.6% of all invocations. Keeping that long tail warm costs more than executing it.

The skew is not one provider's accident. A 2023 study of Huawei Cloud's traces, covering more than 7 months and more than 1.4 trillion invocations, found the same distribution from a different platform: “requests vary by up to 9 orders of magnitude across functions, with some functions executed over 1 billion times per day”, while fewer than 10% of functions in the public trace are invoked at least once per minute. Utilisation and cold starts are therefore where the money is. The honest comparison is compute, networking, storage, operations, and the consequence of delay — priced with the numbers Google and Bing published rather than with a feeling about urgency.

A workload pattern is successful when it meets the decision deadline with acceptable evidence, cost, and recovery behavior. “Real time” is not a maturity level. Regulation (EU) 2024/886 puts one obligation on a 10-second clock and moves another onto a daily one in the same statute. RTS 25 grants 100 microseconds to one activity and 1 second to another under one reference time. MLPerf scores the server scenario against a millisecond budget and the offline scenario against throughput, with latency unconstrained. In each case the clock follows the decision.

The next design artifact should list each prediction path and its clocks, state, fallback, and record of what ran. That record includes the version of every default the path relies on, since Kafka 3.0.0 and 3.1.0 shipped one that was not in force.