Batch Inference Systems

Batch and online scoring are already two measured regimes. MLPerf Inference, the benchmark MLCommons runs, defines four scenarios. In the Server scenario a system must answer each query inside a latency bound “that varies from 15 to 250 milliseconds”, and no more than 1% of vision queries and 3% of translation queries may exceed it. The Offline scenario removes the clock: “The offline scenario represents batch-processing applications where all data is immediately available and latency is unconstrained.” The MLCommons inference rules make that concrete. One query, at least 24,576 samples, sent in a single burst, scored on “Measured throughput”.

Unconstrained latency is not unconstrained anything else. Once the deadline stops defining success, something else has to define what finished means. The sharpest published definition of a completed batch belongs to a regulator, not to a scheduler. The FCA and the PRA wrote it into their final notices of 19 November 2014 against Royal Bank of Scotland Plc, National Westminster Bank Plc and Ulster Bank Ltd, in one line: “That day’s batch processing is complete when all balances are final.” Not most jobs green. Not the workflow exited zero. All balances final.

Silent omission is not a hypothetical either. For four years it lived in the storage layer itself. Until 1 December 2020 Amazon S3 was eventually consistent, so a job could write its output objects and then not see all of them when it listed the bucket. Apache Hadoop maintained an entire subsystem for this, S3Guard, backed by a DynamoDB table, written between 2016 and 2020 to correct, among other things, “Newly created objects excluded from directory listings” and “Newly deleted objects retained in directory listings”. AWS announced strong read-after-write consistency on 1 December 2020, for all applications, in all Regions, at no additional cost: “S3 also provides strong consistency for list operations, so after a write, you can immediately perform a listing of the objects in a bucket with all changes reflected.” The Apache Software Foundation then removed S3Guard from Hadoop in 2022 under HADOOP-17409, recording the reason plainly: “Now that S3 is consistent, there is no need for S3Guard at all.” For four years, a listing had been a guess.

Batch inference therefore needs explicit population manifests, freshness rules, and completion semantics that do not depend on a listing, an exit code, or a green dashboard. A job is complete only when the intended scoring population is accounted for.

The population a batch scores is whatever its selection rule says it is, and a selection rule can be wrong for years without failing.

Scheduled scoring can often recover by recomputing partitions from immutable inputs. This makes batch attractive for high-volume work that does not require synchronous responses.

However, reruns can overwrite newer outputs or duplicate downstream actions. The job should publish versioned results, reconcile expected and actual partitions, and activate a release only after declared completion criteria pass.

Unbounded replay has a documented shape. Manual re-loading of failed jobs pushed Ulster Bank's backlog past a full day: “By 21 June 2012, batch processing for Ulster Bank was more than one day behind. This meant that the next day's batch processing started before the current day's batch processing was complete.” Two days' runs were now competing in the same queues. Recovery work and scheduled work were indistinguishable to the system executing both. The FCA dates the breach period from 1 August 2010, when a Group Internal Audit on mainframe batch processes “which identified the risk of a batch scheduler failure” was issued, to 10 July 2012. The risk had been named in writing for nearly two years. The RBS Group Board minuted it in July 2012, after the fact: “with hindsight, batch processing was taken for granted”.

Airflow changed its mind about this default, and the change reads as a statement about what a scheduler should assume: “Airflow 3.0 changes the default behavior for new DAGs by setting catchup_by_default = False in the configuration file”. A DAG that does not ask will no longer backfill the intervals it missed. Astronomer's upgrade guide records the same change independently — “the default for [scheduler].catchup_by_default has changed from True to False”. A replay policy that lives in a config default is still a replay policy.

Reliable batch delivery keeps scoring the rows separate from switching them on. The manifest, not the scheduler's exit status, is what says the run is done.

Separate scoring, validation, and activation so partial work cannot leak. Step 4 is not a metaphor. In the Apache Iceberg table specification, a commit is the swap of a single table metadata pointer, and the files belonging to a version are tracked by manifest files and a manifest list. Activation is one pointer move. Everything before it is a draft.

How results become visible affects consistency and recovery, and the cost of getting it wrong has been measured. Across Databricks' first cloud years, 2014–2016, “around half the support escalations we received were due to data corruption, consistency or performance issues due to cloud storage strategies”. That count comes from the 2020 Delta Lake paper, which also gives the structural reason in one sentence: “Any transaction that needs to write or update multiple objects risks having partial writes visible to other clients.” The heading it sits under is “No atomicity across multiple objects”. By the time of the paper, Delta Lake was deployed at thousands of customers processing exabytes per day. The Apache Iceberg table specification arrives at the same remedy from an independent codebase: writers commit by swapping one table metadata pointer, with each version's file set recorded in manifest files and a manifest list. Two formats, one conclusion. A multi-object write becomes safe only when a single-object swap decides when it is visible.

The user may never call an endpoint. Late or incomplete results are still a service failure, and one such failure has a published price. On 19 November 2014 the FCA and the PRA each issued a Final Notice against Royal Bank of Scotland Plc, National Westminster Bank Plc and Ulster Bank Ltd over a batch scheduler failure: £42,000,000 from the FCA and £14,000,000 from the PRA, cut from £60,000,000 and £20,000,000 by a 30% settlement discount. It was the PRA's first financial penalty. The incident began on 18 June 2012 and affected at least 6.5 million UK customers, 92% of them retail. Ulster Bank's batch scheduler was short of full functionality until 10 July 2012. The trigger was a rollback: backing out a scheduler upgrade caused “the subsequent release of incomplete batches in Ulster Bank's and NatWest's systems”.

So measure completion against the downstream decision deadline, not only scheduler duration. A good batch run can be replayed, compared, rolled back, and reconciled without rewriting the meaning of history.