Pipeline Design and Orchestration

A promotion task moves an alias, then times out before it can report success. The alias has already moved. The orchestrator's record of the attempt is not a record of what changed outside it.

The largest published version of that failure was written up by the company it happened to. AWS's summary of the 19-20 October 2025 disruption in the Northern Virginia (US-EAST-1) Region describes a delayed actor overwriting newer external state, and then a piece of automation removing what it had just written.

On 4 October 2020 Public Health England published a statement about the previous week's COVID-19 case counts. The automated transfer had run. The output was silently short.

“A technical issue was identified overnight on Friday 2 October in the data load process that transfers COVID-19 positive lab results into reporting dashboards. After rapid investigation, we have identified that 15,841 cases between 25 September and 2 October were not included in the reported daily COVID-19 cases.” — Michael Brodie, Interim Chief Executive of Public Health England, 4 October 2020.

Over 75% of the missing records — 11,968 of the 15,841 — fell in the three days from 30 September to 2 October. Every individual still received their test result normally. What failed was the load into the reporting dashboards, and with it contact tracing, which waited until the backlog was transferred by 1am on 3 October. The pipeline did not crash. It reported success and published a number that was 15,841 cases wrong.

Orchestration proves that declared tasks followed declared transitions. It does not prove that the workflow modeled the right dependencies, or that external side effects were safe.

The orchestrator sees only part of the workflow reality. Workflow state is task dependencies, attempts, scheduling, timeout, and completion status. Artifact state is the datasets, models, reports, images, and manifests that tasks produce. External system state is warehouse commits, registry records, aliases, tickets, and traffic routes. Decision state is approvals, exceptions, release conditions, and rollback intent. Only the first of those four lives inside the scheduler.

The separation is not academic. In the October 2025 US-EAST-1 event the data plane stayed healthy while the control plane failed, and AWS says so directly: “Existing EC2 instances that had been launched prior to the start of the event remained healthy and did not experience any impact for the duration of the event.” What broke was the ability to change anything. New launches failed from 11:48 PM PDT on 19 October until full EC2 recovery at 1:50 PM PDT on 20 October. A workflow that only watches its own control plane cannot tell those two situations apart.

A DAG becomes misleading when every relationship is drawn as a simple arrow. Three different things then look identical.

A data dependency means a task needs a specific artifact produced upstream. Its identity should be explicit. It supports caching and replay, and it can be validated before execution — evaluate model digest M on dataset snapshot D.

A control dependency means a task must wait for an event or an approval. It may transfer no data at all. It represents ordering or authority, and it needs a timeout and an escalation path — publish only after risk approval.

A conditional branch means the next task depends on observed evidence. It requires a recorded predicate, it must handle every outcome, and it can create selection bias — promote only if slice and latency gates pass.

Drawn identically, the three become indistinguishable in a diagram. The review question each one demands — which artifact, whose authority, what predicate, recorded where — stops being asked.

A task boundary is useful when its inputs and outputs can be named, inspected, cached, retried, and owned. Hidden reads from mutable tables or configuration services do the opposite. They make the DAG look simpler while weakening reproducibility.

Side effects require extra care. Writing a registry entry, sending a notification, or switching traffic is not equivalent to producing an immutable dataset partition. Retry semantics must reflect that difference.

The code around the model has been measured. In 2015 D. Sculley and nine colleagues put a ratio on it: “Because a mature system might end up being (at most) 5% machine learning code and (at least) 95% glue code, it may be less costly to create a clean native solution rather than re-use a generic package.” The same sentence had appeared in their own workshop paper the year before. The ratio is an argument about where boundaries belong, not a number anyone should copy onto a slide about their own pipeline. Most of what an orchestrator coordinates is not the model. It never was.

Use the contract for each task before connecting the DAG: name inputs and outputs with immutable identities and declared schemas; declare every external write, alias move, message, and ticket; define retry semantics — idempotency, reconciliation, backoff, maximum attempts; emit task state, artifact identity, timing, resource use, and failure reason; and document how to resume, skip, compensate, or rebuild the task.

Two independent organisations publish that contract in their own documentation, and both derive it from the retry. Airflow's best-practices page begins there: “Airflow can retry a task if it fails. Thus, the tasks should produce the same outcome on every re-run.” It then makes the rule concrete. An INSERT during a re-run “might lead to duplicate rows in your database”, so replace it with UPSERT. On the input side: “Read and write in a specific partition. Never read the latest available data in a task.”

Google's Cloud Composer documentation states the granularity rule independently: “Each task should be an idempotent unit of work. Consequently, you should avoid encapsulating a multi-step workflow within a single task, such as a complex program running in a PythonOperator.” Between them the two documents name the two ways a task contract is usually broken. A task that reads whatever is newest. A task that hides a multi-step workflow behind one name.

A pipeline diagram should make the workflow easier to reason about. If it hides mutable reads, external effects, or manual decisions behind generic task names, it is decorative.

Review one failed run from the artifact graph outward: what existed, what changed, what was retried, and what recovery remains safe?

The same 2015 paper names the shape a decorative DAG takes. It calls it a pipeline jungle. “As a special case of glue code, pipeline jungles often appear in data preparation. These can evolve organically, as new signals are identified and new information sources added incrementally. Without care, the resulting system for preparing data in an ML-friendly format may become a jungle of scrapes, joins, and sampling steps, often with intermediate files output.” Evolve, not grow: nobody designs a jungle. It arrives one reasonable addition at a time, each of them locally defensible, and the DAG that draws it looks busy and healthy the whole way.