Production Contracts, Risk Tiers, and Ownership

“Predict equipment failure” sounds like a requirement, but it leaves the system undefined. How early must the warning arrive? Which failure types count? Who receives it, and what maintenance capacity exists?

A production contract turns a modeling aspiration into a testable operating agreement. It records the population, timing, output semantics, allowed actions, service limits, review rules, and recovery obligations.

Every clause on that list has a documented failure behind it. This lesson is built on those rather than on hypotheticals. A US safety board reconstructing a collision second by second. A European regulation that writes oversight, and its price, into statute. A court in The Hague voiding a deployed scoring system. An English judgment on software whose output was treated as fact. A medical-informatics review that counted how often clinicians click past the alert. A US regulator that now authorises the change process instead of only the frozen model.

One automated workflow has been investigated to the second, and it shows exactly which clauses a contract has to contain. On 18 March 2018, in Tempe, Arizona, a Volvo XC90 running Uber ATG's developmental automated driving system collided with a pedestrian. The National Transportation Safety Board reconstructed the sequence and adopted its highway accident report on 19 November 2019. The Congressional Research Service summarised the same findings for Congress in 2020. Read as a contract, almost nothing that failed was the model.

A suggestion that a user can ignore is not governed like an automated denial. The same model score can be low risk in a drafting interface and high risk when it triggers an irreversible action.

Risk tiering therefore belongs to the workflow, not only the model family. A legislature has drawn that line in exactly those terms. Regulation (EU) 2024/1689 tiers by the role the system plays in a decision: “By derogation from paragraph 2, an AI system referred to in Annex III shall not be considered to be high-risk where it does not pose a significant risk of harm to the health, safety or fundamental rights of natural persons, including by not materially influencing the outcome of decision making.” (Article 6(3).)

Article 6(3) then enumerates four qualifying situations. Performing a narrow procedural task. Improving the result of a previously completed human activity. Detecting deviations from prior decision-making patterns without replacing the human assessment. Performing a preparatory task. Each of the four is a statement about the decision, not about the architecture. The article also closes the door it opens: an Annex III system that performs profiling of natural persons is always high-risk. And Article 6(4) requires a provider claiming the derogation to document its assessment before the system is placed on the market. That written, pre-market assessment is the artefact a risk tier should always produce. A recorded call about the system's role in a decision, made before release, available to be checked afterwards.

A complete contract names what the system must do and how failure is contained. Scope fixes the population, the use case, the exclusions and the decision moment. Semantics fixes the meaning of outputs, thresholds, confidence and abstention. Operations fixes latency, throughput, availability, freshness and capacity limits. Control fixes review, escalation, overrides, appeals, fallback and rollback. Accountability fixes owners, approvers, on-call roles, evidence retention and retirement authority.

The last two clauses are the ones teams treat as paperwork. A court has already treated them as a precondition for operating at all. On 5 February 2020 the District Court of The Hague decided the challenge to the Dutch SyRI welfare-fraud legislation, brought by NJCM and others against the State of the Netherlands. The court held that the legislation failed the necessity test in Article 8(2) ECHR and did not strike the required fair balance. It said why in a sentence about inspectability: “The court holds that the legislation pertaining to the application of SyRI is insufficiently transparent and verifiable.” It declared Section 65 of the SUWI Act and Chapter 5a of the SUWI Decree to have no binding effect.

A scoring system was stopped not because its accuracy was disproved but because nobody outside it could examine how it scored. Transparency and verifiability are Control and Accountability clauses. Here they were legal preconditions, not good practice.

Three classes of requirement take three kinds of evidence. The functional contract asks what transformation or decision the system performs: inputs and outputs, valid and invalid requests, output semantics — rank maintenance jobs by expected urgency. The reliability contract asks how consistently the service performs under expected load: latency, availability, dependency and capacity failures, recovery objectives — complete the nightly batch before 05:00. The safety and governance contract asks which outcomes are unacceptable and who controls them: prohibited actions, required review or abstention, appeal and audit paths — never auto-close a high-risk medical alert. Mixing these categories produces vague acceptance criteria.

The third column is the one whose absence has the longest documented tail. On 16 December 2019 the High Court decided the Horizon issues in Bates & Ors v Post Office Ltd. Mr Justice Fraser found: “In summary terms only, Legacy Horizon was not remotely robust.” Even the Post Office's own expert accepted strong evidence of at least 12 bugs causing lasting discrepancies in branch accounts.

The scale of the missing governance clause was established later. Sir Wyn Williams' statutory Post Office Horizon IT Inquiry published Volume 1 of its final report on 8 July 2025. It found it likely that approximately 1,000 persons were prosecuted and convicted across the United Kingdom on Horizon evidence. At least 59 people contemplated suicide and ten attempted it. Employees of Fujitsu and the Post Office knew the system was capable of error while the Post Office maintained the fiction that its data was always accurate. The functional and reliability contracts can be satisfied while that is happening. There was no appeal path and no audit of the system's errors, so a court and then a statutory inquiry had to become the appeal path the contract never contained.

Convert each clause into evidence that can pass or fail. Define the scoring moment: state when the decision is made and what information is available then. Name the output semantics: rank, class, estimate, interval, or proposal. Quantify the operating limits: freshness, latency, throughput, capacity, recovery targets. Define the unacceptable outcomes: prohibited actions, protected slices, escalation conditions. Then assign decision rights over release, traffic pausing, threshold changes and retirement.

That last step is usually written as a name on a wiki page. Article 26(2) of Regulation (EU) 2024/1689 writes it as a duty, and specifies what the named person must actually possess: “Deployers shall assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support.” Competence, training, authority, support. Four attributes, each of which can be evidenced or found missing at a release review. That is more than a name in a box can be.

Getting the tier wrong carries a price in the same instrument. Article 99(3)-(4) sets administrative fines of up to EUR 35,000,000 or 7% of total worldwide annual turnover for breaching the Article 5 prohibitions, and up to EUR 15,000,000 or 3% for breaching the operator obligations in Articles 16, 22, 23, 24 and 26. The classification in the mechanics section and the ownership assignment in this one are the same decision seen from two ends. Both are now enforceable.

A requirement that never blocks promotion is documentation, not a gate. Each critical clause should produce a test, approval, or operational control with a named owner.

The contract will evolve, but changes must be versioned. Otherwise teams can improve the model while quietly changing what “success” means. A regulator has now built exactly that instrument. On 4 December 2024 the U.S. Food and Drug Administration published its final guidance on predetermined change control plans for artificial intelligence-enabled device software functions. The summary of the notice states the shape of the artefact: “This guidance recommends that a PCCP describe the planned AI-DSF modifications, the associated methodology to develop, validate, and implement those modifications, and an assessment of the impact of those modifications.”

Notice what is being authorised. Not the model as frozen. The planned modifications, the methodology that develops, validates and implements them, and an assessment of their impact — reviewed as part of the marketing submission, so that changes inside the plan need no new submission. The boundary of the plan is the gate. Everything inside it is pre-agreed, and everything outside it falls back to a fresh submission. A production contract that cannot say where that boundary lies has not versioned its definition of success. It has only recorded one.