Capstone: Design and Defend a Speech & Audio AI System

That last boundary has a consequence people find surprising. A capstone that ends by recommending against deployment passes. When the evidence cannot support the proposed use, saying so is the finding, provided the reasoning for the judgement is on the page. What fails is a verdict the evidence does not carry, in either direction.

Nor is the capstone won by selecting the largest model. A defensible design begins with what operators actually need and where the law stops you. For this brief the law has already spoken. The EU AI Act puts a whole class of emergency systems in the high-risk category, at Annex III, point 5(d): “AI systems intended to evaluate and classify emergency calls by natural persons or to be used to dispatch, or to establish priority in the dispatching of, emergency first response services, including by police, firefighters and medical aid, as well as of emergency healthcare patient triage systems.” That is the capstone brief, line for line. Under Article 6(2) the classification attaches to the use, not to the score. No accuracy figure moves the system out of it.

So the contract is written against a boundary that already exists. It preserves the raw evidence and every transformed version of it. It keeps provisional output separate from action. It states in advance when the automation has to abstain. Those commitments are the operating contract, and the contract is written first for a reason: the capstone is graded on whether it still holds everything the lifecycle plan at the far end will depend on. Break that link and no accuracy figure on names and addresses, however clean, can carry the deployment decision.

Here is the brief the contract has to be written against. Design an assistance system for emergency call centers offering live transcription, language routing, speaker attribution, key-event detection, operator prompts, and optional speech translation. The calls arriving at it carry noise, distress, overlap, children, code-switching, medical terms, addresses, and legally sensitive evidence.

The trap is sprung by the first of the four decisions above, and the corpus imbalance behind it is public and quantified. OpenAI trained Whisper on 680,000 hours of audio. Of those, 117,000 hours cover the 96 non-English languages combined. The 2023 paper also fits a cost to that imbalance: “We find a strong squared correlation coefficient of 0.83 between the log of the word error rate and the log of the amount of training data per language,” it reports. On that fit, word error rate halves for every 16-fold increase in a language's training data. Rarity is not a vague disadvantage. It is a slope you can read off. And accuracy figures for names and addresses still look clean while it operates, because the corpus goes quiet about precisely the callers it never contained. A clean report is what silence looks like from the outside.

The second decision has already been through a court. In United States v. Cruz-Zamora, decided in 2018, the drugs seized from the defendant's car were suppressed. Consent to search had been obtained through Google Translate. Interpreters testified that the officer's rendering was not how a Spanish speaker asks to search a car, and that the defendant said he did not understand on nine separate occasions during the stop. “For these reasons the court finds that the good-faith exception does not apply as it is not reasonable for an officer to use and rely on Google Translate to obtain consent to a warrantless search, especially when an officer has other options for more reliable translations,” wrote Judge Carlos Murguia. The brief lists optional speech translation and legally sensitive evidence in the same sentence. That is what it costs when the translation step changes what was asked.

Three answers to that question exist, and choosing between them is the design.

Operator-assistance design surfaces uncertain evidence and keeps the decisions with trained personnel. It is the commitment that builds the confirmation step in and pays for it in operator time. Automated decision design takes the step out, and with it takes on the burden of showing that unaided output is fit to act on. Non-deployment or redesign holds that the evidence does not support the use as written. The three are related, but they are not degrees of one another. Each has to be argued from different evidence, and a number that establishes one of them establishes nothing about the other two.

That claim has been tested on an emergency line. A trial in Copenhagen randomised 5,242 emergency calls to the city's EMS between 1 September 2018 and 31 December 2019, and reported the result in JAMA Network Open in 2021. On confirmed out-of-hospital cardiac arrest, the machine-learning model alone beat the dispatchers on sensitivity: 85.0% against 77.5%. Its positive predictive value was far worse, 17.8% against 55.8%. Then the alerts were put in front of the dispatchers. Recognition with them (93.1%) did not differ significantly from recognition without them (90.5%, P = .15). “This randomized clinical trial did not find any significant improvement in dispatchers' ability to recognize cardiac arrest when supported by machine learning even though artificial intelligence did surpass human recognition,” the paper concludes. The model won the accuracy comparison. The deployment gained nothing measurable. That is the whole argument for keeping the three designs apart, in one trial: evidence for automated decision design was in hand, and it did not transfer to the assisted system built out of it.

Any of the three you land on is argued in four documents, and those documents are what the capstone has to produce. They divide the argument rather than restate it: two describe the system, two describe what happens around it once it is running.

The division that matters most is between the evaluation suite and the operator study, because they do not measure in the same units. The suite counts what the system got wrong on recordings. The study counts what a person then had to do about it, and the Copenhagen trial is the demonstration that the first does not predict the second. A system that misses one address in a thousand and a system that misses one in a thousand while flagging its own doubt score the same in the suite and produce very different call floors.

For the third of the four decisions, the suite has a public benchmark to be built against. The third DIHARD challenge evaluated speaker diarization across 11 domains, on 499 valid submissions from 30 organisations (21 teams). Its 2021 results section reports: “For two of the remaining domains, clinical interviews and conversational telephone speech, DER generally ranges from 10% to 20% for track 1 and 15% to 25% for track 2, indicating that even in the two-party case there remain substantial challenges, particularly in the presence of speakers with unusual characteristics (i.e., children in the clinical interviews).” Meeting speech, web video and restaurant audio came in at a median track-1 diarization error rate of 35–45%. Telephone speech is the easy end of that table. Two-party calls are the easy case within it. And children's voices are already named as the hard part — which is the emergency brief, described by somebody who was not writing about emergencies.

The grading rests on four terms, and each has already appeared in this lesson under a plainer name. The address in the emergency brief is a critical entity. The confirmation step, and whether an operator under load actually uses it, is a human-factors question — that is the thing the Copenhagen trial measured and found unmoved. Non-deployment, written down before the evidence arrives rather than after, is a stop rule. The whole argument bound together is the decision dossier. Each of the four names evidence somebody has to produce and a person who has to sign for it.

The dossier is finished when another team can read it and fail to show that the design was trained without the rare languages and distressed voices it will actually hear. That test is deliberately hostile, and it is the one the first of the four decisions has earned. The team running it has 117,000 hours out of 680,000, a 0.83 log-log correlation, and 1.4% of transcriptions with invented sentences to run the test with.

Working toward it, record three things and keep them on the same page: what defining the operating contract assumes, one counterexample that would break the assumption, and what planning the lifecycle then requires because of it. The system map shows every signal, model, state, tool, person, and storage boundary. The risk register covers acoustic, linguistic, identity, safety, privacy, and operational failure. The acceptance gates state thresholds, uncertainty, pilot scope, stop rules, and independent review. The defence explains why release, limited pilot, redesign, or rejection follows from the evidence.

The headline number is critical-entity and address accuracy measured together with confirmation behavior. The failure it exists to catch is the one in the title: an address heard wrongly and confirmed anyway, on one of those 240 million calls. Taken alone, that number still says nothing about which callers were left out of the training data, because those callers are not in it to be counted.

What that omission is worth has been measured on commercial systems. Five commercial ASR systems — Amazon, Apple, Google, IBM and Microsoft — were tested on 19.8 hours of interviews with 42 white and 73 black speakers across five US cities. “We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers,” write Koenecke and nine co-authors in PNAS in 2020. All five, not one bad vendor. And the gap traced to the acoustic models rather than to what was being said, because it persisted on identical phrases. Same words, same task, error rate nearly doubled by who was speaking. A single pooled accuracy figure for such a system is the average of those two numbers and reports neither.

So report what the number counts, and then report the rest of it — which callers it was measured on, how uncertain it is, and what the call quality was like — alongside the evidence below.