Capstone: Design, Evaluate, and Defend an End-to-End NLP System

Each stakeholder supplies different requirements and failure evidence. The MyCity case shows why one group alone is not enough. The business owners who asked got a fluent answer and no signal. The policy owners had a correct rule on file that the bot contradicted. The reading that matters legally came from outside the product team entirely: advising a landlord to refuse Section 8 vouchers is advice to commit illegal source-of-income discrimination. Legal Services NYC and the NYC Hospitality Alliance said so, reacting in a newsroom investigation. The Office of Technology and Innovation, meanwhile, held a metric that pointed the other way — thousands of people served with what it called timely, accurate answers.

Map each group below to the specific evidence it owns, and to the failure only it can see.

Do not assign one model the vague job of understanding customer messages. A single free-text path from question to answer has no place where a policy claim can be checked against a source. It has no place where low confidence can turn into a question. It has no place where a prohibited action can be refused. So when such a system is wrong, it is wrong fluently and repeatedly — as it was for all ten Markup staffers.

Each step below is a separate contract with its own output type, its own evidence requirement, and its own failure control.

The capstone should defend a choice rather than assume the most complex design is best.

The “Higher verification burden” listed against the LLM-centered assistant has been measured, in a domain where a wrong citation is disqualifying. Dahl and colleagues profiled how often general-purpose models invent answers to verifiable questions about federal cases. They reported the result in the Journal of Legal Analysis in 2024: “we show that LLMs hallucinate at least 58% of the time”. ChatGPT 4 sat at that floor. Llama 2 ran as high as 88%.

That is the case for retrieval. The case against treating retrieval as the fix came from the same group. They ran the first preregistered evaluation of the retrieval-augmented commercial tools built for exactly this problem, and published it in the Journal of Empirical Legal Studies in 2025: “While hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time.” The vendors had advertised “hallucination-free” citations.

Read the two numbers together before choosing a row. Grounding a generative component in authoritative sources moved the error rate from at least 58% to between 17% and 33%. It did not move it to zero. The products measured were built by document companies over their own authoritative corpora, and a transit assistant drafting refund and penalty replies over policy PDFs is not in a better position.

That residual is the whole argument for the hybrid governed row: deterministic gates on the consequential intents, claim-level attribution to a retrieved passage with a policy version, and abstention when support is missing. Those controls assume the generator will still be wrong sometimes. The alternative assumes retrieval has fixed it.

The plan should separate what was convenient to build from what the team claims is ready to run in production.

Eight languages are not eight equal columns in that plan, and the inequality starts before any model runs. A 2023 NeurIPS paper measured it: “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.” Even character- and byte-level models differed by over 4 times. A separate team audited OpenAI's API across 22 typologically diverse languages in 2023. Speakers of many supported languages are overcharged while obtaining poorer results, and they tend to come from regions where the APIs are least affordable.

For the transit system this converts three fairness intentions into engineering quantities. The same complaint, in two of the eight languages, can consume up to 15 times the tokens. The per-message cost differs by language. So does the latency, and so does the point at which a long thread is truncated out of the context window. A corpus and evaluation plan that does not record token length per language cannot tell later whether a language performed worse because the model is weaker on it, or because half its case history was silently cut.

No single metric can certify the complete system.

Step 3 is where the transit authority's speech-transcript channel stops being an implementation detail. Five commercial speech recognizers — Amazon, Apple, Google, IBM and Microsoft — were tested on 19.8 hours of interviews with 42 white and 73 black speakers across five US cities. Koenecke and colleagues reported the result in PNAS in 2020: “We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers.” The gap persisted on identical phrases, and it was traced to the acoustic models. It was not an artefact of what people happened to say.

A separate audit ran popular English ASR services against more than 2,700 speakers born in 171 countries, drawn from the Speech Accent Archive. Service performance had a statistically significant relationship to the speaker's country of birth. That held for every service tested.

A single headline word error rate averages 0.35 and 0.19 into one acceptable-looking figure. Every component downstream inherits the difference: intent routing on a corrupted transcript, entity extraction that loses a route number, a summary that drops the accessibility request. Report the slice or do not claim the channel works.

The team should know which evidence can block launch and which limitations require the scope to be cut. For part of this the team no longer writes the list itself.

Article 14 of Regulation (EU) 2024/1689, the Artificial Intelligence Act, sets out human oversight for high-risk systems as named components. Overseers must be enabled to understand the system's relevant capacities and limitations and to duly monitor its operation. They must remain aware of “the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)” — the statute names the failure the mammography study measured. Under Article 14(4)(d) they must be able “to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system;”. The article also requires an interruption path: a “stop” button or a similar procedure that allows the system to come to a halt in a safe state.

Article 14(5) goes further for the biometric identification listed at Annex III point 1(a). No action or decision may be taken on the basis of the identification unless it has been “separately verified and confirmed by at least two natural persons with the necessary competence, training and authority”. That requirement does not apply to certain law enforcement, migration, border control and asylum uses where Union or national law considers it disproportionate.

A RAND report summarises the same obligations as “human oversight measures to enable deployers to not use the system or to disregard it, override it, reverse its outputs, or interrupt it and bring it to a halt in a safe manner”. It records 2 August 2026 as the application date for most of the Act. The Chapter III high-risk requirements, Article 14 among them, were deferred by the Digital Omnibus on AI: to 2 December 2027 for stand-alone Annex III systems, and to 2 August 2028 for AI embedded in regulated products under Annex I.

Whether this transit assistant falls inside Annex III is itself a finding the design review must record rather than assume. Either way the article supplies a gate checklist a team can be held to. A named overseer. Evidence of the system's limits in front of them. A documented override that actually changes the outcome. A stop procedure that has been exercised. Each with a test, not a paragraph.

A credible capstone is a decision package, not a polished architecture diagram alone. Each item below should be readable by someone looking for the thing that will fail.

The design may conclude that generation should remain agent-assist only, that rare safety cases need deterministic routing, or that two languages lack enough evaluation evidence for launch. Such constraints are evidence of sound engineering rather than failed ambition.

One company published both ends of that argument using its own numbers. In its first month, Klarna's AI assistant did the equivalent work of 700 full-time agents, across 23 markets and more than 35 languages. Resolution time fell from 11 minutes to under 2. Repeat inquiries fell 25%. The company put the profit improvement for 2024 at an estimated $40 million. Its announcement on 27 February 2024 led with the volume: “The AI assistant has had 2.3 million conversations, two-thirds of Klarna’s customer service chats”. Fifteen months later Klarna was recruiting human agents again. CEO Sebastian Siemiatkowski told Fortune on 9 May 2025 that cost had been “a too predominant evaluation factor” and that “what you end up having is lower quality”.

Read the first list again knowing the second. Volume, handling time and repeat contacts all moved the right way, and every one of them is a throughput measure. The quality the operator eventually cared about was not among them, so nothing in the launch dashboard could report its decline. That is the case for the outcome evidence in the data plan, and for a bounded launch. Not caution for its own sake: it is the only way the thing that failed here becomes visible before fifteen months have passed.

A strong final defense distinguishes what the system knows, what it estimates, what it retrieves, what it is permitted to do, and what remains a human or policy decision.

The habit this lesson resists has a name too. A small set of influential benchmarks “operate as stand-ins”. They stand in for “a range of anointed common problems that are frequently framed as foundational milestones on the path towards flexible and generalizable AI systems”. And “State-of-the-art performance on these benchmarks is widely understood as indicative of progress towards these long-term goals.” Raji and colleagues wrote that in 2021, and their position paper asks whether that inference is valid at all. A capstone states what its evidence covers, and what it does not. It is answering the same objection at product scale.