Robustness, Bias, Privacy, and Domain Shift

A municipal help desk launches a multilingual classifier with strong held-out accuracy. Within three months, its most costly failures come from inputs that the original test set barely represented.

Each of the five pressures below has a measured, published instance in the sections that follow. Not one of them requires anybody to touch the model.

The remedy depends on what changed. So “drift” should not be used as one diagnosis.

Two of the five rows below have been measured, and the figures are worth carrying into the diagram. Surface variation is not free. Five commercial speech-recognition systems — Amazon, Apple, Google, IBM and Microsoft — were run over 19.8 hours of interview audio from 42 white and 73 black speakers across five US cities. Koenecke and colleagues reported the result in PNAS on 23 March 2020: “We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers.” A 2024 study of nurse-patient speech in home healthcare, in JAMIA Open, reproduced the gap on AWS Transcribe: a median WER of 33% for white patients against 50% for black patients (P = 0.016). The help desk that swaps web forms for voice inherits that penalty on the day the channel changes. Its own accuracy report will still be about typed text.

The adversarial row has a published vocabulary too. NIST's adversarial machine learning taxonomy, published in March 2025 with the U.S. AI Safety Institute and the U.K. AI Security Institute, separates two kinds of prompt injection. Direct injection is made by the system's own user, through query access. Indirect injection is delivered through resource control by a third party — a retrieved document, an email, a web page. OWASP places the same split at LLM01 in its Top 10 for LLM Applications 2025. The consequence is the part teams miss, and §3.4 states it plainly: “In fact, in many cases, it is the primary user of the model who is harmed by the compromise of the integrity, availability, or privacy of the GenAI system through an indirect prompt injection attack.” The attacker need never touch your input box.

One aggregate robustness score hides whether meaning should stay stable or change.

The taxonomy below is not a house convention. Ribeiro and colleagues introduced CheckList at ACL 2020 with three test types: Minimum Functionality (MFT), Invariance (INV) and Directional Expectation (DIR). The middle two are the first two columns below, under the names their authors gave them.

The worked example is the argument. A commercial sentiment model, already shipped, failed 76.4% of templated negation cases generated from “I {NEGATION} {POS_VERB} the {THING}.” It failed 20.8% of NER invariance cases, where Chicago was swapped for Dallas, and 34.6% of directional cases. A model can hold an aggregate accuracy figure that satisfies a review board and still get three-quarters of a single grammatical construction wrong. The construction is a sliver of the held-out set and a large share of real complaints. The skill also transfers: “In another user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.”

Historical text can encode unequal treatment, stereotypes, and gaps in who was documented. Annotation guidelines can reproduce institutional assumptions, while product thresholds can distribute errors unevenly even when the model is unchanged.

Removing a protected attribute does not remove correlated language, geography, occupation, or access patterns; fairness analysis therefore needs a workflow-aware account of where disparities arise.

The annotation pathway has been measured. A classifier trained on the DWMW17 corpus falsely flagged 46.3% of non-offensive African American English tweets as “offensive”, against 9.0% for white-aligned tweets. On FDCL18 the same comparison was 26.0% against 4.5%. Sap and colleagues reported that at ACL 2019, and their abstract names the mechanism: “models trained on these corpora acquire and propagate these biases, such that AAE tweets and tweets by self-identified African Americans are up to two times more likely to be labelled as offensive compared to others”. Davidson and colleagues found the same shape independently across five annotated Twitter datasets, with black-aligned tweets classified offensive at 0.173 against 0.065 for white-aligned tweets, a ratio of 2.65. The intervention that worked in the ACL 2019 study acted on nobody's architecture: priming annotators to consider dialect significantly reduced their offensiveness ratings. The defect was upstream of the loss function.

Another of those pathways sits inside the representation itself. The finding, reported by Bolukbasi and colleagues in 2016, was blunt: “even word embeddings trained on Google News articles exhibit female/male gender stereotypes to a disturbing extent”. The bias had a geometry: “gender bias is first shown to be captured by a direction in the word embedding”. That is what made it measurable. It is also what kept it invisible to anyone auditing the corpus for a protected column.

The review should connect a failure mechanism to a test, control, owner, and response.

For a high-risk deployment in the EU, steps 3 and 5 are not left to the team's own judgement. Article 15 of the EU Artificial Intelligence Act makes robustness a legal requirement, and Article 15(5) names the attack classes by their technical names: “The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent, detect, respond to, resolve and control for attacks trying to manipulate the training data set (data poisoning), or pre-trained components used in training (model poisoning), inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion), confidentiality attacks or model flaws.” The Regulation was published in the Official Journal on 12 July 2024. It came into force on 1 August 2024 and applies from 2 August 2026.

Two neighbouring paragraphs bind the rest of this lesson to the same text. Article 15(3) requires declared accuracy levels in the instructions of use. Article 15(4) requires that feedback loops from continued learning be addressed. A statutory list is a floor for step 3, not a substitute for the domain experts and incident history it asks you to add.

A one-time audit cannot anticipate new slang, policy changes, attacks, or data sources; keep high-value tests in regression suites and refresh them from production incidents without leaking protected evaluation sets into routine tuning.

Google Flu Trends is the documented case of a text-driven system decaying while nobody touched the model. GFT ran high in 100 of the 108 weeks from 21 August 2011 to 1 September 2013. It overshot the 2011-2012 season by more than 50%, and by February 2013 it was predicting more than double the CDC's proportion of doctor visits for influenza-like illness. Lazer and colleagues reported that in Science on 14 March 2014. Part of the cause was Google's own product: suggested search terms arrived in June 2011 and symptom-based diagnoses in February 2012, changing the search behaviour the model was reading. The authors call that “blue team” dynamics, and distinguish it from the other direction of pressure: “Red team dynamics occur when research subjects (in this case Web searchers) attempt to manipulate the data-generating process to meet their own goals, such as economic or political gain.” Olson and colleagues had already reported independently, in PLOS Computational Biology, that GFT missed the first wave of the 2009 A/H1N1 pandemic and greatly overestimated the 2012-13 A/H3N2 epidemic. One hundred consecutive-season weeks of error is what an unmonitored deployment looks like from the inside. Nothing crashed.

Monitor input characteristics, output rates, confidence, abstention, user corrections, privacy incidents, and delayed task outcomes by meaningful slices; a statistical alert should send somebody to investigate, not make an automatic claim of harm or model failure.

Choose a concrete system, such as complaint routing, clinical extraction, or moderation. List at least eight risks across representation, privacy, security, bias, shift, and workflow. Use the cases in this lesson as templates rather than as reading: an ASR channel with a 0.35 against 0.19 word error rate gap, an annotation pipeline flagging 46.3% against 9.0%, a retention rule that a stipulated order priced at $25 million, an indirect injection arriving in a retrieved document, and a hundred high weeks out of 108 that nobody noticed.

For each risk, write one observable indicator, one preventive control, one detection test, one fallback, and one accountable owner. Name the detection test by type — Minimum Functionality, Invariance or Directional Expectation, slice or adversarial — so that the register says what a passing result would mean. Mark any risk that cannot be measured with the data currently collected.