Bias, Privacy, Security, and Safety

A risk review should follow the system lifecycle. Focusing only on training data or model architecture misses many ways harm can arise.

Systemic bias arises from institutions, policies, and historical patterns that shape opportunities and records. Statistical or computational bias arises from sampling, measurement, modeling, or evaluation choices. Human cognitive bias influences framing, annotation, interpretation, and use.

These sources interact. A dataset can accurately reflect an unfair process, and a model can reproduce that process without a coding bug.

A dated example makes the mechanism concrete. Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan published in Science on 25 October 2019. They dissected a commercial risk-prediction algorithm of a kind applied, by industry estimates, to roughly 200 million people in the United States each year. It ranked patients for extra care by predicting future health care costs. That is a convenient stand-in for future health needs. It specifically excluded race as an input. Less is spent on Black patients at the same level of illness, so patients holding the same score were not equally sick. At the 97th percentile, where enrolment in the program is automatic, Black patients carried 26.3% more chronic illnesses than White patients. That is 4.8 against 3.8 distinct conditions. Changing that one label would have raised the share of Black patients auto-identified for additional help from 17.7% to 46.5%. The paper names the failure: “the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.”

The lenses overlap, but each directs attention to a different kind of failure and remedy.

Two of the sub-points below have published numbers behind them. On privacy: Latanya Sweeney estimated from 1990 census data that 87% of the United States population was uniquely identifiable. Those 216 million people out of 248 million carried a combination of {5-digit ZIP code, gender, date of birth} likely to be unique to them. Her estimate appeared in a Carnegie Mellon working paper released in 2000. Philippe Golle recomputed it on 2000 census data and reported a lower but still decisive 63%. He presented that on 30 October 2006. None of those three fields identifies anyone on its own; in combination they usually do. That is what “anonymization is not automatic” means in practice. It is why a privacy review has to ask what the retained columns imply together, not only which columns were deleted.

On security: the supply-chain point has a dated instance inside the machine-learning stack itself. On 31 December 2022 the PyTorch project disclosed a compromised package. Anyone who had installed PyTorch-nightly on Linux via pip between 25 and 30 December 2022 received a malicious package named torchtriton. It came from the public Python Package Index instead of the legitimate one on PyTorch’s own index, because “the PyPI index takes precedence”. The planted binary read /etc/passwd and /etc/hosts, the environment variables, every file in the user’s .ssh directory and the first 1,000 files in the user’s home directory. It sent them out through encrypted DNS queries. Stable-release users were unaffected, the dependency was renamed pytorch-triton, and a placeholder was registered on PyPI. No model was attacked, and no model-level test would have found it.

Consider a voice assistant used to schedule public services. A single deployment can create several distinct problems.

The first mitigation should address the actual failure mechanism rather than the most fashionable concern.

Dataset documentation can record motivation, composition, collection, recommended uses, and limitations. Model documentation can state intended use, evaluation conditions, subgroup performance, and known failure modes.

Documents do not guarantee responsible behavior, and self-reporting can be incomplete. They are still valuable because they create a concrete object for review, comparison, incident analysis, and handoff between teams.

Both kinds of document have names and published proposals behind them. “Datasheets for Datasets” — Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III and Kate Crawford, drafted on arXiv on 23 March 2018 and published in Communications of the ACM 64(12) in December 2021 — borrows the idea from electronics, where “every component, no matter how simple or complex, is accompanied with a datasheet that describes its operating characteristics, test results, recommended uses, and other information”. Its questions are grouped into motivation, composition, collection process, preprocessing/cleaning/labeling, uses, distribution, and maintenance. “Model Cards for Model Reporting” — Margaret Mitchell and colleagues, at the Conference on Fairness, Accountability, and Transparency on 29–31 January 2019 — does the same for a trained model, and its central demand is disaggregated evaluation: performance reported “across different cultural, demographic, or phenotypic groups” and across intersectional groups rather than as a single number. Neither is a legal requirement, and neither is audited by default, which is exactly why who fills them in and who checks them is part of the accountability question.