Human Rights, Democratic Values, and the Rule of Law

Human-rights analysis asks whether what the AI is being used to do respects dignity, autonomy, equality and privacy. It asks the same of expression, association, due process, access to justice, and democratic institutions. Rights are not interchangeable weights that one score can trade off against each other. Each right is interfered with separately. Satisfying one of them says nothing about the others. So the analysis connects each right to a concrete interference, to a legal basis, to a legitimate aim, and then to necessity, proportionality, safeguards and remedy.

That chain is not a classroom exercise. A live government risk-scoring system has already lost to it. SyRI was the Dutch welfare-fraud risk-indication instrument. On 5 February 2020 the District Court of The Hague held that the legislation behind it failed the fair-balance test under Article 8(2) of the European Convention on Human Rights. The reason was that its application was insufficiently transparent and verifiable. The court declared the enabling provisions to have no binding effect.

Notice what that judgment settles and what it does not. It tested one interference: the one Article 8 protects. A system can be technically efficient and still fall outside the limits that rights place on public and private power. And a finding on one right leaves the equality and due-process questions exactly where they were.

A government prioritising welfare investigations by risk score is not a hypothetical. Neither is the objection to it. SyRI was the Dutch welfare-fraud risk-indication instrument. The law behind it was Section 65 of the SUWI Act and Chapter 5a of the SUWI Decree. On 5 February 2020 the District Court of The Hague struck those provisions down, in NJCM and others v. The State of the Netherlands. The legislation failed the Article 8(2) ECHR fair-balance test, because its application was insufficiently transparent and verifiable. Only five SyRI projects had been carried out since the legislation entered into force.

The operative words are worth reading in full: “For this reason, the court declares in this judgment that Section 65 SUWI Act and Chapter 5a SUWI Decree have no binding effect, being contrary to Article 8 paragraph 2 ECHR.”

Run the chain over that judgment and every stage is visible.

Right, interference, justification, safeguards, remedy: the analysis moves in that order. Jumping straight to safeguards is how the justification step goes missing. Identify the protected interest and the people who hold it. Describe how data, prediction, ranking, surveillance or automation affects that interest. Test legality, legitimate purpose, necessity and proportionality. Only then limit scope, access, retention, automation and discretion. Only then provide notice, contestability, correction, independent review and institutional accountability.

A police force can lose at the justification step alone, before anything else is reached. South Wales Police lost there. On 11 August 2020 the Court of Appeal of England and Wales allowed the appeal in R (Bridges) v Chief Constable of South Wales Police on Grounds 1, 3 and 5. The force's use of live automated facial recognition engaged Article 8(1) ECHR, and was not in accordance with the law for the purposes of Article 8(2). Nothing was weighed against operational benefit. The legal framework was simply deficient on two questions, the who and the where: “In relation to both of those questions too much discretion is currently left to individual police officers. It is not clear who can be placed on the watchlist nor is it clear that there are any criteria for determining where AFR can be deployed.”

The same judgment found two further failures. One was a breach of section 64(3)(b) and (c) of the Data Protection Act 2018. The other was a breach of the public sector equality duty in section 149 of the Equality Act 2010. Three separate legal tests, three separate failures, one deployment. Running the analysis once per system would have found one of them.

A utility assessment weighs benefits against costs. A human-rights assessment asks whether the interference is permissible at all. A principle review asks neither of those with any force. State v. Loomis is the case where those three answers come apart on the record.

A sentencing court may consider a proprietary COMPAS recidivism score without violating due process. The Wisconsin Supreme Court held that on 13 July 2016. The utility answer survives, and the tool stays in the room. But it survives only subject to limits that no cost-benefit review would generate. The score may not determine whether an offender is incarcerated, or the severity of the sentence. It may not be the determinative factor on community supervision. And every presentence report containing a COMPAS assessment must carry a written advisement listing five stated cautions. The first of them names the problem the utility answer cannot see: “The proprietary nature of COMPAS has been invoked to prevent disclosure of information relating to how factors are weighed or how risk scores are determined.” The Harvard Law Review titled its case comment for exactly that outcome: a warning required before use.

Now read the three modes against Loomis. A utility assessment is useful for resource allocation. It depends on contested valuations, it can aggregate rights violations away, and it is insufficient where rights impose limits. It would have scored COMPAS as an input and stopped. A human-rights assessment centres rights holders and duties. It requires legality and proportionality, includes remedy and oversight, and can reject efficient but impermissible uses. It is what produces limits of the Loomis kind — fencing off what the score may decide, rather than whether it may be consulted. An ethical principle review can address harms beyond legal rights, and it supports reflection under uncertainty. But it may lack binding authority, and it should complement law rather than replace it. A professional norm against opaque scoring would not have put a written advisement into every presentence report. A court did.

Rights holders first, remedy last. Name the individuals, groups, bystanders and communities affected. Trace data collection, inference, decision and institutional consequence. Assess legality, purpose, necessity, alternatives and proportionality. Limit scope, authority, retention, automation and downstream reuse. Ensure notice, reasons, correction, appeal, independent oversight and learning. The tests in the middle are legality, necessity and proportionality. Any one of them can stop a system being deployed at all, as Bridges stopped at the first of the three.

For the welfare-scoring scenario this lesson has been working with, that sequence is not the course's own design. It is law. The EU Artificial Intelligence Act — Regulation (EU) 2024/1689 — was adopted on 13 June 2024 and published in the Official Journal on 12 July 2024. Annex III, point 5(a) classifies as high-risk: “AI systems intended to be used by public authorities or on behalf of public authorities to evaluate the eligibility of natural persons for essential public assistance benefits and services, including healthcare services, as well as to grant, reduce, revoke, or reclaim such benefits and services”.

Article 27(1) then tells the deployer what to do about it. Deployers that are bodies governed by public law, and private entities providing public services, must assess the impact on fundamental rights before deployment. The assessment has to cover the categories of persons likely to be affected, the specific risks of harm, human oversight, and internal governance and complaint mechanisms. Persons affected, risks of harm, oversight, complaint route: almost the same list as the control above. Owed before the system runs, not after someone complains.

Rights language costs nothing on a policy page. It costs something the day a system that works is restricted because the interference cannot be justified. Australia's Robodebt automated income-averaging debt system is the priced version of that day.

It ran from July 2015 to November 2019 and was abandoned as unlawful. In orders made on 11 June 2021, in Prygodicz v Commonwealth of Australia (No 2), Murphy J recorded the Commonwealth's admission that it had no proper legal basis for such debts. At least $1.763 billion had been unlawfully asserted against approximately 433,000 people. About $751 million had been recovered from about 381,000 of them. The settlement was $112 million. The official summary accompanying the orders does not describe an administrative hiccup: “The proceeding has exposed a shameful chapter in the administration of the Commonwealth social security system and a massive failure of public administration.” Two years later, on 7 July 2023, the Royal Commission's final report, delivered by Commissioner Catherine Holmes, said it in plainer words: “Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals”.

Define when human-rights governance requires the provider to redesign, restrict, remedy or retire the system. Define it before deployment. Robodebt is what the last of those four costs when the first three were never done.