Automated Decisions, Profiling, and Individual Rights

Michigan's unemployment agency ran fraud determinations through a system called MiDAS. For nearly two years the software decided alone. Then people were added.

The litigation record says what that human step was worth. This is the 2018 opinion in Cahoo v. SAS Institute, reciting the amended complaint: “In the years material to this suit, the UIA largely did not employ any human review in making these automated determinations. Between October 2013 and August 2015, MiDAS made all fraud determinations. After August 2015, the UIA continued to use MiDAS, but made use of agency personnel to exercise some oversight in the process. However, those personnel exercised no more discretion than their MiDAS counterpart.”

Read the second phase twice. People were added. The determinations did not change.

What that cost is countable. The Michigan Auditor General reviewed 22,427 robo-adjudications. Over 93% did not involve fraud at all.

Profiling produces a score. The score exerts some degree of influence: advisory, prioritising, presumptive, or effectively determinative. That influence lands on a person as a legal, economic, employment, educational, health or access effect. A human step sits somewhere in the chain. It carries authority, competence, time, evidence, independence and override — or it carries none of them. Rights attach at the far end: notice, access, meaningful information, intervention, contestation, correction and remedy.

That last box is not a wish list. A court has given it content. An applicant was refused a €10-per-month mobile contract on an automated creditworthiness assessment, and asked what had been done to him. The Court of Justice answered in February 2025, in Case C-203/22. Article 15(1)(h) GDPR entitles him to an explanation of the procedure and principles actually applied. “The controller must describe the procedure and principles actually applied in such a way that the data subject can understand which of his or her personal data have been used, and how they have been used, in the automated decision-making.”

Disclosing the algorithm is not a sufficient explanation. Where the controller claims trade secrets, it hands the information to the supervisory authority or the court, which balances the interests. The right is to understand the decision, not to be handed the model.

The conditions on the human step are not this course's invention. Europe's data-protection regulators wrote them down. Their guidelines on automated individual decision-making and profiling, adopted in October 2017 and revised in February 2018, say it in two sentences: “To qualify as human involvement, the controller must ensure that any oversight of the decision is meaningful, rather than just a token gesture. It should be carried out by someone who has the authority and competence to change the decision.” The same guidelines say a controller “cannot avoid the Article 22 provisions by fabricating human involvement”, and that oversight has to consider all the relevant data. The European Data Protection Board endorsed them on 25 May 2018.

Now read the Michigan recital against that. Personnel who exercise no more discretion than the software have neither the authority nor the competence the guidelines ask for. And adding them after August 2015 is close to the thing the text warns about by name.

Every one of those conditions costs money. Authority, competence, time, information, a genuine ability to change the result. In practice that means review time, headcount, and a manager who accepts overrides.

Under GDPR Article 22 and related guidance, particular conditions apply to decisions based solely on automated processing that produce legal or similarly significant effects. Rights and obligations depend on context, lawful basis, safeguards, transparency and jurisdiction. Get qualified legal advice.

All three arrangements put a person somewhere near the decision. Nominal human review satisfies a workflow diagram. A person appears, may rubber-stamp under time or incentive pressure, often lacks independent evidence, and is not enough for meaningful oversight. Meaningful human involvement means someone can understand, question and change the result. That takes time, authority, relevant evidence and training, an override that is not punished mechanically, and monitoring in practice. A solely automated decision is one whose outcome is settled without meaningful human intervention. It may trigger specific legal protections, needs careful scope analysis, and can include systems framed as recommendations.

A court has run that test on a documented review step. Uber deactivated three drivers' accounts and said real human involvement had come first. Its evidence was exhibits 14, 15 and 16. After an ERAF fraud signal, one, two or three members of the EMEA Operational Risk team working from Kraków wrote a note.

On 4 April 2023 the Amsterdam Court of Appeal said what those exhibits showed: “maar ze tonen geenszins aan dat die handeling veel meer is geweest dan een louter symbolische handeling en dat die medewerkers daarbij alle relevante gegevens in hun analyse hebben betrokken, zoals de EDPB richtsnoeren voorschrijven.” They in no way show that the act was much more than a purely symbolic one, or that the reviewers took all the relevant data into their analysis, as the EDPB guidelines require.

The drivers were never heard. Uber never stated the reviewers' qualifications or level of knowledge. That evidence lies largely inside Uber, so the court held that Uber bore a heightened duty to substantiate its denial — verzwaarde motiveringsplicht. It quashed two points of the district court's final order, one of them in so far as it had rejected the drivers' Article 15(1)(h) information requests. It upheld the rest. And it ordered Uber to grant those requests on pain of €4,000 a day.

The line between the second column and the third is now statutory in one jurisdiction. Section 80 of the UK Data (Use and Access) Act 2025 replaces Article 22 of the UK GDPR with Articles 22A-22D. The Act received Royal Assent on 19 June 2025. It gives UK statute its first definition: “a decision is based solely on automated processing if there is no meaningful human involvement in the taking of the decision”. Article 22C requires safeguards enabling information, representations, human intervention and contestation. The argument about what “solely” means now has an answer in the text.

Watch the reviewers before you read the policy. The null hypothesis is already on the record. Across 41 policies prescribing human oversight of government algorithms, Green found that “people are unable to perform the desired oversight functions”. Assume that until the observation says otherwise.

The sequence runs from mapping the decision to monitoring override quality, and it can stop at any of the five steps. One, map the decision: profiling, score, recommendation, human step, final effect. Two, test influence: time, anchoring, information, override, and what happens to a reviewer who disagrees. Three, assess rights and basis: applicable law, lawful basis, transparency, safeguards, DPIA needs. Four, redesign involvement: independent evidence, authority, time, escalation, alternative routes. Five, monitor practice: override quality, disparities, complaints, corrections, significant effects.

Step three is not a formality. One judgment shows it ending a system outright. SyRI was the Dutch welfare-fraud risk-profiling scheme. Its risk model, its indicators and the data it processed were all secret. On 5 February 2020 the District Court of The Hague ruled that this left the system insufficiently transparent and verifiable, and that it failed the fair-balance test of Article 8(2) ECHR. The court's conclusion: “For this reason, the court declares in this judgment that Section 65 SUWI Act and Chapter 5a SUWI Decree have no binding effect, being contrary to Article 8 paragraph 2 ECHR.”

The system was not struck down for producing bad scores. It was struck down because secrecy left its effect unverifiable and uncontestable by the people it landed on.

Automated decision and profiling governance is a claim about influence. So influence is what has to stay visible, measurable and correctable while the system runs.

Michigan's personnel were visible, and nobody measured their influence. Then an auditor counted 22,427 robo-adjudications and found over 93% with no fraud in them at all. Uber's reviewers were visible, and their influence could not be measured, because their qualifications were never stated. SyRI's influence was invisible by design, and that is what ended it.

Write down in advance which findings would force the provider to redesign, restrict, remedy or retire the system. A list written after the numbers arrive is not a trigger. An oversight requirement that cannot force anything is exactly what Green means by legitimising the systems it fails to control — the regulators' “just a token gesture”, one level up.