Humans and AI: Assistance, Automation, and Oversight

AI can support perception, memory, search, drafting, forecasting, and consistency. People contribute goals, contextual judgment, moral responsibility, lived experience, negotiation, and the ability to recognize when the task itself is wrong.

Good design allocates work according to those strengths and to the consequences of failure. It never asks which is better in the abstract, humans or AI. The question has no answer.

The same model can occupy different roles. Moving right increases speed and scale, but also increases the importance of safeguards and recoverability.

Collaboration failures often arise from role design rather than raw model accuracy.

Most of these were named before any of them involved software. Lisanne Bainbridge set them out for process control in “Ironies of Automation”, Automatica 19(6), 775–779 (1983). On skill erosion she is exact: “physical skills deteriorate when they are not used, particularly the refinements of gain and timing. This means that a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one.” Her one-sentence statement of the whole problem opens the paper’s section on human-computer collaboration: “By taking away the easy parts of his task, automation can make the difficult parts of the human operator’s task more difficult.” And the irony she keeps for last is the one training budgets still miss: “it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training.”

Oversight is strongest when the system routes uncertainty deliberately and learns from structured corrections without treating every click as truth.

Oversight should be strongest where errors are hard to detect, difficult to reverse, or unevenly distributed.

The last step is the one most often skipped. The cost of skipping it has been measured. Fenton and colleagues studied computer-aided detection for screening mammography. They covered 429,345 mammograms from 222,135 women at 43 facilities in three US states between 1998 and 2002. The result appeared in the New England Journal of Medicine in 2007. Read together with the radiologists, the software moved specificity from 90.2% to 87.2%. It cut the positive predictive value from 4.1% to 3.2%, and raised the biopsy rate by 19.7%. The increases in sensitivity and in the cancer detection rate were not statistically significant. Their conclusion is a statement about the team, not about the detector: “the use of computer-aided detection is associated with reduced accuracy of interpretation of screening mammograms.” A component can be good and the pairing still worse.

Over-trust leads people to accept plausible errors. Under-trust wastes useful assistance and can push users toward unmonitored workarounds.

Calibrated trust means relying on the system when the evidence supports reliance, and checking or rejecting it when conditions change. Products can help. They set realistic expectations, show the relevant evidence, and fail in ways a person can understand.

Both halves of that have been observed in one experiment. Tschandl and colleagues tested AI-based decision support for skin cancer recognition against clinicians at every level of experience. They reported in Nature Medicine in 2020. Good quality AI-based support “improves diagnostic accuracy over that of either AI or physicians alone”. And “the least experienced clinicians gain the most from AI-based support”. The same abstract records the other half: “faulty AI can mislead the entire spectrum of clinicians, including experts”. Expertise did not protect the experts. What separates the two situations is not how confident the reader feels. It is whether the reader can tell which system is in front of them. That is a property of the evidence the product puts on the screen.