AI Literacy, Competence, and Role-Based Training

Boards, product owners, technical teams, operators and procurement each need a different competence. None of them needs all of it. The map below is not a hierarchy. A procurement officer who cannot ask a vendor whether a model has ever been externally validated fails one way. A clinician who cannot judge what a score is worth at the bedside fails another. One syllabus written for both misses each of them.

The Epic Sepsis Model was a proprietary tool implemented at hundreds of US hospitals. Nobody outside the vendor had ever measured it. In 2021 Wong and colleagues ran the external validation the purchase never had: 38,455 hospitalizations of 27,697 adults at Michigan Medicine, between 6 December 2018 and 20 October 2019. JAMA Internal Medicine gave the result in one line: “The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64).” The model missed 1,709 of 2,552 sepsis patients (67%). It still fired alerts on 6,971 of 38,455 hospitalizations (18%). Every clinician on those wards was being asked to read that score at the bedside. Whoever signed for the tool had bought a model nobody had checked. Those are two different jobs. No general awareness module tells them apart.

AI literacy means grasping what an AI system can do, where it stops working, what it puts at risk, and when it fits the job in front of you. Competence adds the skill to perform a role reliably. Training should be proportional to the person's authority, exposure and task. A builder handles the data, evaluates the model, keeps it secure, tests it for fairness and writes down what they did. A leader sets the risk appetite, names who answers for a decision, knows what to ask a vendor before signing, and reads the evidence. Affected staff need clear rights and safe reporting channels.

The operator's syllabus is not a matter of taste. Article 14(4) of Regulation (EU) 2024/1689 enumerates it in five points. Whoever holds human oversight must be enabled to understand the system's capacities and limitations and detect anomalies (a), to remain aware of automation bias (b), to correctly interpret the output (c), to decide not to use the system or to “disregard, override or reverse the output” (d), and to intervene or stop the system (e). Point (b) is worth reading as the legislator wrote it: “to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons”. Interpret, judge the pull of the machine, override, stop. That sequence is a legal specification, not a training designer's preference.

One syllabus lands differently on different roles. That is measurable, and it has been measured. Give 138 radiologists and 127 internal and emergency medicine physicians the same eight chest X-ray cases, with the same advice attached, some of it deliberately wrong. The two groups do not respond alike. Gaube and colleagues reported the split in npj Digital Medicine in 2021: “We define clinical susceptibility as the propensity to follow incorrect advice, and we find that 41.73% of IM/EM physicians are susceptible, i.e., they always give the wrong diagnosis with inaccurate advice. This is true only for 27.54% of radiologists.” In the other direction, 28.26% of radiologists refuted every piece of incorrect advice, against 17.32% of the IM/EM physicians. The abstract names the mechanism: “As a group, radiologists rated advice as lower quality when it appeared to come from an AI system; physicians with less task-expertise did not.” Same cases, same advice, two roles. Different competence at rejecting a bad output.

Training that ends at a slide deck gets measured by counting who turned up. The five steps below measure what people decide instead. The first two are already written down in a public framework, with clause numbers you can cite. NIST's AI Risk Management Framework, published in January 2023, puts training under GOVERN 2: teams and individuals are “empowered, responsible, and trained for mapping, measuring, and managing AI risks”. GOVERN 2.1 requires roles, responsibilities and lines of communication to be documented and clear throughout the organization. That is step 1, mapping roles to the decisions they own. GOVERN 2.2 requires that “The organization’s personnel and partners receive AI risk management training to enable them to perform their duties and responsibilities consistent with related policies, procedures, and agreements.” Training tied to the duties a person must actually perform is step 2. Neither clause counts attendance. And the word “partners” is why the agency-supplied clerk is inside the scope too.

Awareness recognizes the vocabulary. Role competence performs a task under realistic pressure. Expert qualification carries independent assurance. The line between the first two tiers has legal weight wherever the system is high-risk. Article 26(2) of Regulation (EU) 2024/1689 requires a deployer to assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support. A person who can recite the policy and name the vendor's terms, and who has never handled a live output or tested a control, sits at the first tier and not the second. Whatever the completion log records, and whatever the job title says.

The last drill is the awkward one: confirm that workload, interface and authority actually let the trained behavior happen. The documented version of that failure is a crash report. A vehicle running an automated driving system crashed in Tempe, Arizona on 18 March 2018. A trained safety operator was behind the wheel. The National Transportation Safety Board adopted its probable cause on 19 November 2019: “The National Transportation Safety Board determines that the probable cause of the crash in Tempe, Arizona, was the failure of the vehicle operator to monitor the driving environment and the operation of the automated driving system because she was visually distracted throughout the trip by her personal cell phone. Contributing to the crash were the Uber Advanced Technologies Group’s (1) inadequate safety risk assessment procedures, (2) ineffective oversight of vehicle operators, and (3) lack of adequate mechanisms for addressing operators’ automation complacency—all a consequence of its inadequate safety culture.” Reporting on the determination records that she “looked down 23 different times” in the final three minutes. The report also records what had changed around her. During September–October 2017 Uber ATG folded the work of two vehicle operators into one, noting that “the consolidation of responsibilities also increased the task demands on the now-sole operator”. Two of the three contributing factors the board named are environment, not syllabus.

Competence decays with every model update. The legal floor moves too. The operative wording of Article 4 changed on 27 July 2026, so a training record written against the old “ensure, to their best extent, a sufficient level of AI literacy” formulation is dated in a way its author cannot see from the inside. And a training record carrying no date beside a system version proves nothing about anything.

Define when a literacy and competence finding requires the reviewer to redesign, restrict, remedy, or retire the system. And record, beside every assessment, three things: the date, the system version it was taken against, and whether the person assessed actually holds the authority and the support that Article 26(2) of Regulation (EU) 2024/1689 places in the same sentence as their training.