Training, Inference, and Feedback

Later paths examine optimization and evaluation in depth. Here the aim is to understand which activity is happening and what evidence each stage can provide.

During training, an algorithm adjusts model parameters so that outputs better match an objective on examples or experience. Two earlier choices set the limits. The model family decides what can change; the training procedure decides how improvement is measured.

Training does not simply pour facts into a container. It shapes a mapping that may generalize to new inputs, memorize parts of the data, or optimize a proxy in unintended ways.

The memorization part has been measured rather than assumed. Nicholas Carlini and colleagues presented “Extracting Training Data from Large Language Models” in August 2021. They queried GPT-2 XL, a 1.5-billion-parameter model trained on scrapes of the public web. They generated 1,800 candidate outputs. An author then manually checked every one of them. “In total across all strategies, we identify 604 unique memorized training examples from among the 1,800 possible candidates, for an aggregate true positive rate of 33.5%.” The recovered strings included names, phone numbers and email addresses. IRC conversations, code and 128-bit UUIDs came out too. The attack worked “even though each of the above sequences are included in just one document in the training data”. Their other result is the one to carry into any conversation about scale. “Worryingly, we find that larger models are more vulnerable than smaller models.”

Training, evaluation, and inference can use similar-looking inputs, but they serve different purposes and must remain conceptually separate.

The separation is physical as well as conceptual. Meta’s report on Llama 3, posted to arXiv on 31 July 2024, records what pre-training cost. A 405-billion-parameter model was trained on 15.6 trillion text tokens. That took 3.8 × 10^25 floating-point operations, on a cluster of up to 16,000 H100 GPUs. The published model card puts that single run at 30.84 million GPU-hours. The same report notes that “During a 54-day snapshot period of pre-training, we experienced a total of 466 job interruptions”, 419 of them unexpected. Inference on the result is one pass through the parameters that run produced, repeated once for each word-piece (token) of the answer. Training is a one-off industrial process measured in weeks of continuous cluster time. Inference is what happens every time someone presses a key.

LineSight, the invented defect detector from lesson 7, is small enough to follow through all three activities. Every number below is invented too.

“The AI learns from you” can describe personalization, periodic retraining, online updates, or merely saving preferences. Those are not the same mechanism.

Feedback looks valuable because it comes from real use. It can still be selective, delayed, strategic, or shaped by the very system it is meant to judge.

The click case in particular has been measured under controlled conditions. Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke and Geri Gay published “Accurately Interpreting Clickthrough Data as Implicit Feedback” in 2005. They put an eye-tracker on search users. Then they manipulated the ranking behind their backs. Users “click substantially more often on the first than on the second link, while they view the corresponding abstract with almost equal frequency”. When the top two results were quietly swapped, “there is still a strong bias to click on link one even if the second abstract is more relevant”. When the whole list was reversed, the average rank of a clicked document moved from 2.66 to 4.03. The average number of clicks per query fell from 0.80 to 0.64. The users reacted to the ranking, not only to the documents. The paper’s one-line conclusion is the sentence to keep: “clicks are informative but biased.”

A new dataset, label policy, threshold, prompt, retrieval index, model version, or interface can change system behavior. Any of them. Versioning and evaluation therefore have to match the real unit of change, which is rarely the model alone.

The central beginner insight is simple: deployment does not end learning work. It changes the questions from “Can we fit a model?” to “Is this version still useful and acceptable here?”

European law now writes that obligation down. The EU AI Act was published in the Official Journal on 12 July 2024. Its Article 43(4) provides that a high-risk system already assessed once “shall undergo a new conformity assessment procedure in the event of a substantial modification”. That holds “regardless of whether the modified system is intended to be further distributed or continues to be used by the current deployer.” Article 3(23) defines a substantial modification. It is a change made after the system is placed on the market or put into service “which is not foreseen or planned in the initial conformity assessment carried out by the provider and as a result of which the compliance of the AI system with the requirements set out in Chapter III, Section 2 is affected or results in a modification to the intended purpose for which the AI system has been assessed”. Both conditions matter: the change was not planned, and it touches compliance or purpose. The carve-out is the instructive part. Some systems “continue to learn after being placed on the market or put into service”. For those systems, some changes are “pre-determined by the provider at the moment of the initial conformity assessment”. Written into the technical documentation, those changes “shall not constitute a substantial modification”. Continuous learning is not forbidden. It is allowed to the extent that the shape of the change was described and assessed in advance. The requirements those assessments check are not yet in effect, though. They are among the high-risk requirements held back by the postponement set out in lesson 1.