Generative AI Without the Hype

A conversational interface can hide several distinct mechanisms. Identifying them helps you diagnose errors and choose the right evidence.

A text generator estimates plausible next tokens given instructions, context, and prior generated tokens; image, audio, and video generators use different representations and procedures, but they too create outputs by modeling patterns in training data.

“Plausible” is not the same as “true.” The training objective rewards pattern fit, while factual reliability depends on evidence, system design, evaluation, and the conditions of the request.

That gap is not only an engineering shortfall; part of it is provable. Adam Tauman Kalai and Santosh S. Vempala posted “Calibrated Language Models Must Hallucinate” on 24 November 2023. It was published in 2024. They derive “an inherent statistical lower-bound on the rate that pretrained language models hallucinate certain types of facts”. That bound has “nothing to do with the transformer LM architecture or data quality.” Some facts are “arbitrary”: their veracity cannot be determined from the training data. For those, a model satisfying a calibration condition generates a hallucination. The probability is “close to the fraction of facts that occur exactly once in the training data”. This holds “even assuming ideal training data without errors.” Their result is bounded in a useful way. It gives no statistical reason to expect the same behavior for facts that appear more than once. Nor does it for systematic facts such as arithmetic. That is one reason post-training, retrieval, and tools are the usual places to look for a remedy.

A robust application adds context, controls, and validation around the model call. Stakes decide how many of those steps survive. Skip several for low-risk creative work; keep all of them for anything consequential.

Whether those steps exist is not only a design preference. It can settle who is answerable. In Moffatt v. Air Canada, decided on 14 February 2024, British Columbia’s Civil Resolution Tribunal held the airline responsible. A support chatbot had told a passenger booking travel after a death in the family that a bereavement fare could be claimed retroactively. The airline’s own policy page contradicted that. The tribunal recorded the airline’s defense and rejected it. “Air Canada argues it cannot be held liable for information provided by one of its agents, servants, or representatives – including a chatbot. It does not explain why it believes that is the case. In effect, Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission.” It found that Air Canada “did not take reasonable care to ensure its chatbot was accurate”. It added: “While a chatbot has an interactive component, it is still just a part of Air Canada’s website.” The generated sentence was the organization’s sentence.

Generative products create or transform many types of data. Each modality has different evaluation and misuse concerns.

Those misuse concerns already have regulators attached to them. The United States Federal Communications Commission adopted a Declaratory Ruling on 2 February 2024. It was released on 8 February. It confirmed “that the TCPA’s restrictions on the use of ‘artificial or prerecorded voice’ encompass current AI technologies that generate human voices”. Calls using a cloned voice therefore require the prior express consent of the person called. Commissioner Geoffrey Starks described the trigger in a statement attached to the same document. “In January, potential primary voters in New Hampshire received a call, purportedly from President Biden, telling them to stay home and ‘save your vote’ by skipping the state’s primary. The voice on the call sounded like the President’s, but of course it wasn’t. Those were voice cloning calls.” One modality, one product category, one law that already applied.

The routine changes with stakes. Creative brainstorming may tolerate uncertainty; medical, legal, financial, or security decisions require much stronger controls.

The step “Limit tool authority” has a specific attack behind it. OWASP’s Top 10 for Large Language Model Applications ranks prompt injection first, as LLM01. Its 2025 edition separates two forms. Direct prompt injections “occur when a user’s prompt input directly alters the behavior of the model in unintended or unexpected ways”. Indirect prompt injections “occur when an LLM accepts input from external sources, such as websites or files”. That content redirects the model once it is read. The class was named and demonstrated on 23 February 2023. The paper is “Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection”. Its authors are Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz and Mario Fritz. They showed that processing retrieved prompts “can act as arbitrary code execution, manipulate the application’s functionality, and control how and if other APIs are called.” It is not a thought experiment. Microsoft published CVE-2025-32711 on 11 June 2025, under the title “M365 Copilot Information Disclosure Vulnerability”. It describes “Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network.” Least privilege matters here for one reason. The attacker may be a document your assistant was asked to read.

A later course examines tokenization, transformers, pretraining, post-training, decoding, retrieval engineering, tool use, evaluation, safety, serving, memory, and multimodal design in depth.

For now, keep the boundary. Generative AI produces candidates from learned patterns and supplied context; product reliability comes from the system that grounds, validates, authorizes, presents, and monitors those candidates.