Security, Privacy, and Abuse-Resistant Design

Attackers can shape inputs directly. They can also shape them indirectly, through content the system retrieves later and then acts on. The second route has a founding paper. Greshake and colleagues named indirect prompt injection in February 2023 and demonstrated it remotely against production systems, including Bing's GPT-4-powered Chat and code-completion engines. Their diagnosis is architectural rather than anecdotal: “We argue that LLM-Integrated Applications blur the line between data and instructions.”

That blurred line is what the boundaries below exist to redraw. Each one is a place where something the application did not write becomes something the application acts on.

No single filter eliminates prompt injection or misuse. The size of what layering does buy has been measured. AgentDojo, published in June 2024, is a public evaluation environment for tool-using agents, populated with 97 realistic tasks and 629 security test cases. Its introduction gives the three numbers a defender needs: “Current LLMs solve less than 66% of AgentDojo tasks in the absence of any attack. In turn, our attacks succeed against the best performing agents in less than 25% of cases. When deploying existing defenses against prompt injections, such as a secondary attack detector, the attack success rate drops to 8%.”

Read the sequence as an argument for layers and as a warning about them. Under 25% down to 8% is a large reduction and a poor guarantee: roughly one attack in twelve still lands. The reduction is not free either. All defences lose 15–20% of utility under attack. A control that costs a fifth of the system's usefulness has to be chosen deliberately, action by action, rather than switched on everywhere.

Training data, fine-tuning records, prompts, retrieved documents, tool responses, conversation state, outputs and logs can all contain personal or confidential information. A private model endpoint does not make the whole workflow private.

The training-data end of that lifecycle has a published price. In November 2023 Nasr and colleagues showed that a divergence attack made ChatGPT emit training data at a rate 150× higher than normal. They reported the result as a budget line: “Using only $200 USD worth of queries to ChatGPT (gpt-3.5-turbo), we are able to extract over 10,000 unique verbatim-memorized training examples.” The longest ran to over 4,000 characters. And 93% of the memorized strings were emitted only once — memorization that a sampling-based audit would mostly miss.

Collect only necessary data. Separate tenants, redact where appropriate, define retention and deletion, restrict reviewer access, and test for unintended disclosure. Explain which records may persist and how users can correct or remove them.

Follow data and authority from input to irreversible effect. For high-risk systems the exercise is not a suggestion. Article 15(5) of the EU AI Act, Regulation (EU) 2024/1689, makes adversarial resilience a legal requirement, and it enumerates the classes providers must handle: “The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent, detect, respond to, resolve and control for attacks trying to manipulate the training data set (data poisoning), or pre-trained components used in training (model poisoning), inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion), confidentiality attacks or model flaws.” Five verbs, five attack classes. The statute is itself a threat-modelling checklist.

Security improves when unsafe proposals cannot become unauthorized effects. Every number in this lesson points the same way. 84% transfer for automatically generated suffixes. 63%, and over 30%, of improvement that still leaves models vulnerable. 8% residual attack success after a dedicated detector. $200 to pull 10,000 verbatim training examples out of a production endpoint. One CVE with no privileges and no user interaction. The goal is not a prompt that never fails. It is a system whose boundaries contain failures and expose them for response — the design posture NIST AI 100-2e2025 recommends for any model exposed to untrusted input.

The next lesson addresses model choice and economics. Security and privacy controls affect cost, latency, architecture, and which requests should reach a powerful model at all. AgentDojo's 15–20% utility cost under attack is a reminder that those controls are priced in the same currency.