Prompting as Interface Engineering

A finance assistant receives “Review this contract and flag anything important.” It returns a polished summary but misses the renewal clause that the legal team actually cares about. Three documents later the team agrees the prompt works and ships it.

That approval procedure cannot tell a working prompt from a lucky one. Hold the task, the model and the examples fixed, change only the order the few-shot examples are presented in, and the score moves anyway. Lu and colleagues ran that experiment and reported it at ACL 2022: “We demonstrate that the order in which the samples are provided can make the difference between near state-of-the-art and random guess performance: essentially some permutations are “fantastic” and some not.” The models were GPT-3-family. Good orderings did not transfer between them, so the permutation that made one model look near state-of-the-art was no guide to the next. Their entropy-based selection method recovered a 13% relative improvement across eleven text classification tasks.

A prompt approved on three self-chosen documents rests on exactly the evidence base that cannot separate the prompt from the permutation.

A prompt can define roles, priorities, terminology, and response structure. It can also tell the model to distinguish source text from instructions found inside that source. The model may still misread evidence, ignore a condition, or follow injected content. Permissions, source filtering, schema validation, and transaction controls must remain outside prompt prose.

There are two ways in, and NIST names both in its Generative AI Profile. In the direct case, “attackers might craft malicious prompts and input them directly to a GAI system”. Indirect prompt injection instead “occur[s] when adversaries remotely (i.e., without a direct interface) exploit LLM-integrated applications”. Greshake and colleagues demonstrated the indirect case against real deployed applications in 2023. Such applications blur the line between data and instructions, and an attacker can compromise one by planting a prompt in data the system will later retrieve.

It is no longer only a research demonstration. On 11 June 2025 Microsoft published CVE-2025-32711, “M365 Copilot Information Disclosure Vulnerability”, and described it in a single line: “Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network.” The record classifies the flaw CWE-74. Microsoft rates it CVSS 3.1 base score 9.3, Critical (AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:L/A:N), while recording it as not publicly disclosed, not exploited, and already fully mitigated with no action for users of the service. The National Vulnerability Database carries the identical description and republishes that 9.3 as a secondary metric. But NIST's own primary CVSS 3.1 assessment is 7.5, High (AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N). Two competent parties scored one shipped assistant's injection flaw and did not agree on how bad it is. That is a reason to build the boundary in code, not a reason to argue about the number.

A longer prompt is not automatically a stronger interface. Wording is not even the largest source of variation. Change the formatting of a few-shot prompt without changing its meaning, and the score moves. Sclar and colleagues measured how far and reported it at ICLR 2024: “We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B.” Format performance was only weakly correlated between models, so a formatting choice validated on one model carries no promise to the next. Their response was not a better format but a better report. They released FormatSpread, which gives a performance interval instead of a single number.

That is why the useful comparison is not short against long but untestable against testable. A vague request is short and easy to demo. It relies on the model to infer hidden goals and constraints, so task interpretation comes out inconsistent, objective evaluation is difficult, and unsupported completion is encouraged. An overloaded instruction — many rules, exceptions and tasks in one block — appears comprehensive instead. It creates conflicts and priority ambiguity, it is hard to debug after a failure, and it may waste context and attention. A testable contract separates task, evidence, constraints, schema and fallback. It supports component-level diagnosis, enables structured evaluation, makes missing evidence visible, and works with deterministic controls.

Only the third kind can be given an interval at all. The first two report a single number with a 76-point neighbourhood nobody has looked at.

Treat each change as a release candidate rather than a clever phrase. Write the task contract first: success, evidence, exclusions and failure behavior. Build representative cases spanning ordinary, rare, ambiguous, adversarial and out-of-scope inputs. Establish a simple baseline with direct instructions before adding elaborate scaffolding. Then change one mechanism at a time, adding examples, decomposition or context only in response to a measured failure.

Scoring, the fifth step, has a documented ceiling. A strong model judge agrees with human preferences about as often as humans agree with each other. Zheng and colleagues showed that at NeurIPS 2023, alongside MT-Bench and Chatbot Arena and a release of 3K expert votes and 30K human-preference conversations: “Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.” That single sentence is both the case for automated scoring and its limit. The same work documents position, verbosity and self-enhancement biases in LLM judges. That is why the judge must not be the model whose own variants are being ranked, and why human inspection of disagreements and tails stays in the loop.

Versioning, the sixth step, is no longer house style. It is written into two published baselines, and they ask for the same thing: “Developers should ensure that they have an audit log of changes to system prompts or other model configuration (including prompts) that affect the underlying working of the systems.” That is provision 8.3 of the UK DSIT Code of Practice for the Cyber Security of AI, published 31 January 2025, under Principle 8, “Document your data, models and prompts”. The matching provision 5.2.4-3 of ETSI TS 104 223 V1.1.1, April 2025, carries the same requirement with a standard number attached. Release prompt, model, tools, schemas and evaluation results together, and log the change. A reader who is asked why can now hand over a provision number instead of a preference.

The best prompt is not the most literary or the longest. It makes intended behavior, evidence, and failure visible enough to test.

The menu of things you could add instead is larger than it looks. Schulhoff and colleagues surveyed it in The Prompt Report in June 2024: “We present a detailed vocabulary of 33 vocabulary terms, a taxonomy of 58 LLM prompting techniques, and 40 techniques for other modalities.” They identify conflicting terminology and a fragmented ontology as the field's core problem. So the catalogue cannot simply be worked through from the top, and one technique may reach you twice under two names.

The next lessons focus on demonstrations, decomposition, and reasoning workflows. With 58 catalogued text-only techniques on offer, each technique should address a diagnosed problem rather than become ritual prompt decoration.