Operating Foundation and Generative AI Systems

A refund is supposed to happen once, however many times the request is sent. The contract that makes that true says who generates the key. An IETF draft, revised on 15 October 2025, puts it in one sentence: “An idempotency key is a unique value generated by the client which the resource uses to recognize subsequent retries of the same request.” Generated by the client. On a repeat of an already completed request, the draft says the resource “SHOULD respond with the result of the previously completed operation, success or an error.”

The draft itself expired on 18 April 2026. It is marked "Expired & archived" on the IETF Datatracker and was never published as an RFC. The contract survived anyway, in production. Stripe's API implements exactly it. The client generates the key. Stripe saves the status code and body of the first request carrying that key and replays them on later requests, 500 errors included. Keys may be removed once they are at least 24 hours old. And the idempotency layer compares the incoming parameters against the original request, erroring if they differ.

Now the failure. A provider begins filling an optional idempotency field differently, and the model — not the tool layer — becomes the thing minting the key. Every retry carries a fresh value. So the resource never recognises a retry: it sees a first request, and the stored first result is never replayed. Nothing in the JSON schema was violated. The schema describes shape. The contract describes provenance.

A support assistant begins producing shorter answers after a hosted model update. The application code, prompt name, and retrieval index are unchanged. Yet user satisfaction and citation behavior shift.

Foundation-model operations must account for external model versions, provider routing, prompt and policy changes, retrieval state, tools, evaluators, and conversation context. The release is a compound system.

The most literal version of this is published on a calendar the team does not control. OpenAI's deprecations page lists gpt-4-32k, gpt-4-32k-0613 and gpt-4-32k-0314 as shut down on 6 June 2025. It lists gpt-4-0613, gpt-4-turbo, gpt-3.5-turbo-0125 and o1-2024-12-17 as shutting down on 23 October 2026. Those are strings sitting in configuration files today, and on dates already fixed they will resolve to nothing. Nothing in the repository will move on those dates either.

The language model is one layer inside a broader application, and a failure can be hard to attribute across those layers. That difficulty is not folklore. It is an enumerated risk in federal guidance.

NIST AI 600-1 is the generative-AI profile of the AI Risk Management Framework, published in July 2024. Executive Order 14110, of 30 October 2023, had given NIST 270 days to produce it. It enumerates twelve GAI risks. The twelfth is "Value Chain and Component Integration": non-transparent or untraceable integration of upstream third-party components. §2.12 states the consequence plainly: “As GAI systems often involve many distinct third-party components and data sources, it may be difficult to attribute issues in a system's behavior to any one of these sources.”

That is what the layers below are for. Each one is a place a behavior change can originate. A decision trace that cannot say which layer moved is the twelfth risk realised.

Neither option removes operational work. What the hosted route has acquired, at least in the EU, is a documentation obligation that runs downstream toward the integrator rather than sitting with the provider alone.

The EU Artificial Intelligence Act writes it out twice. Article 53(1) obliges providers of general-purpose AI models to draw up and keep up to date technical documentation covering the training and testing process and the results of the evaluation. That set is Annex XI, held for the AI Office and national competent authorities. Article 53(1)(b) then obliges the same providers to “draw up, keep up-to-date and make available information and documentation to providers of AI systems who intend to integrate the general-purpose AI model into their AI systems”. That second set has a floor: at minimum the elements of Annex XII. The European Commission states that these general-purpose AI rules became applicable on 2 August 2025.

So "need contractual and evaluation controls" is no longer only a procurement preference on the hosted side. It is a named annex an integrator can ask for, on a date. Self-serving weights moves that evidence in-house, and moves the runtime, scaling, patching and supply-chain burden with it. Routing across providers multiplies both the evidence to collect and the behavior to compare.

A prompt can change hourly, an index daily, tool schemas weekly, and a hosted model without a local deployment event. Evaluation cases and grader models also evolve.

The last of those clocks is mechanical, and one provider has written down its gearing. Microsoft's Foundry Models lifecycle policy sets a GA model's retirement date programmatically, at 18 months from launch. It gives at least 60 days' notice. After retirement, all inference returns HTTP 410 Gone. Standard deployments are auto-upgraded: gpt-4o version 2024-05-13 retires on 2026-10-01 and is replaced by gpt-5.1. Provisioned deployments are never auto-upgraded. One deployment type has its behavior changed for it, the other stops answering, and both events are scheduled outside the repository. The policy's FAQ closes the negotiation in three words: “Retirement dates aren't extendable.”

Treat each behavior-changing component as a versioned asset with effective time. A decision record should preserve the exact provider or model identifier when available, the prompt, the retrieval corpus, the tool policy and the fallback. A retirement date belongs in that record too. It is not a calendar invite.

Freeze the complete behavior envelope before comparing candidates. And decide what a single eval score is worth before promoting on one.

τ-bench, published at ICLR 2025, built the gate that step 2 needs. It introduced pass^k: the probability that an agent solves the same task on all of k independent reruns. Its authors ran it over retail and airline customer-service domains. The abstract reports the result. “Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail).” Under 50% average success on one run. Roughly 25% once the same task must come out right on all of eight. Sierra, which built the benchmark, calls that “a staggering 60% drop”.

The operational reading is narrow and useful. A single-run score on a tool-calling suite can overstate the agent by more than half, and the gap is invisible until the suite is rerun. Representative evals therefore means reruns as well as coverage: groundedness, tool traces, safety, edge cases, cost and user tasks, each measured with the consistency the production path actually requires, before shadow or canary traffic begins.

A unified model interface can simplify code while hiding tokenization, context, safety, tool, and version differences. Preserve provider-specific evidence behind the abstraction. Where Article 53(1)(b) applies, the Annex XII documentation the upstream provider owes you is part of what there is to preserve.

The application should degrade safely when the model, retrieval, or tool layer is unavailable. It should never expand authority merely to preserve a fluent response.

A national security agency has written down why the boundary belongs outside the model. “Current large language models (LLMs) simply do not enforce a security boundary between instructions and data inside a prompt”, the UK’s NCSC wrote in December 2025, because “under the hood of an LLM, there’s no distinction made between ‘data’ or ‘instructions’; there is only ever ‘next token’”. Its conclusion is that “it’s very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be”. Design as though the filter will be bypassed.

It has already been bypassed in a shipping enterprise product, with a number on it. CVE-2025-32711 records an AI command-injection flaw in Microsoft 365 Copilot, classified under CWE-74. Microsoft, the assigning CNA, published it on 11 June 2025 under the title "M365 Copilot Information Disclosure Vulnerability". The CNA description is one line: “Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network.” Read the vector Microsoft assigned — CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:L/A:N — and note UI:N: no user interaction. Microsoft scored the record CVSS 3.1 base 9.3 (Critical). NVD's own primary analysis of the same record scored it 7.5 (High). Two bodies read one flaw and did not agree on how bad it was. That is a further reason to place authority, idempotency and approval in the tool layer, rather than sizing controls to somebody's severity estimate.