Environment, Observation, State, and Action

An agent acts inside an environment whose relevant state may be only partially observable. Observations are time-stamped views, not the world itself. Actions may succeed, fail, or change state in ways the model did not predict.

That is not a house opinion about agent design. It is a formalism, and it has been on the record since 1998. Kaelbling and two colleagues stated the whole problem in two sentences in Artificial Intelligence: “A POMDP is an MDP in which the agent is unable to observe the current state. Instead, it makes an observation based on the action and resulting state.”

The pieces have names. A POMDP is the tuple (S, A, T, R, Omega, O): an MDP plus a finite set of observations Omega, and an observation function O: S x A -> Pi(Omega). The agent itself splits in two. A state estimator (SE) updates a belief state from the last action, the current observation and the previous belief. A policy maps belief states to actions, never world states. Every runtime that reads a tool response and writes it into memory is implementing SE. Most of them never called it that.

The runtime should therefore define observation sources, freshness, identity, action preconditions, and postconditions. Otherwise the model must infer system semantics from prose and inconsistent tool outputs. A reading taken at one moment is not the world. By the time the agent acts on it, the world may have moved.

Most interfaces an agent meets are partially observable, and some are adversarial. Designs are usually drawn as though every interface were fully observable. Partially observable means the agent sees a slice, and cannot tell which parts it is missing. A design handles that well when state updates are based on observed postconditions rather than model assumptions. It handles it badly when the runtime forgets that a tool response is evidence with provenance, not unquestionable truth, and writes whatever came back straight into state.

The third column is the one usually asserted and rarely measured. It has been measured. AgentDojo, published in 2024, evaluates agents executing tools over untrusted data: 97 realistic user tasks across banking, Slack, workspace and travel environments, and 629 security test cases. Its abstract names the channel exactly: “AI agents are vulnerable to prompt injection attacks where data returned by external tools hijacks the agent to execute malicious tasks.”

Read that sentence next to the middle column and the two collapse into one problem. The attack does not arrive in the user's instruction. It arrives inside the observation the runtime was about to trust. An adversarial environment is a partially observable one in which the missing slice was chosen by someone else.

Environment state becomes an observation. The observation becomes internal state. Internal state becomes an action. That chain is the one to trace, and four of its five nodes already have names from the 1998 formalism. The environment states are S. The observations are Omega. The channel between them is the observation function O: S x A -> Pi(Omega). The node drawn here as "Internal state" is the belief state, maintained by the state estimator (SE) from the last action, the current observation and the previous belief. The policy consumes that belief state, not the world.

That matters for debugging. Because the policy is a function of the belief state, a wrong action has exactly four possible origins along the chain: the policy, the belief the state estimator built, the observation the belief was built from, or a world that changed after the reading was taken. Whoever owns the action should not also own the transition evidence. Trace the chain backwards when something goes wrong. Stop at the first node whose input you cannot reconstruct.

Specify the environment contract for one workflow already in service: observation sources, freshness, identity, preconditions, postconditions. Writing it down usually reveals how much the design trusts what a tool reports on sight. It also leaves a record of which state changes the runtime confirms by looking at the environment afterwards, instead of assuming they happened.

The step "Define action schemas" does not require inventing a notation. HTTP already has one. RFC 9110, published in 2022, defines the If-Match header field. It makes a request conditional on the target resource currently carrying an entity tag that matches one supplied by the client, compared with the strong comparison function. The spec explains why it exists: “If-Match is most often used with state-changing methods (e.g., POST, PUT, DELETE) to prevent accidental overwrites when multiple user agents might be acting in parallel on the same resource (i.e., to prevent the "lost update" problem).” The enforcement is not advisory. An origin server “MUST NOT perform the requested method if the condition evaluates to false”, and may answer 412 (Precondition Failed).

The step "Model uncertain transitions" has a specification too. The same standard makes PUT, DELETE and the safe methods idempotent, for precisely the case where the agent never learned what happened: “Idempotent methods are distinguished because the request can be repeated automatically if a communication failure occurs before the client is able to read the server's response.” MDN puts the same rule in one line: “A client can safely retry a request that uses an idempotent method, for example, in cases where there is doubt as to whether the request reached the server.” The converse is the trap the practice step exists to surface. “A proxy MUST NOT automatically retry non-idempotent requests”, and a client should not either, unless it has some means to know the semantics are idempotent, “or some means to detect that the original request was never applied”.

That last clause is the whole of "Choose verification reads", written by a standards body in 2022. An action whose outcome you cannot re-read is an action you cannot retry.

Start by measuring how often a state change is written from an observed postcondition rather than from what the model assumed would happen. τ-bench, published in 2024, builds that discipline into the scoring itself: “We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state.” The end state, not the agent's account of it.

τ-bench also supplies the number behind the phrase "only counts under repeated cases". It introduces pass^k, the rate at which an agent succeeds on all k independent trials of the same task. State-of-the-art function-calling agents such as gpt-4o “succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)”. A design that passes once and is scored once has not been measured. Halving again across eight repeats is what the second number means.

Track separately whether conflicting observations remain visible until reconciled. Two more measures — that every action declares which state version or resource identity it targets, and that the runtime can distinguish stale data from a genuine failure — count only once they hold under repeated or adversarial cases.

The environment contract is often more important than the prompt. It determines what the model can know, and which mistakes the runtime can prevent. Most of it has already been written down by other people, as the last three sections show: the belief state in 1998, the entity tag and the idempotent retry in RFC 9110, the disagreement itself in Gilbert and Lynch's impossibility proof.

An audit starts with the tool responses. Where they came from, how much weight the design puts on them, and whether anything checked them. It then follows each recorded state change back to the observation that justified it, rather than to the step the model expected to succeed. Where that trace ends in an assumption instead of a reading, you have found the thing that no prompt will fix.