Timeouts, Retries, and Idempotent Tool Use

A timeout means the caller lacks a timely result. It does not prove that the operation failed.

Retrying is safe in three cases only: the tool is idempotent, the original outcome can be queried, or the action carries a stable idempotency key. That test is not house style. It is written into the HTTP standard. RFC 9110 defines idempotency and names PUT, DELETE and the safe methods as the idempotent ones. Then it makes the rule normative: “A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied.”

The same passage closes the two obvious escapes. A proxy MUST NOT automatically retry non-idempotent requests. A client SHOULD NOT automatically retry a failed automatic retry. The entitlement to send a request twice comes from knowledge — of the semantics, or of what happened. It never comes from the mere fact that no answer arrived.

Retry policy should therefore depend on error class, side-effect semantics, and remaining task budget. Uniform retries turn transient uncertainty into duplicated work.

A deadline expires, the error is classified, the idempotency key decides whether a second attempt is safe, and an outcome lookup settles what actually happened. Nobody should own both the outcome lookup and the retry budget. No single test should cover both.

The retry budget is the step that looks most like bookkeeping and is most often the cause of the outage. A 2022 study ranked what kept 22 metastable failures going, across incidents reported by 11 organisations, and retry policy came first: “By far, the most common sustaining effect is due to the retry policy, affecting more than 50% of the studied incidents”. Huang and colleagues name the affected cases individually: GGL2, GGL3, AWS1, AWS2, AWS3, AZR2, AZR4, IBM1, SPF1, SPF2 and CAS2. The same study attributes at least 4 of the 15 major AWS outages of the preceding decade to metastable failures, with outage durations running from 1.5 to 73.53 hours.

Read against the diagram, that finding relocates the danger. The trigger sits at the left of the path and is usually brief. What kept more than half of those systems down was the policy at the right-hand end, still running long after the thing it was compensating for had passed.

Safe read retry, idempotent write, and non-idempotent write are three different contracts. Only the third can turn a timeout into a second charge. The first two are safe to resend: a read changes nothing, and an idempotent write collapses duplicate calls onto one key, so ten attempts leave behind exactly what a single attempt would have. RFC 9110 draws the same line by naming names — PUT, DELETE and the safe methods are the idempotent ones. That is why the argument always happens over a POST or a PATCH.

The third contract makes no such promise, so the decision to send it again cannot be taken inside the tool. It has to be taken by the runtime, before the duplicate is ever on the wire. Moving a method out of that third column is exactly what the Idempotency-Key header is for, and the IETF draft that specifies it says so: “The HTTP Idempotency-Key request header field can be used to make non-idempotent HTTP methods such as POST or PATCH fault-tolerant.” A non-idempotent write does not become an idempotent write because the runtime would like it to be one. It becomes one when a key, a saved result and a defined answer for the in-flight case — HTTP 409 Conflict — are all actually implemented behind it.

Everything else rests on one measurement: that repeated execution produces at most one logical side effect. Separate from it, and easy to conflate with it, is whether the runtime distinguishes unknown outcome from confirmed failure. A timeout says nothing about whether the write landed, and treating it as a failure is a guess. Two more checks only show themselves under repeated or adversarial cases: that retry counts and elapsed budget are visible in the trace, and that non-retryable errors do not enter model-driven retry loops.

The last of those looks like the least important signal on the list. It is not. Almost every catastrophic failure in a 2014 study of 198 randomly selected user-reported failures — from Cassandra, HBase, HDFS, Hadoop MapReduce and Redis — came from code that had already been told something was wrong. Yuan and colleagues put it in one line: “almost all (92%) of the catastrophic system failures are the result of incorrect handling of non-fatal errors explicitly signaled in software”. The systems had detected the problem and said so. In 58% of those cases the fault could have been caught by simple testing of the error-handling code. In 35% the error handler was empty or log-only, aborted the cluster on an over-general exception, or still contained “FIXME”/“TODO”. A retry loop driven by a model is another error handler nobody tested, sitting on the same path where 92% of the damage was decided.

Write the retry matrix for one workflow: every tool down one axis, every error class along the other, and a decision in each cell. The cells you cannot fill in with confidence are the ones where retries can create a second incident after the first action succeeded. A timeout on a write, most often, where the caller cannot tell a lost response from a lost request. Beside each of those, note what the tool would have to offer — an idempotency key or a status lookup — before the cell has a safe answer to put in it.

Give each cell the numbers it has to live inside: the three attempts and the 10% ceiling from Google's SRE book if you are borrowing that policy, or the delay-then-hedge discipline that held Dean and Barroso's extra load to 2%. Then write the stop condition for every non-idempotent tool as its own line, and test it. Knight's router had one. It had been moved in 2005. Nothing on the way to 1 August 2012 asked it to prove it still fired.

The agent should never invent retry semantics. They belong in the tool contract and runtime policy. That is where RFC 9110 puts them: a client SHOULD NOT automatically retry a non-idempotent method without some means to know the semantics are idempotent anyway, or some means to detect that the original request was never applied. A team revisiting retry-safe tool execution should start from the incidents the current policy makes possible, not from the wording of the policy, and only then adjust the budgets and backoff. AWS had a well-tested client back-off on 7 December 2021 and a latent bug that stopped it firing. The wording of that policy was never the problem.

The question to answer before release is narrow. For every call this policy will resend, name the mechanism that stops the second attempt from doing the work twice — the saved status code and body, the 409 on the in-flight case, the outcome lookup. Where no mechanism can be named, the call does not get resent.