Never retry a timeout

Every payments engineer has a version of this story. Mine involves a payment call that took longer than its timeout, a well-intentioned retry wrapper written months earlier for a completely different problem, and a customer who paid twice for one purchase. The money was refunded, the incident was written up, and I walked away with the scar tissue this post is about.

The retry wrapper wasn’t badly written. It was applied to the wrong category of failure, because the codebase treated all failures as one thing. They are not one thing.

A timeout is not a failure

When a network call goes wrong, you are in one of three fundamentally different situations, and everything downstream depends on telling them apart:

The request never arrived. DNS failed, the TCP connect was refused, the TLS handshake died. Nothing happened on the other side, because the other side was never reached. This is a true failure, and it is safe to retry, as aggressively as politeness allows.

The request arrived and you got an answer. A response came back, even an ugly one. A 500, a validation error, a decline. You know the outcome. You can decide what to do because there is no mystery, and retrying is a business decision (is this error transient?) rather than a safety question.

The request went out and the answer never came back. This is the timeout. And here is the sentence I want on a poster: a timeout is not a failure, it is an unknown outcome. The server may have crashed before doing the work. It may have done the work and the response died in the network. It may still be doing the work right now, slower than your patience. You do not know, and nothing in your process can know, because the knowledge lives on the other side of a wire that just proved unreliable.

Retrying an unknown outcome doesn’t “try again.” It potentially does the thing again. If the thing was “charge the card,” you may have just charged it twice. The retry that felt like resilience was actually a bet that the first attempt failed, placed with someone else’s money.

The three honest responses to a timeout

Once you accept that a timeout is a question mark rather than a no, only three honest moves exist:

Make repeats harmless, then retry. Attach an idempotency key to every mutating request. The server records “I have processed key K” and returns the original result on any repeat. Now retrying the unknown is safe, because at-most-once semantics are enforced where the truth lives, on the server. This is the gold standard, and if you control both sides of an API, it’s the contract to build.
Ask, don’t act. If you can’t make repeats safe, stop mutating and start reading: query for the outcome of the thing you attempted. Did an order with my reference get created? Did a charge with my reference land? Reads are safe to retry, so poll until you know, then proceed from knowledge. This is reconciliation, and a small periodic reconciliation loop that sweeps for “things we attempted but never confirmed” is the most underrated component in money systems.
Stop and tell someone. For low-volume, high-stakes operations, the honest answer is to park the item in a “needs attention” state and alert. Undramatic, unfashionable, correct.

What is never on the list: fire the same mutating request again and hope. Hope is not an idempotency strategy.

The rules I actually apply

Split your timeouts. A connect timeout and a read timeout are different beasts: connect failures are retryable by definition, read timeouts are unknown outcomes. One combined “request timeout” setting erases the most important distinction in the failure space. Keep connect timeouts short and retry them freely; treat read timeouts as radioactive.
Classify every endpoint before wrapping it in retries. Reads: retry. Idempotent writes (key-protected): retry with backoff. Non-idempotent writes: never automatic, reconcile instead. If a generic retry wrapper can’t tell these apart, it’s not a resilience tool, it’s a double-charge generator with good intentions.
Design every mutating endpoint by answering “what happens when this is called twice?” in writing. If the answer isn’t “nothing, it’s a no-op,” the endpoint isn’t finished.
Let humans off the retry hook too. A user staring at a dead spinner will refresh and resubmit; that’s a manual retry of an unknown outcome. Honest pending states (“we’re confirming your payment, we’ll notify you”) are the UI half of this same principle. I wrote more about that in Designing for the network you actually have.
Trust reconciliation over heroics. The system that recovers cleanly is rarely the one with the cleverest retry logic. It’s the one where a boring loop compares “what we attempted” against “what actually happened” and settles the difference.

The punchline

Retries are a wonderful tool exactly as far as the failure is known to be a failure. The moment the outcome is unknown, the brave-looking move (try again!) is the reckless one, and the timid-looking move (stop, ask, reconcile) is the professional one.

Some of the best engineering in a payment system is invisible precisely because it’s the retry that isn’t there.