Timeouts, Retries and the Retry Storm

Stagefront is a ticketing service that exists only in this course. A stadium show goes on sale at exactly 10:00, two hundred thousand people press the same button in the same minute, and forty thousand of them get a seat.

On a Tuesday in March the on-sale opened on time and for thirty four seconds it was the best one of the year. Seventeen thousand seats gone. The twenty application boxes were at the seventy five percent CPU that lesson 006 says is normal for that minute, and every purchase landed inside the two second promise lesson 003 published.

At 10:00:34 the payment provider got slow. Not down. Its median response went from about a hundred and fifty milliseconds to about nine hundred and stayed there for roughly half a minute, which on an ordinary morning is a graph nobody opens.

Stagefront's checkout gives each charge one and a half seconds and tries three times. The API layer in front of it waits two seconds and retries a 5xx once. The phone app tries three times. Three sensible numbers, chosen by three different people, never once written on the same page.

By 10:00:52 the checkout tier was offering nine thousand charge attempts a second to a provider it normally asks for five hundred. The first thing that broke was not the provider. It was Stagefront's own pool of a hundred connections, jammed full of charges that nobody upstream was waiting for any more.

Sales did not slow down. They stopped.

At 10:04 somebody halted the on-sale. At 10:40 it opened again with exactly one number changed: the payment client now tries once. The remaining twenty three thousand seats went in forty six seconds.

The provider had a bad thirty seconds. Stagefront sold nothing for the next thirty nine minutes, and almost every one of them was code written to make failures less likely.

A timeout is a guess about somebody else's distribution

Marlow Books, the four person online bookshop that has been breaking in interesting ways since lesson 001, gives the payment provider ten seconds before it gives up. Lesson 019 hung a whole story on that number.

Where did ten seconds come from? Nowhere. It arrived with a library, or got typed once on an afternoon of trying to make something work at all, and was never looked at again. I have never worked with a team that could tell me where their outbound timeouts came from without going to look, and I include every team I have been on.

Here is what the number actually claims. Beyond this point I no longer believe an answer is coming. That is a statement about the far end's latency distribution, and lesson 002 gave you the vocabulary for it: p50 is the median, p99 is the value ninety nine percent of calls come in under.

Galewatch, which collects a reading from each of nine hundred wind turbines every two seconds, has its ingestion API measured properly. p50 12 milliseconds, p95 45, p99 400, p99.9 2,100. Now read that ladder as a menu of timeouts.

The last column is that fraction of Galewatch's 450 readings a second, and the whole table is one idea: a timeout set inside the distribution converts slow into failed. Nothing is broken. The far end is answering. You have decided to stop listening to one call in a hundred and call it an error.

And you do not even get a clean error. Lesson 019 spent itself on that point: a call that times out has told you nothing at all, so those 4.5 a second are 4.5 unanswerable questions a second, every one of which may have succeeded. Then it gets worse in a way that is easy to miss, because lesson 017 established what a turbine does with a write that timed out. It sends the reading again a minute later. So those 4.5 failures come back as 4.5 extra writes, the offered load is 454.5 instead of 450, and higher load means higher latency, which times out a few more.

Whether that settles down or runs away has one answer, and it is lesson 001's hockey stick at eighty percent. Below the knee, each round of retries is smaller than the last and the thing converges on a slightly worse normal. Above eighty percent the curve is vertical, so a one percent bump in load buys a large jump in latency, which fails more calls, which adds more load.

Galewatch is nowhere near that knee. Lesson 013 put 28 writes a second on each of its shards and lesson 014 found the busiest of them takes seventy, against the ceiling of 5,000 a second that lesson 005 derived from a 20 millisecond write and a hundred connections. Even the busiest is at 1.4% of what it can do, so the retry loop is a rounding error and always will be. Stagefront at 10:00:34 was at seventy five percent CPU with a slow dependency, which is the other side of the knee entirely.

Same loop, same arithmetic, two completely different mornings.

So what should Marlow set?

The provider answers in about three hundred milliseconds on a normal day, which lesson 015 published. Ten seconds is thirty three times that. Nobody at the shop knows the provider's p99, and that is the finding rather than an embarrassment: you cannot choose a timeout without a number you probably have not measured, and a week of logging that one call is the cheapest engineering anywhere in this topic.

But notice what changed in January, because it is the thing lesson 019 bought and did not name.

Before the idempotency key, ten seconds was nearly defensible. Giving up early on a charge is how you get seventeen charges with no order behind them, so a shop with no key has to buy certainty with patience. It waits, because being wrong about whether the money moved costs an afternoon with a spreadsheet.

After the key, that trade is gone. The same call can be abandoned at two seconds and sent again with the same key, and the provider will not charge twice. Idempotency does not only make the retry safe. It makes the timeout choosable. The customer waits two seconds instead of ten for an answer that was never coming, and the shop keeps the guarantee it spent January building.

Marlow has not made that change yet. It should, after a week of measuring.

Patience has to shrink with depth

Back to Tuesday, and to the three numbers that made it.

Read the last column from the bottom up. The payment client, the deepest thing in the stack, is prepared to keep working on one charge for four and a half seconds. The API layer that called it gives up on the whole request after two.

So at the two second mark the API layer stops listening and starts a fresh attempt. The first attempt is still running. It holds a connection from that pool of a hundred, it is still occupying the provider, and the answer it eventually gets will be read by nobody, because the socket that wanted it is gone.

That is the mechanism that filled the pool at 10:00:52. Not volume, exactly. Work that nobody is waiting for.

That pool was in trouble before any of it, though. Lesson 002's busy room: five hundred charges a second at nine hundred milliseconds apiece needs four hundred and fifty connections in flight, and Stagefront has a hundred, so the slow provider on its own was enough to fill it. What the retries changed was the ending. A jam caused by slowness clears when the slowness stops, and the provider was answering normally again by 10:01:04. A jam that feeds itself does not, and that is the difference between thirty seconds of a bad dependency and thirty nine minutes of no sales.

The rule is dull and I would put it in a code review checklist tomorrow. Patience has to decrease as you go deeper, and the comparison is the inner layer's total against the outer layer's per attempt. Almost everybody compares the two per-attempt numbers, sees 1.5 against 2, and calls it correct. Multiply by the attempts and it is 4.5 against 2, and the deepest, least informed layer in the building is the most stubborn thing in it.

Stagefront got away with it through forty on-sales a year because the provider answered in a hundred and fifty milliseconds and the timeouts never fired. A timeout that has never fired is not a tested timeout. It is a line of configuration with an opinion in it.

There is a second half to this that costs nothing to build and is skipped almost universally: when the outer layer gives up, cancel the inner work. You do that by passing a deadline down the chain instead of a duration, so every layer is working towards the same instant. Go's contexts and gRPC's deadlines carry one for you; plenty of clients take only a per call duration, and then subtracting the time already spent is your job. Cancelling also stops at your own boundary. It frees your connection, and the far end carries on with the work, which is yesterday's lesson again. Marlow does not do this. Stagefront did not do this. Nearly nobody does it until the afternoon they read a pool graph and understand what is in it.

Retries multiply

One retry policy is easy to reason about. Say a call fails with probability p on each attempt and you try up to three times. The expected number of attempts per request is 1 plus p plus p squared.

On a good day the policy is invisible. It adds half a request a second to five hundred. On the worst day it triples your load. That alone is survivable at most places, and it is why single-layer retries have a good reputation.

Layers are what kill you.

They multiply rather than add. The phone app's three attempts each cause two API attempts, and each of those causes three charge attempts. Three times two times three is eighteen, and Stagefront's five hundred charges a second became nine thousand.

The general form is the one you have probably seen written on a whiteboard: N layers each retrying three times offers three to the power of N requests at the bottom. Four layers is eighty one. What the whiteboard version leaves out is the part that makes this genuinely nasty.

The multiplier switches on at the worst possible moment. It is one when everything is healthy and eighteen when everything is failing, because it is a function of the failure rate. Your dependency gets the smallest load you ever send it while it is well and the largest while it is drowning. It is a system that automatically hits its friends hardest when they are weakest, and nobody designed it, it just fell out of three reasonable local decisions.

This also makes the graph lie to you afterwards. The provider's inbound traffic chart for that Tuesday shows a vertical spike starting at 10:00:40, which reads exactly like a traffic surge that caused an overload. It was the overload's shadow. When you are looking at a dependency's load graph during an incident, the spike is as likely to be the consequence as the cause, and telling them apart means knowing your own amplification factor. Most teams cannot name theirs.

The layer nobody configures

Lesson 001 gave Marlow a Friday when a books podcast named the shop on air at 10:40 and pages took eleven seconds by 10:47. Then it said one thing and handed it to today:

Somebody who has watched a blank page for thirty seconds hits reload.

That is a retry layer. It has no backoff, no jitter, no budget and no cap, it is installed on every client you will ever have, and it is more reliable than anything you write. Lesson 017 counted a hundred and sixty thousand people at a Stagefront on-sale who were never getting a ticket. Not one of them accepts that on the first attempt.

Lesson 006 supplies the last twist. A balancer that routes by least connections sends work to the box with the fewest requests in flight, and a box that is failing instantly has almost none, which is why 006 called the fastest box the broken one. Your retry goes back through that balancer. It is preferentially routed to the machine least able to serve it.

What actually synchronises a herd

Lesson 018 built Marlow's backoff and I am not going to rebuild it: one minute, then two, four, eight, sixteen, thirty two, sixty four, a hundred and twenty eight, jittered. It also published the sentence to carry forward, that the sum of your backoff schedule is the outage you have decided to tolerate, and that you should retry to a deadline rather than to a count.

What today adds is the shape of the load that schedule produces.

Backoff lowers your average load on the far end and raises your peak. Suppose four thousand clients all fail within the same second, because the far end restarted or a link blipped. They all sleep one minute. Sixty seconds later, four thousand requests arrive inside one second, and between those two instants the far end sees nothing at all. Double the sleep and the gap gets longer while the spike stays exactly as tall. This is lesson 008's stampede with a different trigger: there, a shared TTL expiring at the same instant sent a herd at the database, and here a shared failure at the same instant sends it. Exponential backoff preserves that herd perfectly, because everybody in it is multiplying the same number.

Which tells you what jitter is actually for. It is not politeness. Pick each sleep uniformly between zero and the current backoff, and four thousand clients arrive spread over a minute instead of inside a second: sixty seven a second, which any far end can take.

Galewatch is the control case. Nine hundred turbines write independently on their own two second clocks, and a turbine resends a failed write a minute after its own failure, so the resends inherit the spread of the originals. No jitter, no herd. The turbines were never synchronised in the first place.

A herd comes from a shared cause, not a shared schedule. A restart, a deploy, a failover, a cache flush, a network partition healing: anything that fails everybody at the same instant. If your clients can only fail one at a time, you can skip the jitter. If a single event can fail all of them, jitter is the only thing standing between you and a square wave.

A budget instead of a count

Here is the fix I would ship first, and it is two counters.

Stop giving each request a retry allowance. Give the whole dependency one. Count the requests you send and the retries you send over a sliding window of ten seconds or so, and if retries are running above ten percent of requests, do not retry at all. Fail immediately and return the error.

What that buys is the thing the count could never buy. A per-request count of three means your offered load is between one and three times baseline and you do not get to choose which. A ten percent budget means your offered load is between one and one point one times baseline, always. Stagefront's nine thousand a second becomes five hundred and fifty a second, at every failure rate, for ever.

A retry budget makes the load you offer a dependency independent of that dependency's health. That is the exact property the count was missing, and the reason budgets beat every clever backoff curve I have tried.

The honest cost lands entirely above the threshold. Below a ten percent failure rate the budget never binds and every failure gets its retry. At fifty percent you can afford to retry one failure in five, and the other four are returned immediately. That feels wrong the first time you watch it in a dashboard. It is still the correct trade, because at that failure rate a retry is a coin flip that costs the far end another request, and that load is part of why the far end is failing.

Budgets also compose badly, in the sense that three layers with a ten percent budget each still multiply out to about thirty three percent. So the companion rule: retry in exactly one layer and turn it off in all the others. Which layer depends on who owns the deadline. My default is the outermost layer that knows what the request means and knows whether anybody is still waiting for it, and lesson 018's rule applies without modification, because retrying to a deadline needs somebody who holds one.

Almost nobody can say how many layers of their own stack retry. Finding out is an afternoon with a configuration search, and it is the highest value afternoon you will spend on any of this.

One more number to carry, from yesterday. Lesson 019 ended on the observation that your retry deadline and the far end's dedupe window are the same number seen from two buildings. Set the retry budget too and you have three numbers that have to agree, held by three teams, two of whom do not work for you.

When the far end comes back

The most dangerous moment in an outage is the recovery, and retries are why.

Marlow's cache went into a managed Redis in February, and lesson 008 priced what a restart of it costs. At forty requests a second on an ordinary Tuesday morning, nothing at all: lesson 008 calls that first restart a non-event. At Christmas week volume, 384 book page requests a second arrive at a cold cache, every one of them a miss, against a pair of boxes that can push 260 a second at Postgres.

Lesson 001 did that arithmetic in its first section. Four hundred offered against two hundred served has no steady state at all, and neither does 384 against 260. The queue does not settle somewhere unpleasant. It grows for as long as the traffic lasts.

Now put retries on top. The cache is empty, so it needs traffic to warm up, and warming up requires the origin to answer, and the origin is the thing that is saturated. A service that has just restarted has less capacity than usual, from cold caches and cold connection pools and whatever else it lazily builds, and more load than usual, from everybody who queued up while it was gone. Both numbers move the wrong way at once, which is how you get the outage that comes back four times in twenty minutes.

Lesson 004 promised you the client that listens. Here it is.

A 429 or a 503 often carries a Retry-After header, and that header is the far end telling you its own recovery schedule. Honouring it means the service that is recovering gets to control its own inbound rate, which is reasonable, because it is the only party in the conversation that knows whether it is ready. Ignoring it and using your own backoff means you come back at a time chosen by a constant in your code, and so does everybody else who copied the same constant.

Honour it. If a server takes the trouble to tell you when to come back, arriving earlier is not going to work out for either of you, and lesson 021 builds the limiter on the other side of that header.

The catch, and it comes straight from lesson 019: a server can only send you a Retry-After if it is well enough to answer. A dead server sends nothing at all, and a timeout has no status code. So the header helps in exactly the brownout case, the dependency that has gone slow rather than dark, which lesson 019 already told you is the dangerous one. A dependency that is down is safe. A dependency that is slow is the one that duplicates your charges, and now also the one that will take you down with it.

Stop asking

The last thing to say is the honest thing, and Phase 1 can only point at it.

None of what is on this page fixes a saturated dependency. Timeouts decide how long you wait. Budgets decide how much you add. Jitter decides when you arrive. Every one of them is a way of being a better-behaved guest at a party that is already on fire, and the only actual fix is to stop knocking on the door for a while.

That mechanism is the circuit breaker: notice that a dependency is failing, stop calling it entirely, and check back occasionally with a single request instead of all of them. Lesson 042 builds it, and it is where "a dependency that is slow" gets detected rather than merely survived. Lesson 041 has the other half, backpressure, which is how you make the pressure travel back up the chain instead of pooling at the bottom.

Lesson 017 gave you the frame to hold both of them in, and it applies to today with no adjustment. Every system that survives a dependency being unavailable already has a queue somewhere, whether or not anybody chose one. A retry loop is that queue. Its depth is your clients multiplied by their attempts in flight, it drains at whatever rate the far end can manage, and it is the only queue in your architecture that nobody has ever drawn, sized or put an alert on.

Recap

A timeout is a claim about somebody else's latency distribution. Set it inside that distribution and you convert slow into failed, one call in a hundred at the p99, and every one of those failures is lesson 019's unanswerable question rather than a clean error. If you cannot name the far end's p99, you have not chosen a timeout, you have inherited one.

Idempotency makes the timeout choosable. Without a key, a long timeout is how you buy certainty about whether the work happened. With one, you can give up in two seconds and send the same key again, which is the part of yesterday's lesson that pays for itself today.

Patience has to shrink with depth, and you compare totals. Three attempts of one and a half seconds is four and a half seconds of patience, and if the layer above gives up at two, the difference is work nobody will read, holding connections nobody can reuse. Pass a deadline down, not a duration.

Retries multiply across layers and the multiplier switches on at the worst moment. Three layers of three attempts is twenty seven times baseline, and you offer it precisely when the far end can least take it. That is also why a dependency's load spike during an incident is as likely to be the consequence as the cause.

A herd comes from a shared cause, not a shared schedule. Backoff lowers your average load and raises your peak, because everybody who failed together sleeps together. Jitter is what turns four thousand requests in one second into sixty seven a second for a minute.

A budget makes your load independent of the far end's health. Cap retries at a share of traffic rather than a count per request, retry in exactly one layer, and turn it off everywhere else. The count gives you a multiplier between one and three that you do not control; the budget gives you one point one, always.

The recovering service is the one you break. Less capacity than usual and more load than usual, at the same instant. Honour Retry-After, because the far end is the only party that knows when it is ready, and it can only send you that header while it is still well enough to answer.

Check your understanding

1. A service calls a search index with a timeout of 200 milliseconds and retries twice. The index's p50 is 30 milliseconds and its p99 is 350. Say what you can work out about how many of a thousand requests end up making three calls, what you cannot work out and why, and what you would change first.

2. Take the three layer table from the Stagefront section and fix it. Give each layer a per attempt timeout and an attempt count so that the totals decrease with depth and the whole purchase still fits inside lesson 003's two second promise, and say which layer you would let keep its retries.

3. Galewatch's turbines resend a timed out write one minute later with no jitter, and this lesson called that fine. Describe the change to Galewatch that would make it stop being fine, then say whether jitter or a budget would be the better answer to it.

4. Your service calls a payment provider that returns 503 with Retry-After: 120 during a brownout. Your own customer facing deadline is thirty seconds. Say what you do with that header, what you tell the customer, and why lesson 018's dead letter queue is or is not the right place for this request.

5. A colleague argues that retry budgets are dangerous because during a partial outage the requests that get no retry are picked more or less at random, so two customers doing exactly the same thing get different outcomes. Give the strongest version of that argument, then say what you would do about it without going back to a per request count.

Next lesson

021 Rate Limiting: Saying No Gracefully. Today was the client's side of the contract lesson 004 wrote; next lesson is the server's, and it turns out that saying no early and clearly is the most generous thing an overloaded service can do for the people calling it.