Stagefront is a ticketing service that exists only in this course, where a stadium show goes on sale at exactly ten in the morning and two hundred thousand people press the same button in the same minute. Three weeks after the Tuesday in March that lesson 020 is built on, somebody ran the review, and one action item came out of it with everybody's name next to it: add a circuit breaker to the payment client.
An engineer took it and spent an afternoon on the arithmetic, which is more than most action items get.
Here is what they found. The payment client gives each charge one and a half seconds. At 10:00:34 the provider's median went from a hundred and fifty milliseconds to about nine hundred and stayed there for roughly half a minute. Nine hundred milliseconds is comfortably inside one and a half seconds. Every single call succeeded, on the first attempt, with a 200 and a charge reference, right up to the moment Stagefront's own pool jammed.
So a breaker counting failed calls, wrapped around the charge the way that action item meant it, would have counted none. Not a low number. Zero, for the eighteen seconds between the provider slowing down and the pool of a hundred connections jamming solid at 10:00:52, which is the whole of the stretch in which the morning was decided. Lesson 020's own concession is the reason: five hundred charges a second at nine hundred milliseconds apiece wants four hundred and fifty connections in flight, and the pool holds a hundred, so the slow provider on its own was enough to fill it. The provider's error rate stayed at zero. The provider's latency stayed inside every timeout anybody had configured. Stagefront sold nothing for thirty nine minutes.
The engineer closed the action item with a sentence that got quoted for a while afterwards: a breaker here would have watched the wrong number.
They were right, and the version of this lesson that stops there is useless, because something in that building absolutely should have given up and refused. Working out what it should have watched is most of the work.
Three states, and the one everybody has backwards
A circuit breaker is an object that sits in front of a call and keeps a count. It has three states.
CLOSED ---- too many failures ----> OPEN
^ |
| | after a wait
probe ok v
+------------------------------ HALF OPEN
|
probe fails
|
v
OPEN
Closed is the working state. Calls go through, results get counted. Open is the broken state: calls are not attempted at all, they return an error immediately, and nothing touches the network. After a wait the breaker goes half open and allows a small number of trial calls, usually one. If the probe succeeds the breaker closes. If it fails, back to open and the wait starts again.
The names come from electrical circuits, where a closed circuit is the one with current flowing, and roughly half the people in any room have them the other way round. You will too, at least once, in a postmortem, out loud.
Four decisions turn that picture into code, and every one of them is a place to be wrong.
What counts as a failure. The breaker is asking one question: can this dependency answer me. A timeout counts. A 500 counts. A connection refused counts. A 402 for a declined card does not, because the provider answered correctly and the answer was no. Count declines and a run of bad cards closes your checkout. The harder case is 429, lesson 021's refusal, which is a correct answer that also means stop calling: when the far end tells you when to come back, honour its Retry-After and do not guess with a breaker on top.
The window. Last N calls, or last T seconds. The two behave completely differently at low traffic, which is where most of the damage lives.
The threshold. A count of consecutive failures, or a percentage of the window.
The wait, and the probe. How long open lasts, and how many calls get to decide.
Marlow Books is an online bookshop run by four people that exists only in this course, and it is the case where all of this plainly works. On the second Tuesday in September, lesson 027 watched its payment provider accept every connection and answer none of them from 14:05 to 15:35. The founder noticed at 14:20, spent twenty eight minutes editing a refusal into the checkout page, deployed at 14:48, and had the banner live at 14:50. That leaves forty five minutes in which a customer who pressed Pay got lesson 019's ten second timeout and then an apology.
At lesson 027's rate of an order every fifty seconds, forty five minutes is fifty four attempts. Fifty four times ten seconds is five hundred and forty seconds of somebody watching a spinner. Nine minutes of human attention, spent on a decision that was already made.
Put a breaker in front of that call, opening after five consecutive failures, and the first five customers pay ten seconds each and the other forty nine get their answer in about a millisecond, because the breaker is a conditional in the same process. Fifty seconds instead of five hundred and forty.
The breaker saved eight minutes of spinner and it saved exactly zero sales. The money could not be taken at 14:06 and it could not be taken at 14:49, and the breaker changed nothing about that. What it changed was how long each person spent finding out, and how much of the shop was tied up finding out with them.
That second half is where the real money is, and at Marlow it rounds to nothing, because an order is one request in two thousand and the checkout path was never going to exhaust anything. At Stagefront it is the entire incident.
Count what you are running out of
Go back to the pool of a hundred.
Between 10:00:34 and 10:00:52 the provider was healthy by every definition a breaker understands. What was not healthy was Stagefront. Lesson 002's busy room says concurrency is throughput times latency, so a hundred connections at nine hundred milliseconds apiece can finish a hundred divided by nine tenths, which is about a hundred and eleven charges a second. Demand was five hundred. The gap, three hundred and eighty nine charges a second, had nowhere to go, so it went into a buffer nobody had typed a bound for, which is lesson 041's whole subject. In this case the buffer is the wait list for a connection.
A breaker watching connections in flight would have crossed any sane threshold almost instantly. In flight at the healthy rate is five hundred times nought point one five, which is seventy five. To get from seventy five to a hundred at five hundred arrivals a second takes fifty milliseconds. The error breaker sees nothing for eighteen seconds. The saturation breaker sees everything in a twentieth of one.
Count what you are running out of, not what they are doing wrong. Your error rate is a measurement of somebody else's health. Your concurrency is a measurement of your own, and your own is the one that ends the on-sale.
This is lesson 021's signal arriving from the other side. Lesson 021 argued that a load shedder should watch requests in flight rather than CPU, because in flight is the quantity that saturates first. The same argument, aimed outward at a dependency instead of inward at your own front door, gives you an input that would have fired in March.
One other input would have fired, and it is worth knowing it exists: several breaker libraries will also count a call that merely takes too long, under a name like slow call rate, with its own duration threshold underneath the timeout. Set that at four hundred milliseconds and March trips at 10:00:35. It works, and it is still a number about the far end, so it only works if you guessed the provider's bad day in advance and guessed low. Your pool size is a guess you have already made and written down.
There is a second, nastier reason the error breaker is blind here, and it depends on something nobody writes in a design document: whether the timeout you configured covers waiting for a connection, or only the request once it has one. Two HTTP clients in the same language will disagree. If your one and a half seconds starts when the socket is acquired, then a call that waits four seconds for a connection and then runs in nine hundred milliseconds is recorded as a four point nine second success. Your breaker is on the far side of the thing that is actually full.
So: put the breaker outside the pool, or it never sees the failure that matters. If you do not know which side yours is on, that is a ten minute experiment with a pool size of one, and it is worth the ten minutes.
Move it outside and it stops being blind. Look at what it is watching, though. The wait list in front of the pool grows at three hundred and eighty nine a second against a hundred and eleven coming out, so it holds one and a half seconds of queue about four tenths of a second after the pool fills, and the first timeouts are recorded around 10:00:36. Two seconds instead of eighteen is a real improvement. It is also your own saturation arriving a second and a half late with somebody else's name on it, when the number underneath it was already at the ceiling at 10:00:34 and nothing stopped you reading it then.
Lesson 019 has been waiting for this paragraph for twenty three lessons. A dependency that is down is safe, and a dependency that is slow is the one that duplicates. The down case produces errors, trips any breaker you own, and is over. The slow case produces successes, trips nothing, fills your pool, times out your callers, and leaves you with lesson 019's unanswerable questions. Lesson 020 named it a brownout. A breaker built out of error counts is tuned precisely for the failure that was never going to hurt you.
A bulkhead is a number
At a hundred and eleven charges a second, Stagefront could still sell. Not well, but twenty two percent of five hundred is a long way from nothing. A breaker is all or nothing by construction: it is open or it is closed, and open refuses everybody. Half open admits one call, which is a diagnostic and not a serving rate. Over the thirty seconds the provider was slow, a breaker that fired at 10:00:34 would have sold nothing at all, which is the same outcome as the incident, arrived at faster and on purpose.
What you want is the bound. A pool of a hundred connections with no wait list, which refuses the hundred and first caller immediately instead of queueing it, serves a hundred and eleven a second throughout and refuses the rest in a millisecond apiece. That is a bulkhead: a fixed share of a resource, assigned to one dependency or one caller, with an instant refusal when the share is gone.
Run the numbers. Seventeen thousand seats were already sold by 10:00:34. At a hundred and eleven a second for the thirty seconds of slowness, another three thousand three hundred go out, leaving about nineteen thousand seven hundred. The provider was answering normally again by 10:01:04, and the remaining nineteen thousand seven hundred at five hundred a second is thirty nine seconds. The show sells out at about 10:01:43 instead of 10:41.
The mechanism behind that forty minute difference is lesson 020's, not mine. A jam caused by slowness clears when the slowness stops; a jam that feeds itself does not. The three hundred and eighty nine a second that cannot get a connection are what fed it, because each one held an upstream request until the API layer gave up at two seconds and started a fresh attempt. Refuse them in a millisecond and there is nothing left to feed on. The provider only ever sees a hundred and eleven a second, and at 10:01:04 the slowness stops and the jam goes with it.
That is not free and I am not going to pretend it is. Three hundred and eighty nine refusals a second, multiplied by lesson 020's retry layers, land on Stagefront's own twenty application boxes, which lesson 006 says are already at seventy five percent CPU in that minute. And every refused charge is a claimed seat that has to be released, which is lesson 039's compensation running at a few hundred a second against the one database lesson 013 refused to shard. A bulkhead moves the bill. It does not delete it.
Notice what the bulkhead does not need. No threshold, no window, no probe, no idea what a failure is. It is a bound on concurrency, and a bound on concurrency is self enforcing: when the share is gone, the refusal is the truth rather than an inference from a count. A bulkhead is a breaker that trips instantly, with exactly the right blast radius, and nothing to tune. Given one of the two, take the bulkhead. The breaker is what you add afterwards, when refusing fast is not enough and you also want to stop calling.
Galewatch, which collects readings from nine hundred wind turbines and exists only in this course, has the course's clearest example of a bulkhead missing and then appearing by accident.
Lesson 017 is the October afternoon one farm's database ran out of connections and the four ingest boxes shared a single pool of request workers across all nineteen farms. A worker blocked on the backlog from Tarrow Ridge, a sixty turbine farm whose link had just come back after four hours down, is a worker the other eighteen farms cannot have. Nineteen farms, one pool, one farm's storm, and twenty two minutes in which eighteen innocent farms timed out and resent. The blast radius of a slow dependency was everything.
Then lesson 037 put a log in the middle, for reasons that had nothing to do with any of this. The ingest boxes now append to a broker and return a 200 in two milliseconds, so a slow Postgres machine cannot hold a request worker any more. Nobody at Galewatch called that a bulkhead. It is the best one in the system.
The honest half: it moved, it did not vanish. Lesson 037's log is twenty four partitions drained by six writer processes, which is four partitions each, and each writer holds twenty database workers for all four. One slow farm can still take its three neighbours down, the same way, one radius further in. So Galewatch's blast radius went from all nineteen farms to the four partitions one writer owns, which on lesson 037's one farm per partition is four farms.
Four, not one. A bulkhead is a number, and the number is how many innocent things go down with the guilty one. Nobody at Galewatch has ever written that number down, and the only reason it is four rather than nineteen is a partition count chosen for throughput.
A breaker needs a sample rate
Marlow should not put a circuit breaker on its payment provider, and the reason is arithmetic rather than taste.
The shop takes one order every fifty seconds, which is one point two calls a minute to the provider. It runs eight application processes, four on each of the two boxes it has had since lesson 005. A breaker is state, and state in a process is state per process, so what the founder would actually install is eight breakers, each seeing one call every six minutes and forty seconds.
Now set a percentage threshold on a sixty second window. Almost every window in that process is empty. The windows that are not empty contain exactly one call. One failure out of one call is a hundred percent, so any percentage you choose, including ninety nine, opens the breaker on a single timeout. Switch to consecutive failures and you have the opposite problem: five in a row at one call per four hundred seconds is thirty three minutes of evidence gathering, by which time the outage is somebody else's story.
Lesson 021 found the same shape and the mirror image of it. A rate limiter inside the application multiplies your limit by the size of the fleet, because each copy counts separately. A breaker inside the application divides its evidence by the size of the fleet, for exactly the same reason. One number per process is too generous in one case and too thin in the other, and in both the fix is to move the state somewhere shared, which for a breaker means the gateway lesson 049 owns.
A breaker is a statistic, and a statistic needs a sample rate. Before you install one, divide calls per minute by processes and ask whether the number that comes out can carry a percentage.
Which Marlow can answer with its own equipment, because the shop has had a circuit breaker since lesson 006's January configuration and nobody ever used the words. Its balancer runs outlier detection: watch the responses, eject a box that starts producing errors or timeouts, let it back in tentatively after a while. In, out, tentatively back in. Closed, open, half open, for twenty dollars a month.
That one works, because it is swimming in evidence. Forty requests a second gives the balancer twelve hundred responses per box per minute to judge each box on. The breaker the founder is thinking about would see one call every six minutes and forty seconds, which is fifteen hundredths of one call a minute. Same three states, same logic, eight thousand times the sample, and that is the whole of the difference between the breaker Marlow already owns and the breaker Marlow should not buy.
And then the case where no threshold is the right threshold. On the Wednesday of the founder's week off in February, lesson 035 watched the provider go slow rather than dark: about one checkout in five crossed the ten second timeout and the other four went through. Pick any threshold under twenty percent and the breaker opens, and opening refuses four working customers to spare the fifth. Pick anything over twenty percent and it never fires at all. There is no number in between that helps, because the dependency is not broken. It is degraded, and a breaker only knows two words.
I do not have a clean fix for that and I have not met one. The two things February actually needed are both still proposals: lesson 035's (job_name, run_slot) key, so two boxes cannot start the same run, and the cap on rows per run that lesson 041 argued for. Neither is a breaker. If your dependency's failure mode is a brownout, spend the afternoon on your own queueing instead.
The breaker deletes the signal that opened it
Once a breaker is open you have stopped calling, which means you have stopped measuring.
Lesson 026 gave Marlow five graphs, and the founder found September's outage on one of them, fifteen minutes late and only because it happened to be open. Install a breaker and that graph gets better looking and less useful: the provider's latency series goes empty, because no requests are being made, and the error rate goes to zero for the same reason. The checkout path gets faster. Everything on the dashboard improves at the moment the shop stops working.
The half open probe is not only a recovery mechanism. It is your entire remaining measurement of that dependency, at one sample per wait period. A breaker that opens for five minutes has quietly reset your monitoring of the most important third party in your system to one request every three hundred seconds.
Two things follow and both are cheap. Alert on the breaker's state and on its transition count, never on the error rate of a call you have stopped making. And treat the open period as a sampling interval that you are choosing, not just a recovery delay, which is lesson 006's panic threshold wearing different clothes: never let the thing refuse absolutely everything, because a mechanism with no measurement left cannot find out it was wrong.
Write the fallback first
Lesson 027's spine was that if you do not choose your degraded mode, your timeouts choose it for you, and the mode they chose at Marlow was ten seconds of spinner and then an apology. A breaker does not change the mode. It changes the speed at which every customer arrives at it, from ten seconds to one millisecond, which means a mode you have never looked at is now in front of everybody at once.
So the order matters, and it is the opposite of the order in which people ship these.
The fallback goes in first, and on its own it is already useful. Marlow has one: lesson 027 built a two field switch read from a file on each box, and lesson 028 widened it to a checkout field holding "open", "closed" or "delayed". Thirty seconds over ssh, no deploy, no database involved, so it survives the mode where the database is the problem. The founder can enter the degraded mode by hand at any time, which means they can rehearse it, which means somebody has seen it.
The breaker goes in second, and all it does is move that switch without being asked. A breaker is a hand on a switch you already have. Install the hand before the switch and what you have built is a very fast way to show your customers a stack trace.
What the switch should say is a product question and not yours to settle, but lesson 027 left the test: fail open on a suggestion, fail closed on a decision. The breaker version is the same split with a sharper edge, because now the failure is instant and therefore visible. Recommendations gone: render the page without them and say nothing. Payment gone: say so, in a sentence, on the button, before the card number is typed rather than after. Lesson 027's rule was that the failure with a receipt beats the failure with no receipt, and a breaker's refusal is the cheapest receipt in the building.
There is one more thing a breaker buys that nothing else does, and it is the reason to have one even at Marlow's one point two calls a minute. An open breaker is the system writing down, in a place anybody can read, that it has decided a named dependency is unavailable. That sentence used to live in a founder's head between 14:05 and 14:20.
Lesson 029 is where the best version of this argument already sits: the strongest reason to verify a token locally is availability, not speed. The cheapest circuit breaker is the call you deleted.
Recap
Count what you are running out of, not what they are doing wrong. Their error rate is a fact about their health. Your concurrency is a fact about yours, and it is yours that ends the on-sale. Stagefront's provider never returned a single error in the eighteen seconds that killed the morning, because nine hundred milliseconds fits inside a one and a half second timeout.
A breaker is a statistic and a statistic needs a sample rate. Divide calls per minute by the number of processes before you choose a window. Marlow's balancer judges each box on twelve hundred responses a minute and a per process payment breaker would get fifteen hundredths of one call, which is why one of those breakers is right and the other is a coin toss with a shop attached.
A bulkhead is a breaker that trips instantly, with the right blast radius and nothing to tune. A bound on concurrency enforces itself. Given one of the two, take the bulkhead; the breaker is what you add when refusing fast is not enough and you want to stop calling as well.
A bulkhead is a number, and the number is how many innocent things go down with the guilty one. Galewatch went from nineteen to four without anybody deciding to, because lesson 037 put a log in the middle for unrelated reasons. Four is better than nineteen and it is not one, and nobody there has written it down.
A breaker deletes the signal that opened it. The latency series empties, the error rate goes to zero, and the dashboard improves at the exact moment the feature stops. The half open probe is your whole remaining measurement, so alert on the breaker's state and never on the errors of a call you are no longer making.
A breaker is a hand on a switch you already have. Ship the degraded mode first, by hand, with a file on each box and no deploy, and rehearse it. Then add the hand. In the other order you have built a very fast way to show somebody a stack trace.
And the one that binds the lesson to lesson 019: a dependency that is down is safe, a dependency that is slow is the one that duplicates, and an error counting breaker is tuned exactly for the first.
Check your understanding
Your service calls a search provider two hundred times a second through a pool of fifty connections, with a one second timeout. The provider's median goes from eighty milliseconds to six hundred and stays there. Say what a breaker counting errors sees in the first ten seconds, what a breaker counting connections in flight sees, and which number you would put the threshold on.
A colleague proposes one breaker per third party service, keyed on the hostname. Your checkout calls two endpoints on the payment provider: a charge, once per order, and a status lookup, twelve times per sweeper run. Say what that breaker's failure rate is actually measuring, and what you would key on instead.
Galewatch is asked to guarantee that one farm's slow database can never delay another farm's readings. Say what you would change, what it costs, and which number from today tells you whether you have finished.
Your breaker has been open for four minutes against a dependency your team believes recovered three minutes ago. Say how you would find out whether that belief is true, without closing the breaker, and say what that tells you about the open period you chose.
A payment provider starts returning 429 with a
Retry-Afterof thirty seconds for about a third of your calls. Say whether this should open your breaker, what you would do instead, and which earlier lesson owns the other side of that conversation.Marlow's founder reads this lesson and asks you for the smallest useful change to the shop this week. Say what you would ship, say explicitly what it does not fix, and name the February incident it would not have prevented.
Next lesson
043 Storage Engines: B-Trees vs LSM Trees. Today was about deciding a dependency is unavailable and saying so; next lesson goes under the dependency Marlow has leaned on since lesson 001 and asks how a database actually puts a row on a disk and finds it again.