Marlow Books is a four person online bookshop that exists only in this course, and on a Wednesday evening in November its founder spent six minutes proving to a customer that the shop was open.
The email arrived at 20:04. Your site is down, it said, which is the least useful sentence in engineering. The founder opened the shop on their phone. It loaded in under a second. They took a screenshot, sent it back, and got a screenshot in return: a spinner, then a browser page saying the site could not be reached.
Both screenshots were honest.
Lesson 005 left Marlow with two servers and no machine in front of them. Traffic was split by round robin DNS, which means the domain has two address records and resolvers hand them out in turn. Box A was fine. Box B had stopped answering at 19:40, when the underlying host lost its network. Not crashed, not refusing, just silent. The founder's phone had been given box A. The customer's had been given box B, and would keep being given box B, because that answer was already sitting in their resolver's cache.
At 20:10 the founder deleted box B's address record. That is the fix, and it did almost nothing for a day. The TTL on those records was 86,400 seconds, the registrar panel's default from lesson 004, so every resolver already holding the pair went on handing out a dead machine until its own copy expired, which for the unluckiest of them was 20:10 on Thursday.
How box B died decided how much it cost. A machine whose process has exited still has a kernel, and that kernel answers a connection attempt with an immediate refusal, so a browser gives up on that address in a millisecond or two and tries the other one in the list. Almost nobody would have noticed. Box B did something worse: it went quiet. Nothing answered the connection attempt at all, so the client sat there retrying into silence. Linux, left alone, spends about two minutes on that before it admits defeat. Browsers cut it shorter and do move on to the second address, but the customer still watched a spinner for many seconds first, and plenty of non-browser clients never try the second address at all.
Roughly half of new visitors at 20:00, decaying through Thursday as caches expired one by one. Marlow's dashboards showed nothing wrong, because every request that reached a Marlow server was served perfectly.
The machine that fixes this costs about $20 a month. What it needs to know is a much longer list than anyone expects.
What a balancer is, and where it sits
A load balancer accepts a request and chooses which of several machines answers it. That is the whole job. To do it, it has to know two things: which boxes are alive, and which live one this request should go to.
The first question is where the outages live. The second is where the blog posts live.
It can sit in three places. It can sit in DNS, which is what Marlow has: free, no extra hop, no extra thing to fail, and completely blind. It can sit in front of the fleet as a box you run or a service you rent, which is what almost everyone means by the phrase. Or it can sit inside the caller, where the client gets a list of addresses and picks one itself, which is the usual arrangement between services inside a single system and is really lesson 048's territory.
The rented one comes in two depths.
A layer 4 balancer forwards connections. It knows addresses and ports and it does not read what is inside. It is cheap and fast, and it can pass anything.
A layer 7 balancer terminates the connection, reads the request line and the headers, and then makes its choice. It can route /search to one pool and /checkout to another, retry a failed request against a different box, add a header, and count things. It does more work per request, which means it can become the bottleneck itself, and it understands only the protocols it was built for.
The extra hop costs about a millisecond inside one data centre. Set that against lesson 004's 180 milliseconds of setup before the first byte on a distant link and it is noise. Set it against an internal call that already takes 2 milliseconds and it is fifty percent, which is why you will meet people who are strangely passionate about layer 4.
There is a quieter win. The balancer usually terminates TLS, which means 004's certificate lives in exactly one place instead of on every box. Marlow's March Sunday, when a certificate expired at 05:30 and the shop was unreachable for over six hours while every graph stayed calm, becomes a single renewal that a managed balancer does on its own. The founder had renewed that certificate by hand six times.
And the honest cost. Lesson 003's rule was that downtime adds along the request path, and you have just added a component to it. Lesson 005 priced a balancer at 99.99% as 0.876 hours a year, about 53 minutes. If you run one balancer, you have moved lesson 001's single point of failure rather than removed it.
Which boxes are alive
A health check is the balancer asking each box, on a fixed interval, whether it should be sent traffic. A box that fails some number of checks in a row comes out of the pool. It keeps being asked, and after some number of passes it goes back in.
Marlow's first attempt was a GET /health that returned 200 with the body ok, checked every 2 seconds, 1 second timeout, out after three failures, back in after two passes. That check would have caught November in six seconds, because box B answered nothing at all.
Most health checks are wrong in one of three directions, and the second one is the dangerous one.
Too shallow
Lesson 004's certificate Sunday had a health check running all morning: a curl of http://localhost:8080/health from inside the box. It never once left the machine, so it never met the thing that was broken. A check that proves a process is running has proved that a process is running.
The shape recurs even after you move the certificate to the balancer. A box whose database connection pool is exhausted will answer /health in 200 microseconds and cannot serve a single book page. The check has to touch something the real request touches, or it is decoration.
Too deep
So make it deeper. Have the check run a real query.
Now every box in the fleet is coupled to the same dependency. Marlow's Postgres lives on box A, which lesson 005 established and lesson 011 will attack. Give both boxes a health check that runs a query against it, and on the afternoon Postgres stalls for twenty seconds behind a long lock, both boxes fail their check inside the same two second interval. The balancer does precisely what you configured: it removes the entire fleet.
A dependency that was making the shop thirty percent slower has now made it one hundred percent absent. Lesson 003 called this the shared cause is the ceiling. A health check that queries a shared dependency is a machine for converting partial failures into total ones.
There is a good defence and it is one line of configuration. Balancers can be told to stop believing the checks when too many fail at once: if fewer than half the fleet is healthy, serve everything anyway, on the reasoning that a whole fleet failing in the same second is far more likely to be a broken check than a genuinely dead fleet. Envoy calls it a panic threshold and sets it at half the hosts by default. Turn it on wherever you have it. The whole fleet going out together is one of very few failures where the correct move is to disbelieve your own instruments.
Too aggressive under load
Stagefront, this course's ticketing service, sells stadium seats that go on sale at exactly 10:00, when two hundred thousand people press the same button inside the same minute. Twenty application boxes carry that minute, and by 10:00:20 they are running at 75% CPU, which lesson 001 would already call the steep part of the hockey stick.
Health check every 2 seconds, 1 second timeout, three strikes. One box catches a burst and answers its check in 1.1 seconds, three times running. Out it goes, and its share of the traffic goes to the other nineteen.
20 boxes at 75% remove one
19 boxes at 78.9% the next one tips
18 boxes at 83.3%
17 boxes at 88.2%
16 boxes at 93.8%
15 boxes at 100%
Thirty seconds, five boxes gone, and not one of them was ever broken. The health check ate a quarter of the fleet during the only minute of the year that pays for the whole system. The panic threshold does not save you here either, since fifteen healthy of twenty is well above half; it stops the wipeout, not the spiral.
Two cheap fixes. Serve the check from somewhere that does not queue behind real work, a separate thread or a separate port, because a check that waits in the same queue as the requests is measuring queue depth, and you have built a system that removes boxes for being popular. Then set the timeout against your p99 under load rather than your p50 at three in the morning. Stagefront's checkout p99 in the on-sale window is over a second, which lesson 003's two second bound already allows for, so a 1 second check timeout was never going to survive contact with 10:00.
Liveness and readiness are different questions
Go back to Marlow's podcast Friday in lesson 001. At 10:52 a health check timed out, the process manager decided the app had hung, restarted it, and killed every checkout in flight.
That check was answering the wrong question. There are two:
Is this process broken beyond recovery, so that restarting it is the only thing left? That is liveness, and the answer is a kill.
Should this box be sent traffic right now? That is readiness, and the answer is removal from the pool while the process keeps running.
A box that is slow because it is busy is not broken. Readiness lets it work through its queue and come back on its own when the burst passes. Liveness takes its in-flight work down with it. Marlow lost real baskets at 10:52 to that confusion. Go and check which one your process manager thinks it is doing.
Watch the traffic, not just the endpoint
The best signal about a box is the traffic it is already serving. A box returning 500s to real customers while answering /health with a cheerful 200 is common, because the check does not touch the code path that broke.
Balancers can watch the actual responses and eject a box that starts producing errors or timeouts, then let it back in tentatively after a while. Envoy calls it outlier detection; most others have the same thing under a different name. Active checking asks "should I send you traffic". Passive checking asks "were you any good with the traffic I already sent". You want both, and the second is the one people leave switched off.
Marlow's check ended up as: the process answers, and it can check out a database connection from its pool within 100 milliseconds. Which puts a number the whole fleet shares at the centre of a per box question. Lesson 001 gave Marlow four application processes holding pools of 25 connections each, against Postgres's max_connections of 100. Two boxes of four processes would have asked for 200, so the pools were cut to 12 the week box B arrived and the fleet asks for 96 of the 100. A connection pool is a fleet wide number wearing a per box disguise, and every machine you add divides it again. Lesson 010 has the pooler that stops this being your problem.
Which box gets this request
| Algorithm | Decides using | Where it hurts |
|---|---|---|
| Round robin | position in a rotation | boxes or requests that differ |
| Weighted round robin | weights you set by hand | the weights go stale in silence |
| Least connections | requests in flight per box | a box that fails fast |
| Hash of a key | address, header or path | skewed keys, and resizing the fleet |
Marlow finished the year on two boxes at sixteen vCPUs and 64 GB. Lesson 005 measured that pair at about 260 book pages a second, and a third box bought five more, because Postgres never left box A. Christmas week peaks at 400 requests a second, well above that, and the shop survives it on the cache lesson 001 put in front of book pages the week after the podcast Friday. Most pages never reach Postgres.
So the balancer sees a mix. About 96% of those 400 are book pages, mostly cache hits costing a couple of milliseconds. The other 4% are full text searches over 1.2 million descriptions, which no cache helps with because every query is different, and they cost about 60 milliseconds. One request in twenty five costs thirty times what the rest do.
Round robin handles that mix fine: two hundred a second each, and over any reasonable window each box gets the same share of the expensive searches. The problem with round robin is not the average. It is that round robin is open loop: it hands out its share and never looks up to see what happened.
February proved it. Box B's virtual disk started degrading, the way rented hardware sometimes does, and everything it served went from about four milliseconds to 400. Not the queries, since box B does not run Postgres, but every log line the application wrote. Nothing crashed, and the health check passed every two seconds all afternoon.
Each box runs four application processes with twelve workers, one per pooled database connection, so 48 requests can be in flight at once. Lesson 002's busy room says throughput is concurrency divided by latency, so box B at 400 milliseconds a request can finish 48 / 0.4, which is 120 a second, flat out.
Round robin sent it 200.
Lesson 001's rule was that once arrivals exceed the service rate there is no steady state at all, just a queue that grows until something gives. Box B's queue grew for the rest of the peak and everything routed to it eventually timed out. Box A served its own 200 a second without noticing.
Half the customers got a broken shop. The average response time looked poor rather than alarming. A round robin balancer with a passing health check will do this happily until you stop it.
Least connections asks a different question: which box is holding the fewest requests right now? It does not need a theory about box B. Box B's in-flight count climbs to its ceiling and stays there, so it stops being chosen, and it settles at the 120 a second it can genuinely serve. The remaining 280 go to box A, which can take them: the expensive half of a search happens inside a Postgres that was on box A either way. Nobody configured anything, nobody was paged, and the shop stayed up at a slightly worse average.
That is the reframe worth keeping. Round robin is open loop and least connections is closed loop: it uses each box's own behaviour as the signal, so it copes with causes you never anticipated. For anything with variable request costs, which is nearly every system with a search box in it, least connections is the better default and it is usually one line of configuration.
Now the part that is not in the sales material.
The fastest box is the broken one
Suppose box B breaks the other way. Its pool is empty, or its database connection is gone, and it starts returning HTTP 500 in 2 milliseconds.
Least connections looks at in-flight counts, sees box B holding almost nothing, and sends it the next request. And the one after that.
Hold the in-flight count equal on both boxes and each box's throughput is that count divided by its latency, so the rate a box attracts is inversely proportional to how long it takes.
box A: 5 ms per request, healthy
box B: 2 ms per request, every one an error
share to B = (1/2) / (1/2 + 1/5) = 71%
Seventy one percent of your traffic to the machine that is failing, precisely because it is failing. Every algorithm that routes towards speed has this hole: least connections, least response time, and every clever variant of them will find your broken box and feed it.
This is why the passive checking two sections up is not optional. Least connections plus outlier detection is a sound default. Least connections on its own is a loaded gun pointed at whichever box breaks first.
Hashing is the remaining family: send the same key to the same box, every time. It buys cache locality, which lesson 047 cares about, and it is how sticky sessions get implemented. It has two sharp edges. Keys are skewed in ways you did not plan for: a source address behind a mobile carrier's network address translation is thousands of separate people. And adding or removing one box reshuffles nearly every key, which is a big enough problem that lesson 031 exists to solve it.
Sticky sessions, and why you should not want one
A sticky session means the balancer sends a given user to the same box on every request, using a cookie it sets or a hash of their address.
Lesson 005's shared nothing rule says you should not need one. Every sticky session is a small bet that a particular box stays alive, and when it does not, those users lose whatever was living on it. That is the November outage again, applied to a subset of customers, and now it is a design decision rather than an accident. Stickiness also defeats least connections, because a box that happens to hold a busy cohort stays hot and the balancer is not allowed to help it.
Two honest reasons remain. A long lived connection, a WebSocket or a streaming response, cannot move once it is open, which is less stickiness than physics. And a warm per box cache can be worth real money, which is lesson 047's argument to make.
Marlow needs neither. Lesson 007 moves the session into shared storage and the question stops existing.
Deploying without dropping anyone
The founder deploys on Friday afternoons. With one box that meant stopping the app and losing whatever was in flight. Two boxes and a balancer should make it invisible, and usually it does not, because of a race almost everyone runs into once.
Connection draining is the balancer's answer: when a box is taken out, stop sending it new connections and let the ones in flight finish, up to a deadline. Thirty seconds is a common default. For Marlow's millisecond book pages, thirty seconds is an eternity. For a system with genuinely long requests it is not enough, and at the deadline the balancer cuts the connection and your customer gets exactly the failed checkout you were trying to prevent. Set the deadline against your real p99.9, and know what happens when it expires.
The race is in the ordering. The sequence people write is: the deploy script stops the app on box B, the balancer's checks start failing, and three failures later the balancer removes it. Three failures at a two second interval is six seconds, and at Christmas week's 200 a second per box that is about 1,200 requests handed to a machine with nothing listening. Draining did nothing at all, because draining started after the box was already dead.
Tell the balancer first. Fail the readiness check on purpose or deregister the box, wait for the balancer to actually notice, which is your check interval times your failure threshold, so 2 seconds times 3 is 6 seconds, and only then stop the process. That wait is a literal sleep in a shell script and it is the entire trick. Kubernetes has a preStop hook that exists for this and almost nothing else.
Six seconds per box, twice a week. Lesson 003 counted Marlow's 104 deploys a year at 90 seconds each as 2.6 hours of planned downtime, six percent of the annual budget. Done in this order it becomes zero user visible seconds, and nothing anywhere got faster. The removal just became orderly.
The balancer needs a balancer
You have added a component to lesson 003's chain, so it has to be redundant or you have accomplished less than you think.
A managed balancer is already a fleet the provider runs behind one name, and that name resolves to addresses that change underneath you. Which puts you back at lesson 004's TTL dial, with the provider holding it at something short like 60 seconds, and it is why the provider's documentation insists you use the name rather than an address you looked up once.
Running your own means running two, and then giving them one address between them that can move from the failed one to the survivor, or announcing the same address from both machines and letting the network's routing pick. Both work. Both are more operational load than a four person bookshop should be carrying.
Notice the recursion, though, because it is the honest end of this lesson. Whatever you put in front, something still has to send traffic to the front. You cannot balance your way to the bottom. The chain has to end somewhere, and it ends at DNS or at anycast routing, which is to say at 004's TTL or at the internet's routing table, and neither of those belongs to you. Which is not an argument against the balancer, but the reason lesson 004 spent so long on a number in a registrar panel.
Marlow's January: $20 a month, both boxes behind a managed layer 7 balancer, TLS terminated and renewed there, least connections with outlier detection on, readiness checking the connection pool, a six second wait before any deploy stops a process, panic threshold at half the fleet. A repeat of November now costs the requests already in flight on box B and nothing else, and the founder hears about it from an alert instead of from a customer with a screenshot.
What January did not fix: Postgres is still on box A. Box A refusing to start one evening is still the entire shop, still exactly the August outage that started lesson 005, and no balancer on earth helps with that. Lesson 011 begins on it.
One side effect before then. The balancer now sees every request that enters the system, which makes it the best place you have to count things and to refuse them politely. Lesson 021 uses it to say no, and lesson 026 uses it to see.
Recap
A health check is a question, and you choose which one it asks. Too shallow and it misses the failure entirely, the way a curl of localhost missed an expired certificate. Too deep and it couples every box to one dependency, so a slow database removes the whole fleet at once. The check should touch what a real request touches and nothing beyond that.
Liveness and readiness are different answers. Liveness kills a process and takes its in-flight work with it. Readiness stops traffic and lets the process recover. A busy box needs the second one; Marlow gave it the first at 10:52 and lost the baskets.
Open loop and closed loop. Round robin hands each box its share and never looks at the result, so a box that has gone slow keeps receiving traffic it cannot serve. Least connections reads each box's own behaviour and routes around trouble it was never told about.
The fastest box is the broken one. Anything that routes towards speed will discover that a machine erroring in 2 milliseconds looks like the least busy machine you own, and will send it most of your traffic. Passive checks on real responses are what stop it, and they are usually off by default.
Tell the balancer before you tell the process. Draining only works if the balancer has already stopped choosing the box. Mark it unready, wait a check interval times the failure threshold, then stop the app. Six seconds of patience turns every deploy from planned downtime into nothing at all.
The chain has to end somewhere. Every balancer needs something in front of it, and the last thing in the line is DNS or the routing table. You do not own it, which is why its TTL is a design decision rather than a default.
Check your understanding
Your health check runs a query against the shared database, with a 2 second timeout and three strikes. The database pauses for 25 seconds during a maintenance job. Walk through what the balancer does, second by second, and give two changes that would have kept the site serving.
Twelve boxes are running at 70% CPU during a traffic peak. Your check has a 1 second timeout and removes a box after three consecutive failures. Work out the utilisation after each removal and say how many boxes you lose before the fleet is at 100%. What would you change first?
You run two boxes. One loses its database connection and starts returning 500s in 3 milliseconds, while the healthy one takes 8 milliseconds. Under least connections, roughly what share of traffic does the broken box attract? Describe what the error rate graph looks like, and name the feature that fixes it.
Your deploy script stops the application and then removes the box from the balancer. The balancer checks every 5 seconds and removes after two failures. At 300 requests a second reaching each box, how many requests are dropped per box per deploy? Write the correct sequence, with the wait.
Marlow has a balancer now, but Postgres still runs on box A. List which of November's failure modes are fixed, which are not, and what the next machine you would buy is for.
Next lesson
007 Stateless Services and Where the State Actually Goes. Today assumed the boxes behind the balancer are interchangeable, and lesson 005 showed three ways they quietly are not; next lesson takes the sessions, the uploads and the scheduled jobs off the machines and works out where each one belongs instead.