Galewatch, a company that exists only in this course and takes a reading every two seconds from each wind turbine on the farms it monitors, lost the link to Kilmore Sands at ten past seven on Saturday 17 October, in a gale the forecast had been carrying all week.
Nobody rang anybody, for the reason lesson 003 settled: the turbines hold their readings in flash and replay them when the link returns. Kilmore Sands is a hundred and forty turbines at seventy readings a second, which is lesson 014's figure and the one lesson 037 reuses. The link came back at twenty to one.
Five and a half hours at seventy a second is 1,386,000 readings, and they crossed in a couple of minutes. Lesson 037's rule put every one of them on the backfill topic, because each carried a recorded_at hours behind its received_at. The log took them without blinking. That part has worked since July.
Then the lag graph went vertical, and this time something was watching it.
After the August Saturday lesson 041 tells, the writers got the bounded queue that lesson argued for, and somebody wired a scaling rule to the writer tier as well. Three writer boxes of two processes, which is lesson 041's arrangement, six in the group the way lesson 037 left it. Add a box when the group's lag stays over sixty seconds for two minutes running, wait three minutes, look again, up to a ceiling of six boxes. It is a reasonable rule. I would have signed it off.
At 12:45 it added the fourth box. At 12:48 the fifth. At 12:51 the sixth, and the group was at its maximum with twelve writers in it.
Here is what those twelve writers could do about Kilmore Sands. Nothing at all. The log keys by farm, so the whole backlog is one partition, and lesson 037's rule is that a partition goes to exactly one member of the group. That writer runs lesson 041's twenty database workers, and its pool to any one shard is ten connections. A farm is atomic once lesson 014 makes farm the key, so all 1,386,000 records go to one machine through ten connections at the twenty milliseconds lesson 002 timed a Galewatch write at. Five hundred a second. Forty six minutes and twelve seconds. The three other farms on that writer, since lesson 042 counted four partitions to a member, kept writing throughout and cost the drain nothing, because their rows go to their own machines. Lesson 041 priced the same drain at Ardnave Point at nearly twice that rate, because that farm is split over three machines, so its writer ran out of workers rather than connections. Six more writers do not divide a partition.
They do cost something, though, and lesson 037 priced it: a member joining makes the group stop, give up its partitions and take them again, about two seconds when it is done politely. Three steps, six joins, twelve seconds of a fleet that writes nothing.
The expensive part arrived at the ceiling. Every Galewatch shard is a Postgres allowing the default hundred connections, which lesson 001 first walked into and lesson 005 watched break a fifth ingest box, and that pool of ten is held against each of the nineteen machines. The ingest boxes gave up their own write pools when lesson 037 put the log in the middle, and the routing table poll lesson 048 left them reads the small three machine database lesson 033 built rather than the shards. So the hundred is the writers' to spend, and six writers spend sixty of it. Twelve want a hundred and twenty.
Pools fill lazily and the last to ask loses, which that afternoon meant the two newest writers. They collected the error lesson 005 published word for word, retried, gave up and exited, and the process manager started them again a minute later to do it all over. An unannounced exit does not get the polite two seconds; the group waits out the session timeout, which lesson 037 puts at tens of seconds. Call it thirty. Two processes on a one minute cycle took roughly half of every minute away from the ten writers that were working.
The on-call engineer was paged at 12:55, read graphs for nine minutes, and set the group back to three boxes at 13:04. The backlog cleared at 13:35. Left alone it would have cleared at 13:28.
Seven minutes, on a Saturday, with nothing lost. It is a small bill and I want you to hold on to how it was run up, because every line of that configuration was defensible. The signal was the obvious one and the only one that tier publishes. The thresholds were sane. The maximum sat well under the twenty four partitions anybody would have checked against. And the loop spent the afternoon adding capacity to a queue that no amount of capacity could drain, into a ceiling nobody had written down.
A control loop, and the one number nobody has
An autoscaler is a loop. It reads a number, compares it to a target, changes the size of the fleet, waits out a cooldown so its own change settles before it measures again, and reads the number a second time. Every thermostat you have ever argued with is the same machine, and so is every cruise control.
Loops are well understood and they have one failure mode above all others, which is lag. A thermostat that learns the room's temperature twenty minutes late will overshoot every time and never settle, however good its target is. Nobody blames the target.
So take your own loop apart and time it. Marlow Books, the four person online bookshop that has been one server and then two for this whole course, is where the arithmetic is cheapest, because its founder can hold every stage of it in his head.
Suppose he puts the application tier behind a scaling group tomorrow, on CPU, at the usual target. A step in traffic arrives. The stages are:
load steps up
| up to 60 s the metric is a one minute average
| 60 s a second period, so one spike cannot move the fleet
| 50 s the provider allocates and boots a machine
| 10 s warm up: config, credential, first connections
| 6 s health checks pass, the balancer starts choosing it
v
the new box serves its first request
Roughly three minutes, and every term in that stack is one the course has already met. The ten seconds of warm up time, meaning everything between a process starting and it being fit to answer, is lesson 048's five second config poll plus the credential lesson 053 put on the cold start path, and lesson 049 measured the only real one this course owns: nine seconds for a proxy to collect its routes and a certificate. The six seconds at the bottom is lesson 006's health check at two second intervals, three passes to be chosen again, which is its removal strike count read in the other direction.
The fifty seconds in the middle is the one I would not trust. Lesson 005 is the record of what that provider actually does: the founder asked for a bigger machine one evening, was refused, tried a different size, was refused again for ten minutes, and got a different generation of the same size immediately, on the third attempt. The guess lesson 005 offers is that a few thousand other customers wanted the same zone at the same hour. Your boot time is a median, and a scaler meets the tail of that distribution during precisely the hours when every other tenant is also scaling.
Now the number I would want first, and I have never found it on anybody's dashboard. Stamp the scaling decision and the new box's first served request, then subtract.
Compare the loop's lag to the time your load takes to double. That ratio says how much bigger the load is by the time the fleet you ordered turns up. Marlow has two load shapes in the published record and they land on opposite sides of it.
Lesson 001's podcast Friday is the fast one. The mention went out at 10:40 and the shop was over four hundred requests a second by 10:46, from forty. Ten times in six minutes is a doubling every 108 seconds. Three minutes of loop against 108 seconds of doubling means the load multiplies by 2 to the power 1.66, which is a shade over three. The box you ordered when things got uncomfortable arrives into a shop with three times the traffic it had when you ordered it, and the next box is three minutes behind that one.
Christmas is the slow one. Lesson 006 records four hundred requests a second for Christmas week against the forty of an ordinary day, and nobody has published the shape of the climb between them, so take the shop opening quietly at eight and peaking around noon. Ten times over four hours is a doubling every 72 minutes, and the same three minute loop now lets the load grow by 2.9 percent before the box lands.
One autoscaler, one shop, two mornings. On one of them it is a control and on the other it is an observer with a credit card.
That gives you the utilisation target, the load level the scaler steers towards, which everybody argues about and almost nobody derives. The headroom you need, meaning the capacity you are deliberately not using, is whatever your loop lets in, so the target is the reciprocal of that growth factor. At Christmas that is one over 1.029, which is 97 percent, so the real constraint is lesson 001's knee: past roughly 70 percent you are on the steep part of the hockey stick and latency goes vertical whatever your capacity arithmetic says. On podcast Friday the same sum gives 32 percent, which means running the fleet a third full all year to survive one afternoon.
The target is the lower of the knee and the reciprocal of what the loop lets in. When the second one wins, the scaler is telling you to buy a fleet you will not use, and that is the moment to stop configuring and go and change the shape of the load instead. Marlow did exactly that without the vocabulary: lesson 001 put a cache in front of book pages the week after the podcast Friday, and lesson 006 says plainly that the cache is how the shop survives Christmas at all.
A ramp, a step and a backlog
Three shapes, and only one of them is a scaling problem.
A ramp is load climbing over minutes or hours. Marlow's Christmas is a ramp and so is almost everything a business does during a working day. A scaler earns its keep here, which is why the technique exists and why everybody's first experience of it is a good one.
A step is load arriving all at once. Stagefront, the ticketing service in this course where a stadium show goes on sale at exactly 10:00 and two hundred thousand people press the same button inside the same minute, is nothing but steps. Lesson 017 measured the arrivals at 3,333 a second. Take the morning from a quiet ten a second, which nobody in this course has measured, to that inside the minute and it is a doubling every seven seconds, so a three minute loop is twenty five doublings behind. There is no scaler and no target that reaches it. Autoscaling answers a ramp, and a step has to be answered before it arrives. Lesson 017 answered Stagefront's with a queue in front, which turns a step in arrivals into a ramp in waiting, and the twenty boxes lesson 006 counted are provisioned for the minute rather than grown into it.
A backlog is the third shape and the one the hook is about. Nothing arrived faster; work accumulated somewhere while you were not looking, and now the queue has to be drained alongside the live load. That is the one case where capacity genuinely is the answer, and it is also the case where you cannot buy your way out by refusing work: lesson 021 settled that rate limiting Galewatch's ingest path protects nothing, because a refused turbine either sends the reading again a minute later, which is the load arriving anyway, or drops it, which loses the product. A backlog has to be absorbed. And the unit it has to be absorbed in was chosen months ago, which is how Kilmore Sands' five and a half hours came to sit in one partition.
A scaling group's maximum is a capacity claim
The minimum of a scaling group gets argued over, because it is the bill. The target gets argued over, because it feels like the design. The maximum gets a round number somebody is comfortable paying for on a bad day, and that is the one I would check first.
Galewatch's maximum said six boxes. The real ceiling was five, and here is the derivation nobody did: nineteen shards, a hundred connections each, ten per writer per shard, so ten writers fit and the eleventh does not. Five boxes of two.
That ten is worth a sentence of its own, because somebody chose well and never got credit for it. Lesson 013 did this arithmetic one tier over, when the ingest boxes still wrote to Postgres directly, and warned that the pool size nobody changes is 25 because 25 is a number in a configuration file that gets copied. Four boxes of 25 filled every shard exactly, and lesson 013 said in plain text that the fifth box would break it again the way it did in lesson 005. The writers were given ten instead, which is more than three times what lesson 013's own Little's law sum asks for and the only reason there was room for four more writers at all. Then the scaler was allowed six.
So before you set a maximum, find the first thing the tier multiplies that it does not own. Database connections are the classic. The others worth walking: a downstream service's published rate limit, source ports on whatever does your outbound address translation, a per node licence, and in Galewatch's case the partition count, which is the only one anybody checked.
Underneath all of them sits the question the hook is really about. A scaler can only add the unit your bottleneck is counted in. Marlow's founder measured this himself and lesson 005 wrote it down: one box did 200 book pages a second, two did 260, and a third bought five more, because Postgres is still one process on box A and extra application servers deliver requests to the bottleneck more promptly rather than sharing it. Galewatch's six extra writers bought nothing for the same reason in a different costume.
Write the maximum down with its derivation on the same line, the way lesson 054 asked for a removal date in the commit that introduces a feature flag. A number with no derivation beside it is a number that gets raised by whoever is on call the next time a graph is red.
The signal decides which outages you can survive
CPU is the default because it is the metric every provider hands you for nothing, and it is right whenever CPU is what saturates. Lesson 049's Stagefront morning is that case exactly: by 10:00:20 its twenty boxes were at the 75 percent CPU lesson 006 measured for that minute, each taking its share of 3,333 arrivals.
The trouble is the other cases, and there are two.
Lesson 005 already gave you the first. The fifth ingest box fails on connections while every machine in the fleet sits at 15 percent CPU and every dashboard looks healthy. A tier at its ceiling on something other than CPU looks, to a CPU scaler, like a tier with room to spare.
The second one is worse, because the number moves the wrong way. Lesson 042 has Stagefront jammed on a slow payment provider: a hundred connections at nine hundred milliseconds apiece finish about a hundred and eleven charges a second against demand of five hundred, and the gap goes into a wait list nobody had bounded. A worker spending nine hundred milliseconds waiting on a socket is not using the processor. So those boxes were doing roughly a fifth of the work they do on a good morning, and a CPU graph reports that honestly, as a quiet fleet.
Point a scale-in rule at it and the scaler will remove boxes from a service that is failing. I have watched this happen, and the post mortem ran two hours because nobody could believe the scaler had been correct.
What you want instead is the quantity lesson 002 named the busy room: concurrency equals throughput times latency. Measure concurrency directly, as the fraction of each box's workers that are busy, and it rises in both stories. Lesson 006 counted Marlow's pool at four processes of twelve workers, 48 in flight per box, so the number is already there and costs nothing to publish. Lesson 047 ran that same law backwards, using a measured wait to work out how many requests were in flight, and said so while doing it. Here it runs forwards, and it beats a processor.
| Signal | Rises when | Misses |
|---|---|---|
| CPU | compute saturates | pool and connection ceilings |
| workers busy | either, including jams | work queued outside the process |
| queue depth or lag | consumers fall behind | a bottleneck that is not the consumer |
For a consumer, lag is the signal worth having, and it still has the hook's hole in it. Lag says the work is piling up. It never says that more hands would help. Lesson 041's rule survives unchanged: a buffer you never look at is correct until it is full, and knowing that it is full tells you nothing about where the fix lives.
The metric you scale on decides which outages you can survive. A fleet scaled on CPU survives the outages that burn CPU and sleeps through the rest, and that is a design decision, made in a dropdown, usually by whoever created the group.
Scaling in is the direction nobody drills
Every team I have worked with tests scaling out. Load generator, watch the boxes arrive, screenshot for the retrospective. I have never seen one test scaling in, and it is the direction that takes capacity away from a running system with nobody in the room.
Three things live in there.
A scale-in terminates a box that is serving requests. Lesson 006 measured what that costs when it is done wrong: three failed checks at two second intervals is six seconds in which the balancer is still choosing a machine with nothing listening, which at Christmas week's two hundred a second per box is about 1,200 requests. Marlow's founder waited those six seconds by hand in lesson 054. A scaling group does whatever its termination hook says, and the default hook is frequently nothing at all.
Then there is lesson 054's mixed window, arriving without anybody pressing anything. A scaling group launches from an image or a template, and the running fleet was last changed by a deploy. Let those two disagree and a scale-out at three in the morning puts the previous release into the pool beside the current one, so every compatibility property lesson 052 taught now applies to a deploy with no deploy order, no operator and no bake time. An autoscaling group is a deploy that fires when it feels like it. The cheap defence is to make the first scale-out after a release part of the release, which is lesson 054's first restarted box used as a probe and pointed at your own build pipeline.
Last, growth has a dependency, and lesson 048 named it already. Fail static buys you the fleet you have and nothing you are about to start, and a configuration outage is collected one restart at a time, by deploys and by autoscaling. A scaler is therefore the second collector of every control plane outage in the building, and it collects during exactly the load that made it scale. Lesson 053 put the sharpest version of that bill on a credential: its lifetime is how long you may be unable to reach the thing that mints it, so a short lived credential shortens the fleet you can grow to, down towards whatever is already running.
I do not have a good answer for picking a scale-in threshold. Every value I have shipped was a guess, and I have never seen one derived from anything. What I would defend is a slow one: scale out on two minutes of evidence and scale in on twenty, because the two mistakes cost completely different amounts.
Bought to save money, configured to protect latency
Marlow's bill is small enough to state in full, and it makes the argument better than a large one would.
Lesson 005 left the shop on two boxes of sixteen vCPUs at about $500 each, a thousand dollars a month. Its ordinary day is forty requests a second against a pair lesson 005 measured at 260 book pages a second, and since lesson 006 counted 96 percent of the shop's traffic as pages those two are close enough to divide: the fleet runs around 15 percent loaded almost all the time. A scaler looks at that and sees three quarters of a bill to delete.
It cannot delete any of it. The floor is two boxes because one box is a single point of failure, which is the limit lesson 005 says vertical scaling never moves and lesson 003 prices in minutes of the year. That floor has nothing to do with load, so the money an autoscaler saves is the gap between the fleet your traffic needs and the fleet your failure budget needs, and at this shop the gap is negative.
The other direction is worse. Six extra boxes, the most a sensible group would allow, buys 260 plus six times five, which is 290 book pages a second against the Christmas peak of 400 requests a second that lesson 006 sets it against. Marlow's autoscaler can spend $3,000 a month to buy thirty book pages a second, or about $700 if it only runs them for Christmas week, and the cache lesson 001 shipped in an afternoon is what actually carries the shop through December.
Which brings me to the opinion I hold hardest here. Most autoscaling is bought to save money and configured to protect latency, and those two want opposite settings. Saving money wants a low floor, a high target and an eager scale-in. Protecting latency wants a floor that covers a failure, a target below the knee, and a scale-in so slow it barely happens. One group cannot do both, and the argument everybody has about the target is really an argument about which of the two they are buying, conducted in a vocabulary that cannot say so. Write the answer in the group's description.
And when you know the calendar, use the calendar. Scheduled scaling beats reactive scaling every time the event has a published date, because it has no loop and therefore no lag. Stagefront's on-sales are announced by its own marketing department weeks ahead, forty a year inside the 800 minutes of scored time lesson 050 counted out of 525,600. Marlow's Christmas has had the same date for a while now. Galewatch's gales arrive on a forecast, which lesson 041 mentions in its first paragraph and which nobody has ever wired to anything, and I think that is the most underused scaling signal in this course.
Recap
An autoscaler is a control loop, and its lag is the number nobody measures. Take it apart into metric window, confirmation, boot, warm up and admission, and Marlow's comes to about three minutes, with a boot time whose tail belongs to every other tenant in the zone.
Compare the loop's lag to your load's doubling time. Marlow's podcast Friday doubles every 108 seconds, so the load slightly more than triples while one box is on its way; a Christmas morning climbing from forty to four hundred between eight and noon doubles every 72 minutes, so it grows by 2.9 percent. The utilisation target falls out of that as the lower of lesson 001's knee and the reciprocal of what the loop lets in, which makes it derived rather than chosen.
Autoscaling answers a ramp, and a step has to be answered before it arrives. Stagefront's 3,333 a second doubles every seven seconds on the way up and no loop reaches it, which is why lesson 017 put a queue in front and why its twenty boxes are provisioned rather than grown. A backlog is the third shape, the one where capacity really is the answer provided the work can be divided, and the one where you cannot refuse the work instead, because lesson 021 showed a refused turbine just sends the reading again.
A scaling group's maximum is a capacity claim. Galewatch's said six boxes and its connection arithmetic said five, one tier over from the failure lesson 005 published and lesson 013 predicted would happen again. Derive the maximum from the first thing the tier multiplies that it does not own, and write the derivation beside it. Underneath that, a scaler can only add the unit your bottleneck is counted in: Marlow's third box bought five book pages a second, and Galewatch's six extra writers bought none, because a partition goes to one member of the group.
The metric you scale on decides which outages you can survive. A tier at its connection ceiling sits at 15 percent CPU, and a tier jammed on a slow dependency reports less CPU than when it was healthy, so a CPU scale-in rule will shrink a service that is failing. Lesson 002's busy room, run forwards as the fraction of workers busy, rises in both.
Scaling in is the direction nobody drills, and a group that grows also launches whatever image the template holds rather than what you last deployed. Growth stops altogether while the control plane is down, since lesson 048's fail static buys you only the fleet you already have. Scale out on two minutes of evidence and in on twenty, and when the event has a published date, schedule the capacity, because a calendar has no lag.
Check your understanding
Your service runs a scaling group on CPU at a 60 percent target, and a dependency goes slow for twenty minutes: latency triples, throughput halves, nothing errors. Say what the group does, what you would have wanted it to do, and the one metric you would add before Friday.
Marlow's founder wants autoscaling for Christmas week. Using this lesson's figures, tell him what it would cost, what it would buy, and what you would spend the money on instead. Then say what would have to change about the shop before your answer flips.
You inherit a group whose maximum is 20 and whose minimum is 2, with no comment anywhere. Describe how you would work out what the maximum should be, name the three shared resources you would check first, and say what you would do on the day you find the real ceiling is 9.
A team drains a six hour backlog by taking their consumer group from four processes to sixteen, and the drain takes exactly as long as it did last time. Say what you would look at first, what you expect to find, and the change that would actually make the next drain faster.
Galewatch's on-call proposes deleting the writer tier's scaling rule outright after the Saturday above. Argue for keeping some version of it, name the signal you would scale on instead of lag, and say what maximum you would set and why.
Your scale-in rule triggers at 40 percent CPU over ten minutes, and your deploy pipeline updates the running fleet but not the launch template. Walk through a Tuesday night where traffic falls away, the group scales in, and traffic recovers at 07:00. Say what is serving requests at 07:10, and which property from lesson 052 decides whether anybody notices.
A finance colleague asks why your fleet runs at 15 percent utilisation and wants the minimum halved. They are not being unreasonable. Give them the honest answer, say what you would have to measure to argue it properly, and name the one change that would let you say yes.
Next lesson
056 Cost as a Design Constraint. Today priced one fleet and found that Marlow's cheapest capacity decision was a cache rather than a box; lesson 056 turns that into a way of designing, where the bill is a requirement you gather at the start rather than a surprise you explain at the end.