Marlow Books is a four person online bookshop invented for this course, and on a Tuesday in August its founder turned the shop off on purpose for four minutes. It stayed off for forty.
The four minutes were justified. Full text search over descriptions and reviews had pushed the working set, the data actually being queried rather than the data merely stored, from 14 GB to a bit over 40 on a machine holding 32, which is where lesson 001 left it. The catalogue had been going slow on quiet afternoons with no traffic to blame it on. More memory was obviously the fix. Eight vCPUs and 32 GB of RAM going to sixteen and 64.
Half past nine on a Tuesday is the quietest hour Marlow has. Stop the instance, change the type, start the instance.
The start failed. Insufficient capacity for that machine type in that availability zone, try again later. An availability zone is one data centre, or a tight cluster of them, inside a cloud region. Machines are counted per zone, and Marlow's disk was in that one.
Here is the bit that surprises everyone once. Stopping a cloud machine hands the physical host back. You aren't holding a computer while it sleeps, you're holding a receipt for one, and starting it again is a fresh request for a machine of that type with room in that data centre. The answer is allowed to be no. The 500 GB volume was fine the whole time, sitting there intact, with nothing to attach it to.
The founder tried the original eight vCPU type. Refused too, for ten minutes. You never find out why; nobody outside the provider does. The plausible guess at half past nine at night is that a few thousand other customers' overnight jobs wanted the same zone at the same hour. The third attempt, a different generation of the same size, came up immediately.
21:33 to 22:13. Perhaps forty orders at that hour.
The forty orders weren't the problem. The problem was that for forty minutes there was no version of Marlow Books anywhere on earth, not a slow one and not a degraded one, and whether the shop existed at all was being decided by a scheduler that belonged to strangers.
Three weeks later there were two servers.
Lesson 001 put the two directions side by side in a four row table and deliberately declined to choose. Today is the choosing, and the surprise is which of the two arguments ends up settling it.
Two directions, and what each one actually buys
Scaling up, or vertical scaling, is the same shape of system on a bigger machine. Same code, same database, same one log file, more cores and more memory under it.
Scaling out, or horizontal scaling, is more machines of the same size, with the work spread across them.
They sound like opposites and they aren't. Marlow did both inside four months, and nearly every real system sits somewhere on both axes. The question is only ever which one you spend your next month on.
Lesson 001 already listed what a bigger box gives you and where it stops. It's the cheapest change you'll ever make, because it costs zero design work. It has a top, somewhere around a few hundred vCPUs and a few terabytes of RAM before the specialist machines take over. It buys concurrency rather than single request speed, so a 96 core machine runs Marlow's publisher payout job exactly as slowly as the eight core one did. The resize is itself an outage. And it leaves the single point of failure precisely where it was.
August added a fifth limit that 001 didn't have, and it's the meanest of them because it can't be planned around: the capacity has to exist, in that building, at the moment you ask for it. This gets worse as you climb. There are far fewer very large machines in any given data centre, they're harder to place, and a request for one is easier to refuse. The four minute resize in the design document is four minutes if the answer is yes.
Some platforms will grow a disk or hot add memory without a restart, and if yours can do the specific change you need, take it. The change people usually want is a different machine type, and that one almost always means a stop and a start.
The money, honestly
The line you'll hear in a design review is that vertical scaling gets exponentially more expensive, so you should scale out to save money. That's mostly false, and the part of it that's true isn't the part being claimed.
Across the mainstream families, price tracks resources fairly closely. Twice the cores and twice the memory costs roughly twice the money, and the price per unit of resource stays flat over most of the range. Marlow's original eight vCPU box ran about $250 a month. The sixteen vCPU box costs about $500. No premium, no penalty, no cleverness.
Where the curve does bend is above the mainstream families, at the specialist machines with terabytes of memory that 001 mentioned in passing. Those are priced differently per gigabyte, they aren't offered in every region, and some of them are only sold with a one or three year commitment attached. So the expensive doubling is the last one, not all of them, and if you're nowhere near the top of the catalogue the cost argument you're being sold does not apply to you.
The cost argument that does hold is about granularity, and it's better than the one it replaces.
Marlow's working set was a bit over 40 GB, and the operating system, the connections and a little slack sit on top of that, so call the real requirement 45 GB against the 32 it had. About 1.4 times. The smallest thing it could buy was twice. So the bill went from $250 to $500 for something worth $350, and $150 a month, thirty percent of the new bill, sits idle until the shop grows into it. Which it will, in about a year. Until then the money is parked.
That's real, and here's the correction nobody makes when they quote it at you. At one box, going horizontal is also a doubling.
| Step from Marlow's one box | Monthly | Capacity |
|---|---|---|
| Stay on 8 vCPUs and 32 GB | $250 | 1x |
| One box at 16 and 64 | $500 | about 2x |
| Two boxes at 8 and 32, plus a balancer | $520 | about 2x |
Two identical boxes is two identical bills, plus about $20 a month for a managed load balancer before traffic charges, plus the weeks of work in the next section. The money is a wash and the effort is not close.
Granularity only starts paying you when the fleet is big enough that one machine is a small slice of it.
| Fleet size | Smallest capacity step you can buy |
|---|---|
| 1 box | +100% |
| 4 boxes | +25% |
| 20 boxes | +5% |
| Vertical, at any size | +100% |
The bottom row never changes. Call it the doubling is the only size you can buy: vertical scaling has exactly one step, forever, and it's the whole machine again. Some families offer a size in between, so in practice the ratio is often 1.5 rather than 2, but the lump is always a large fraction of what you already have and it gets larger in absolute terms as you climb.
That's the true cost argument for scaling out, and it's worth almost nothing to you until you have five or six boxes. Below that, anyone telling you to scale out to save money is quoting arithmetic they have not done.
What a second box needs before it helps
Marlow's second box arrived three weeks after that August evening. The founder copied the application onto it, added a second address record for the domain, and let DNS hand out both addresses in turn. Postgres stayed on the first box, and the new one ran only the application, connecting over the network. That's the shape almost everybody builds first, and it's the right first shape.
Round robin DNS is the cheapest load balancing there is and it has two holes. Nothing checks whether either box is alive, so a dead machine keeps collecting its share of visitors. And when you do pull its record, lesson 004's TTL dial decides how long the internet keeps sending traffic to a machine that isn't there, which with Marlow's registrar default of 86,400 seconds is the following day. Lesson 006 is about the box that fixes both, and it's next.
Three things broke over the following weeks, and underneath they were all the same thing.
Sessions went first, within days. A customer logged in on box A, their next click landed on box B, and they were logged out with an empty basket. The session lived in the application process's memory, which had been a completely reasonable place for it when there was one process.
Then cover images. Staff upload a cover when a new title arrives, and the file wrote to the local disk of whichever box happened to serve the upload. Half the requests for that cover afterwards got a 404. Lesson 001 already flagged 216 GB of covers sitting on the database volume as a bad idea for storage reasons; this is the other bill arriving.
Then the first Sunday of October at 2 am, when the publisher payout job from 001 ran on both boxes, because it lived in the application's crontab and the crontab got copied along with everything else. Both machines loaded a year of order lines, both grouped by publisher, both emailed a CSV. Nobody was paid twice. The founder established that by spending Sunday morning proving it.
Every one of those is state sitting on a machine. Shared nothing is the property you need: a machine that holds nothing another machine would need in order to answer the same request. Sessions moved into the database, and later into a cache, which is lesson 008. Covers moved to object storage, which is 016 and again at 044. The payout job needed exactly one owner, which sounds like a five minute fix and is actually leader election, lesson 032, and job scheduling, lesson 076.
The sentence to keep is that you don't scale out a service. You scale out a stateless service, and making it stateless is the work. Lesson 007 is entirely about where the state goes instead.
Scaling out moves the ceiling, it does not remove it
Here the story stops being encouraging, which is probably why it usually gets left out.
One box, everything on it, did about 200 book pages a second. Lesson 001 measured that on a quiet evening and it was a ceiling rather than a calculation, because the queries contend for one buffer pool, one lock table and one disk.
With the application on two boxes, the pair measured about 260 pages a second.
Not 400. The founder added a third application box to check, got 265, and stopped adding boxes.
The reason is in 001's own description of where the 5 milliseconds goes: mostly Postgres, and Postgres is still one process on one machine doing all of the database work for the whole fleet. Extra application servers don't divide a buffer pool. They deliver requests to the bottleneck more promptly.
That generalises, and it is the sentence to carry out of today. Scaling out a stateless tier converts a compute limit into a database limit. The ceiling moves once, to a place where the tools are different and considerably more expensive: read replicas in 011, sharding in 013, caching in 008 and 009.
Moving it is genuine progress. It's a long way from the unbounded win the word horizontal implies.
Nine hundred wind turbines, each sending a reading every two seconds, feed a telemetry service called Galewatch, invented for this course in the same spirit as the bookshop. Its ingestion path is the friendliest case for scaling out that exists. Four hundred and fifty readings a second, every reading independent of every other, nothing to coordinate, no session, no ordering anybody cares about. Run two ingest processes or twenty; the turbines don't notice.
And it still runs into the same wall, from a different angle.
Every ingest box holds a pool of database connections. Postgres allows 100 by default, which is exactly the limit lesson 001 watched Marlow hit with four application processes at 25 connections each. Galewatch's ingest boxes hold 25 too. The fourth box takes the cluster to exactly 100, and the fifth box gets "too many connections" while every machine in the fleet sits at 15% CPU and every dashboard looks healthy.
Underneath that limit is a throughput ceiling worth writing down. A Galewatch ingestion write takes about 20 milliseconds, so one connection completes 50 writes a second and 100 connections complete 5,000.
1 / 0.020 s = 50 writes per second per connection
50 x 100 = 5,000 writes per second, all connections
current load = 450 readings per second
Eleven times the load it carries today. That looks like years of comfort right up until the afternoon somebody signs a contract.
The other half of Galewatch doesn't move at all. Adding ingest boxes adds no storage whatsoever. Lesson 001's 7.8 GB a day and 2.8 TB a year land in exactly the same place whether one process wrote them or twenty. Scaling out the writers does nothing for the thing being written to, and the query that reads a year back for one turbine is not one microsecond faster for any of it.
The runway question
Headroom is the gap between your peak and the ceiling you'd hit soonest, measured in the units of whatever runs out. Marlow's headroom on a normal Tuesday is 40 requests a second against 200, which sounds enormous until you remember the podcast Friday put 400 through the same box.
Headroom tells you whether you need capacity. It says nothing at all about which direction. For that, three numbers.
The first is peak utilisation of whichever resource runs out first, which is rarely the one on the dashboard. Above about seventy percent you're on the steep part of 001's hockey stick and you need capacity this quarter, in some direction.
The second is your runway, and it's a division. Count the doublings left between your machine and the top of the catalogue, then divide by your doubling time. Marlow's orders have grown about sixty percent a year, which is a doubling every eighteen months. From eight vCPUs to a couple of hundred is between four and five doublings.
4.5 doublings x 18 months = 81 months, just under seven years
So the honest capacity advice for Marlow is: buy the bigger box, then buy the next bigger box, and revisit the question in the 2030s. Anyone telling that founder to shard the catalogue is selling something.
Galewatch cannot use that division, and why not is worth more than the sum. Marlow's traffic grows the way traffic grows, a bit every week, which is what makes a runway meaningful. Galewatch's grows when a contract is signed. One deal can double the reading rate on a date chosen in a meeting nobody in engineering attended. A runway assumes smooth growth. Step functions don't have runways, they have lead times, and yours is however long the build takes after somebody in sales says yes. If your growth arrives in steps, stop computing months of headroom and start agreeing a notice period with the people signing the contracts.
Stagefront, this course's ticketing service, is the extreme version and never had a runway to consider at all. A stadium show goes on sale at exactly 10:00, two hundred thousand people press the same button inside the same minute, and an oversold seat is a lawsuit. It was horizontal before it had a single customer, because the peak was known in advance and no single machine has ever been able to serve it.
The third number is what a missing machine costs.
Lesson 003 priced this for Marlow already. The one box at 99.5% is 43.8 hours a year, and 003 split that into about 32 hours from causes a second machine cannot help with, bad Friday releases and expired certificates and one database, plus 11.8 hours of per box hardware. A second identical box behind a balancer lands near 32 hours and 99.63%, a twenty seven percent reduction rather than the thirteen minutes a year that independence would have promised.
And the balancer is a new component in the request path, which 003 said adds downtime rather than multiplying it. At 99.99% it contributes another 0.876 hours, about 53 minutes a year, so the honest figure is closer to 33 hours than 32.
Marlow did not collect that figure in the autumn, and the reason is worth more than the arithmetic that produced it. Round robin DNS hands out both addresses whether or not both machines answer, so when a box died half the visitors were still sent to it, for as long as 004's TTL dial said. And Postgres never left box A. If box A refuses to come back one evening, box B is an application server with nothing to talk to, and the shop is exactly as gone as it was in August.
So the autumn pair bought throughput, and it bought surviving a crashed application process. It did not buy surviving the machine holding the database, and it did not buy surviving a dead box politely. Lesson 006 buys the second of those. Lesson 011 starts on the first.
Eleven hours a year, minus one back, and only once the balancer is real. Worth weeks of work?
For a four person bookshop where the founder is also the on call rotation, yes, and not for the eleven hours. For the forty minutes in August. What the second box starts buying is an answer to "does our shop exist right now" that isn't owned by somebody else's capacity planner.
Most of the teams I've watched make this move made it for that reason. They went horizontal because they couldn't tolerate a machine disappearing, or because they couldn't afford the reboot, and the extra capacity arrived afterwards as a side effect they were happy to accept. Capacity is what gets written in the design document. Availability is what actually signs it off.
So: if the runway is longer than the time it takes to build the horizontal version, and you can live with 003's downtime figure, buy the bigger box and put a date in a calendar. If the runway is shorter, start now, because this work always takes longer than the estimate. And if it's the downtime you can't live with, the runway is irrelevant and you've already decided, whatever the arithmetic says.
Marlow finished the year on two boxes at sixteen vCPUs and 64 GB, still split by DNS, about $1,000 a month against July's $250. The database was still one machine, still the ceiling, and lesson 011 is where that gets attacked. The founder still deploys on Friday afternoons.
Recap
The capacity you don't own. A stopped cloud machine is a receipt, not a possession. Starting it again is a request that can be refused, and the larger the machine the easier it is to refuse. Every plan that says "four minutes of downtime for the resize" is assuming an answer it hasn't been given.
The doubling is the only size you can buy. Vertical scaling has one step size and it's the whole machine again, so you always overshoot and pay for capacity you won't use for months. Horizontal scaling's step is one box, which is a hundred percent when you have one box and five percent when you have twenty. That's the real cost argument, and it doesn't apply to small fleets.
Shared nothing. A machine that holds nothing another machine needs to answer the same request. Sessions in process memory, uploads on local disk and a cron job in the image are all state hiding in the application tier, and each one turns into a bug in the weeks after you add the second box.
Scaling out moves the ceiling. Adding stateless machines converts a compute limit into a database limit. Marlow went from 200 pages a second to 260, and then a third box bought five more. Reaching the next ceiling is real progress, and the next ceiling is a harder and more expensive problem than the one you left.
The runway question. Doublings left in the catalogue, divided by your doubling time, gives you months. Seven years says stay and spend the effort somewhere else. But the division assumes growth arrives smoothly, and if yours arrives when contracts are signed you need a notice period rather than a runway.
Check your understanding
A service peaks at 72% CPU on a 16 vCPU machine, its traffic doubles every 11 months, and the largest machine in that family has 128 vCPUs. Work out the runway in months, then say what you would actually do this quarter and why.
You add a second application server behind a load balancer and customers start getting logged out at random. Name the state that's living in the wrong place, two other things that will break within the month, and where each one should move instead.
Galewatch's ingest boxes each hold 25 database connections against Postgres's limit of 100. You have four boxes and a new wind farm needs a fifth. Say what breaks, what the CPU graph on every box will show while it breaks, and give two fixes with their trade-offs.
A colleague argues that scaling out is cheaper than scaling up because you avoid the premium on large machines. At what fleet size does that argument start being true, and what is the correct cost argument for scaling out below that size?
Your product's traffic grows in steps, when contracts are signed, rather than smoothly. Say what you would measure instead of a runway, and what specific agreement you would go and get from the people signing those contracts.
Next lesson
006 Load Balancers: Spreading Traffic Without Losing Anyone. Today's second box was shared out by round robin DNS, which cannot tell a live machine from a dead one; next lesson puts a real balancer in front of the fleet and works out what it needs to know about each box to route around one that's gone.