Four people run a small online bookshop. Two of them buy the stock, one packs the parcels, and the founder writes the code and ships it on Friday afternoons when the orders go quiet. The shop is called Marlow Books, it exists only in this course, and for two years it ran on a single server that nobody thought about.
One machine in a data centre in Mumbai. Eight vCPUs, 32 GB of RAM, a 500 GB SSD, nginx in front, the application behind it, and Postgres writing to the same disk. A catalogue of 1.2 million titles. About 40 requests a second on an ordinary Tuesday.
Then a books podcast with half a million listeners spent nine minutes on a novel Marlow happened to have 400 copies of, and read the shop's name out on air. The Friday before Christmas, 10:40 in the morning.
By 10:44 the site was taking ninety requests a second. By 10:46, just over four hundred. At 10:47 a book page needed eleven seconds to load. At 10:52 the health check timed out, the process manager decided the app had hung and restarted it, and that restart killed every checkout in flight and sent a few hundred browsers back in against a cold cache. The founder was in the packing room with a tape gun, and found out from a customer email.
Nothing was broken. No bug shipped, no deploy went out, no disk failed. The box was doing the only thing a machine can do when more work arrives than it can finish: fall behind, and keep falling behind.
Three things run out on one server. They run out in a particular order, and the first one is the one almost everybody misses.
The one box that was genuinely enough
Give the machine its due first, because the rest of this course is mostly an attempt to buy back what it was handing over for free.
Marlow's entire architecture was a web server accepting HTTPS, an application process, a database writing to the local disk, and a domain name pointing at that machine's address.
browser
|
| HTTPS
v
+------------------------+
| one server |
| |
| nginx |
| app process |
| Postgres -> disk |
+------------------------+
Call it the one box. You'll hear engineers sneer at it, and you can ignore them, because large systems spend millions of engineering hours trying to recover the properties this shape gives away.
Every read is fresh, because there's only one copy of the data. Two writes sit inside one database transaction, so either both land or neither does. When something goes wrong there is one log file and one place to look. A query joining five tables just works, because all five tables are on the same disk. Marlow's founder can answer "did that order actually get paid for" with one SQL statement, at any hour, and be right.
That's not a toy. Stack Overflow ran one of the busiest sites on the internet for years on a handful of machines, on purpose, because the simplicity was worth more to them than the fashion was. A modern eight core server with 32 GB of RAM will handle a few hundred ordinary CRUD requests a second without complaining. Plenty of businesses never need more than that in their entire life.
So the useful question is what runs out, and when.
Ceiling one: it gets slow before it gets full
Here's the intuition nearly everyone brings, and it's wrong in an interesting way: the server has eight cores, I'm using four, so I'm at half capacity with room to spare.
Capacity doesn't behave like a bucket. It behaves like a road.
Two words, and they are not the same quantity. Latency is how long one request takes, from the moment it arrives to the moment the response leaves. Milliseconds. Throughput is how many requests the system finishes per second. A road has a speed limit and a number of lanes. Adding cars changes nothing at all, and then it changes everything.
Work Marlow's numbers. A book page costs the box about 5 milliseconds: parse the request, run two queries against the Postgres sitting on the same machine, render, respond. Most of that time is the database. Eight cores does not turn that into eight times the pages, because those queries contend for the same buffer pool, the same locks and the same disk. Loaded up on a quiet evening, the whole box managed a little over 200 book pages a second before response times started climbing, so 200 is a measured ceiling rather than a calculated one. Call it the box's service rate, the number of requests it finishes per second when it's working flat out.
Requests don't arrive politely spaced. Real people arrive at random, so sometimes three land in the same millisecond and two of them wait. That waiting is queueing, and queueing is where your latency actually comes from.
There's a standard approximation for one queue with one worker and random arrivals. It flatters reality, and it gets the shape exactly right:
response time = 1 / (service rate - arrival rate)
Service rate is 200 a second. Put arrival rates in; utilisation is just the arrival rate as a fraction of that service rate.
| Arrivals per second | Utilisation | Average response |
|---|---|---|
| 40 | 20% | 6 ms |
| 100 | 50% | 10 ms |
| 160 | 80% | 25 ms |
| 190 | 95% | 100 ms |
| 198 | 99% | 500 ms |
Read that table twice. Between 50% and 80% you added sixty percent more traffic and paid 15 milliseconds for it, and nobody on earth notices 15 milliseconds. Between 95% and 99% you added eight requests a second, four percent more load, and response time got five times worse.
This is the hockey stick at eighty percent, and it's the most useful shape in this course. Load does not degrade a system gradually. It does nothing, nothing, nothing, and then everything at once.
Now look again at what the podcast did. Four hundred requests a second against a service rate of 200 is not "200% utilisation", a phrase that means nothing. Put 400 into that formula and the denominator goes negative, which is the arithmetic's way of saying there is no steady state at all. The queue doesn't settle at some unpleasant number. It grows for as long as the traffic lasts, and every request in it waits behind a longer line than the one before it.
Eleven seconds at 10:47 was not the bad case. It was an early reading on the way up.
Then it got worse, for a reason worth naming now. Somebody who has watched a blank page for thirty seconds hits reload, and any client with a timeout and a retry policy does the same thing without being asked. That second request is brand new work arriving while the first one is still sitting in the queue, asking the box to do the same job twice. Arrival rate goes up because response time went up, which pushes response time up again. That loop is a retry storm, and lesson 020 takes it apart properly.
The thing you can use tomorrow: a box sitting at 80% CPU is not "80% used". It is one lucky press mention away from an incident. Run at 50 to 60% and treat the gap as the thing that absorbs your Friday, not as waste.
Two footnotes on that morning, because they cut in opposite directions. Marlow had 400 copies of the novel, so no architecture on earth was going to convert half a million listeners into sales; being slow for twenty minutes was arguably the correct outcome. But the checkouts killed at 10:52 belonged to people who had one of those 400 copies in their basket, and those are the customers the shop actually lost. Capacity problems rarely cost you the traffic you couldn't serve. They cost you the traffic you could.
Ceiling two: it fills up
The second ceiling is arithmetic, which makes it the easiest to see coming and the most embarrassing to be surprised by.
Marlow's version is dull and instructive. The founder put all 1.2 million cover images on the same volume as the database, at roughly 180 KB each.
1,200,000 covers x 180 KB = about 216 GB
Two fifths of a 500 GB disk holding the cheapest, least changing data in the shop, on the most expensive storage the shop owns, sharing throughput with every order write. Nobody decided that. It accumulated.
Covers are a one time cost, though. The ceiling that really catches people is a rate, and rates need a system that produces one, so meet a second invented business.
Nine hundred wind turbines, each sending a reading every two seconds over links that drop for hours in bad weather and then come back carrying a backlog, feed a telemetry service called Galewatch. It exists only in this course too. Engineers use it to decide which machine to climb today.
Nine hundred turbines at one reading every two seconds is 450 readings a second, forever, at about 200 bytes a reading once you count the timestamp, the turbine id and a handful of numbers.
450 readings/sec
x 86,400 seconds in a day
= 38,880,000 rows per day
x 200 bytes
= about 7.8 GB per day of raw rows
Call it 2.8 TB a year before you index anything. Then index it, because every screen in the product asks "show me this turbine's readings for that day", so you need turbine and time at the very least. Indexes commonly add 50 to 100 percent on top of a table of narrow rows like these, so the honest figure is 4 to 5.5 TB a year of real disk.
Galewatch sized its storage volume during a pilot with 12 turbines. Twelve turbines is 6 rows a second, about 104 MB a day, and a 500 GB volume would have lasted thirteen years. Solved forever, obviously. At 900 turbines, with indexes, the same volume fills in about five weeks.
Nobody misjudged that by a factor of a thousand. They misjudged it by one wind farm, which is the usual size of the mistake, and the arithmetic that would have caught it takes ninety seconds and no tooling.
Disk is the ceiling you can most easily raise, so raise it and move on. The others are meaner.
RAM fills, and this one bites harder than it looks. A database is fast mostly because the pages it needs are already in memory. When the working set, the data actually being queried rather than the data merely stored, outgrows RAM, the database starts going to disk instead. A read from RAM takes about 100 nanoseconds. A read from a good SSD takes about 100 microseconds. That's a thousand times slower for every page that misses, and nothing anywhere reports an error. When Marlow added full text search over descriptions and reviews, the working set went from 14 GB, comfortably inside 32, to a bit over 40, and the catalogue got slow on a Tuesday with no traffic spike and no code change. Lesson 002 makes those numbers permanent; lesson 008 explains what to do about them.
Connections fill too, in a way that makes your dashboards look like liars. Every Postgres connection is a whole operating system process with memory attached, and the default limit is a hundred. Marlow's founder ran four application processes with a pool of 25 connections each, which is exactly 100, and then added a nightly report that wanted one more. It got "too many connections" while the CPU sat at 40%, looking entirely innocent.
File descriptors. Ephemeral ports. Inodes. Each one is a limit you've never thought about until the night you meet it.
Capacity is a couple of dozen numbers, and you are limited by whichever one you hit first, which is almost never the one on the dashboard.
Ceiling three: it goes away
The third ceiling ends careers rather than afternoons.
Marlow's box will stop existing. Not might. The disk fails, the hypervisor gets retired, a kernel patch needs a reboot, someone runs the wrong systemctl in the wrong terminal, the availability zone loses power, or the founder ships a bad release at 4:50 on a Friday. When any of that happens there is no second copy, and if the disk is what died then the orders are gone too, back to whenever the last backup ran.
Put a number on it. Say the machine and everything on it is up 99.5% of the time, which is fair for one unattended box with occasional patching and the odd bad deploy. A year is 8,760 hours.
0.5% of 8,760 hours = 43.8 hours of downtime per year
Nearly two full days. Spread over a year that's an unlucky afternoon every month and a bad Tuesday every quarter. For a four person bookshop, survivable, and the founder might reasonably decide it's the right trade.
Selling tickets is a different business. Picture a stadium show going on sale at exactly 10:00, two hundred thousand people pressing the same button in the same minute, and an oversold seat that is a lawsuit rather than an apology. That ticketing service is called Stagefront, it's the last of this course's invented systems, and one box was never on the table for it at all. Lessons 006 and 021 are largely about why.
There's a second, meaner piece of arithmetic here. When a request needs several components and all of them have to work, their availabilities multiply. Three parts at 99.9% each:
0.999 x 0.999 x 0.999 = 0.997
That's 99.7%, about 26 hours a year, from three components you'd each have called "three nines". Adding parts to a chain makes reliability worse. Only redundancy, two of something where either one will do, makes it better. Chains multiply downward, copies multiply upward, and that one distinction is the engine of every highly available system you will ever build. Lesson 003 gives it a full chapter.
Today's point is narrower and harder. On one box there is nothing to be redundant with. It's a single point of failure, a component whose loss takes the whole system with it. And you can't even deploy without an outage, so every release is a small planned failure, which is why teams on one box deploy less often, which is how a codebase gets frightening.
Everything on one box shares fate
One failure mode escapes all three ceilings, and it produces the most confusing 2 am pages of your career.
On a single machine, unrelated things share a CPU, a memory bus, a disk and a kernel. So they can kill each other.
Marlow's version arrives on the first Sunday of the month. The founder wrote a job that pays publishers: pull every order line for the year, group by publisher, render a CSV, email it. Not user facing. Runs at 2 am. It also loads the whole year into memory to do the grouping, and by its third summer that was 21 GB on a machine with 32.
Linux does not politely refuse the allocation. The out of memory killer wakes up, scores every process largely on how much memory it's holding, and kills the one that scores worst. It has no idea which process caused the shortage. Sometimes it takes the report job, which is the outcome you were hoping for. Often it takes Postgres, because a database configured with 8 GB of shared buffers presents the kernel with a row of backend processes that each count those shared pages in their own resident size, so every one of them looks enormous. And a Postgres backend killed with signal 9 is treated by the postmaster as a crash: it restarts the whole cluster and rolls back everything in flight.
So on some Sundays the report lands in the founder's inbox and nothing else happens, and on the others the API is fine, the report job is fine, and the database is dead. Because of a feature nobody would ever have written down as risky.
Call it shared fate: on one machine, any component's worst moment is every component's worst moment.
The industry's most expensive lesson in this exact shape came from Amazon on 28 February 2017. An engineer on the S3 team, debugging the billing system, ran a playbook command with a typo in it and removed far more capacity than intended from S3 in us-east-1. That took down the index and placement subsystems, which had not been fully restarted in years and took hours to come back, and with them a large fraction of the web. The detail worth keeping is the small one: the AWS status dashboard could not be updated to say S3 was down, because the dashboard's own assets were served from S3 in us-east-1.
Moving the database onto its own machine is usually the first architectural change a team makes, and shared fate, not capacity, is usually the honest reason.
Buy a bigger box, and know where that stops
The obvious answer to all three ceilings is a bigger machine. That's vertical scaling, and you should do it first, every time. It's the cheapest change you'll ever make, because it costs zero design work. Doubling the RAM under a struggling database has bought more teams more time than any clever architecture in this course.
It stops in four specific ways.
There's a top. Cloud catalogues run out somewhere around a few hundred vCPUs and a few terabytes of RAM, with a handful of specialist machines above that at prices that end the conversation, and when you arrive there no warning shot arrives with you.
It doesn't make one thing faster, and this is the part people get wrong in design reviews. A 96 core machine runs Marlow's publisher report exactly as slowly as the eight core machine did, because that report runs on one core. Vertical scaling buys concurrency, not speed. If your problem is a single 40 second query, a bigger box will not touch it.
The resize is itself an outage: stop the instance, change the type, start it, four minutes of downtime if nothing goes wrong. You pay for thin capacity out of the availability that was already thin.
And it fixes none of ceiling three. A bigger single point of failure is still a single point of failure, and arguably a worse one, because by the time you've bought it more of the business is standing on top of it.
| One bigger box | More boxes | |
|---|---|---|
| Effort to do it | An afternoon | Weeks, then forever |
| Removes single point of failure | No | Yes |
| Helps one slow query | No | No |
| Has a hard limit | Yes | Practically none |
That last row is the whole deal. Horizontal scaling, more machines rather than a larger one, is the only direction without a ceiling. Lesson 005 argues the choice properly. It's also the direction that costs you everything the one box was giving away free.
What more machines actually cost you
Be honest about the bill, because most of this course is the bill.
The moment there are two machines, the network sits between them, and the network is not a function call. It's slow by comparison, roughly half a millisecond across a data centre against a hundred nanoseconds for a memory read. Worse, it fails halfway. A call that returns an error has told you something. A call that times out has told you nothing at all: the other side may have done the work, or may not have, and you cannot find out from where you're standing. That single fact is responsible for a startling share of all distributed systems complexity, and lesson 019 is entirely about living with it.
You also hand back the properties from the top of this lesson. Two copies of the data can disagree, which is lessons 008, 011 and 012. A change spanning two services stops being one transaction, which is lessons 015 and 039. One log file becomes forty, which is lesson 026. And "is the site up" stops being a yes or no question, because now some of it is up and some of it isn't, which is lesson 027.
The one box is simple and has a ceiling. Many boxes have no ceiling and are not simple. Every lesson after this one is about paying that bill deliberately instead of accidentally.
So when do you actually leave one box?
Not when a conference talk says to.
Leave when the arithmetic says so, which means doing ceiling two's calculation with your real numbers before you build: rows a second, bytes a row, days you must keep them. Galewatch could have known on day one that 900 turbines needed a different storage design from 12, and the sum fits on the back of a train ticket.
Leave when you're past the knee. If the steady state usage of any resource, CPU or memory or connections or disk throughput, sits above roughly 70%, you're on the steep part of the hockey stick, and your next good day is an incident rather than a slowdown.
Leave when downtime costs more than the complexity does. Forty three hours a year is fine for a bookshop and unthinkable for a ticket sale that happens once, at 10:00. Notice that this is a business judgment wearing an engineering costume, and that you will be better at it than the business is, because you're the one who knows the number.
And leave when shared fate bites twice. Once is bad luck. Twice means those components want different machines and have been telling you so.
When none of that applies, stay on the one box, keep the arithmetic in a file where you can find it again, and spend your effort somewhere it buys something. Premature distribution has killed more products than load ever has.
Marlow's founder, for the record, stayed. The podcast traffic drained away by early afternoon, the shop lost about ninety minutes of checkouts, and the fix that week was a cache in front of book pages and an alert at 120 requests a second. The second server came eight months later, for a reason that had nothing to do with that Friday.
Recap
The one box. One machine running the app and the database is a legitimate architecture, and it gives you fresh reads, real transactions, and one place to look, at no cost. Everything after this lesson is an attempt to buy some of that back.
The hockey stick at eighty percent. Latency stays flat as load climbs and then turns vertical near saturation. A box at 80% isn't 80% used, it's one busy hour from an incident. Past the service rate there's no steady state at all: the queue just grows.
Three ceilings, in this order. It gets slow, from queueing, before anything looks full. It fills up, on disk or RAM or connections or whichever limit is smallest. And it goes away, because one box is a single point of failure and every deploy is a planned outage.
Chains multiply downward. Components that must all work multiply their availabilities together, so adding parts in series lowers reliability. Redundancy, two of something where either will do, is the only thing that raises it.
Shared fate. On one machine, one component's worst moment becomes every component's worst moment. The report job that killed Postgres wasn't a freak event. It was the design.
Vertical first, then horizontal. A bigger box is the cheapest fix and you should take it, but it has a top, it rarely makes a single request faster unless that request was starved of CPU or memory, and it leaves the single point of failure exactly where it was.
Check your understanding
Marlow's box handles a book page in 5 ms of server time and sees 40 requests a second on a normal Tuesday. Roughly what arrival rate puts it on the steep part of the curve, and what would you set an alert on this afternoon?
A colleague says "CPU is only at 45%, we have plenty of headroom before we need to do anything." Give two specific ways Marlow's box could still fall over this month.
A Stagefront request passes through a load balancer, an application server and a database, each up 99.95% of the time. What is the availability of the whole path in hours per year, and what would you have to change to improve it?
Galewatch stores 450 readings a second at about 200 bytes each, and a customer contract requires three years of history. How much raw data is that, and name two consequences beyond the disk bill.
Name one property of the one box that Marlow loses the instant the database moves to its own machine, and one problem that move solves.
Next lesson
002 Latency, Throughput and the Numbers Every Engineer Should Know. Today you used numbers like "100 nanoseconds from RAM" and "half a millisecond across a data centre" on trust; next lesson builds the whole table of them into your head, so you can judge whether a design is plausible before writing a line of it.