Marlow Books is an online bookshop run by four people, and it exists only in this course. Three weeks after Christmas, on a wet Saturday in January, the founder sat down to draw it.
The reason was a week off. They had not taken one since the shop opened, and why is not complicated. Lesson 029 has a temporary member of the packing staff closing the shop by accident on a December Saturday, a buyer messaging the founder at 11:25, and the founder fixing it by 11:29. Four minutes is fast. It is also the entire problem, because those four minutes live in one person's head and that person wanted to go to the coast.
So: one page. What the shop is, how it works, what to do when it stops.
They started with boxes, because everybody starts with boxes. The content delivery network from lesson 022, out at the front. The managed balancer lesson 006 bought in January for twenty dollars a month. Box A, which runs Postgres and half the application. Box B, which runs the other half. The streaming replica from lesson 011, five hundred dollars a month since June. The small managed Redis lesson 008 moved the page cache into. The object storage bucket lesson 016 put 216 gigabytes of covers in. The queue lesson 018 installed for confirmation emails, and the email provider on the end of it. The payment provider, unnamed since lesson 015, which lesson 027 watched accept every connection and answer none of them for ninety minutes in September. The metrics service from lesson 026. The nine dollar uptime monitor from lesson 003.
Twelve boxes. Six years ago there was one, and nobody ever sat down and decided to have twelve.
The drawing took an afternoon and it was genuinely good. Every arrow pointed the right way, every box had its monthly cost in the corner, and you could hand it to a stranger and they would understand the shape of the shop in about ninety seconds.
On the Monday one of the two buyers lesson 010 built the publisher page for looked at it over the founder's shoulder, put a finger on the Redis box, and asked the only question that matters. If that one goes red, can I still sell books?
The founder knew the answer, because lesson 008 had done the arithmetic and lesson 027 came back to it in September. Without the cache the shop is offered 148 percent of what its two boxes can serve, so the honest answer is yes, for a while, badly, and only if somebody sheds the browsing first. None of that is on the page. There is no column for it. A box and an arrow cannot hold it.
Then the buyer moved their finger to the replica, which sits off to one side with a single arrow coming out of it, and the founder had to stop and think.
The answer there is no. Lesson 011 routed four kinds of read to that machine, and lesson 027 added them up: a dead replica takes sign in, which takes the basket, which takes the checkout, while box A, the machine that actually holds the money, is perfectly healthy. The box furthest from the money kills the money. On the diagram it is the smallest thing on the page.
What a book page actually costs
Start with the cheapest useful exercise in system design, which is following one request all the way down and adding up what it touched. Twenty nine lessons have each fixed one piece of this shop. Nobody has walked a single page load through all of them at once.
Christmas week, 400 requests a second (lesson 006), of which lesson 008 counts 384 as book pages. A customer in one of the nine cities lesson 022 found on its edge map asks for /book/9780140449136.
The HTML does not come from India. Lesson 022 put the book page document behind the edge in March at a ten minute expiry, so unless this customer's city is the first to ask in the last ten minutes, the twelve kilobytes lesson 008 measured come off a machine in their own town.
The covers do not come from India either, 62 percent of the time. That is lesson 022's measured hit rate on 1.2 million immutable files, and the 38 percent that miss go to the origin shield lesson 022 put in front of the bucket lesson 016 filled in December, rather than to Postgres.
Then the browser makes the second request. Lesson 022 invented it to get the customer's name and basket count out of the cached document, lesson 023 pushed the price into it after May's promotion bug, and lesson 027 added two more fields in September: whether checkout is open, and the sentence to show when it is not. Lesson 028 named its shape. This one is cached nowhere, so it is the only part of the page that crosses the country.
It arrives at the balancer, which picks a box by least connections, and lands on one of the eight application processes lesson 006 counted across the pair. Then:
whether checkout is open a file on this box 027
the price and the stock the replica, 0.2 ms 011, 009
the name and basket count the replica, 0.2 ms 011, 007
Two primary key reads, on a machine that is not the one taking money. Lesson 007 measured 0.2 milliseconds for the session lookup, and lesson 009 took that same figure for the stock read when it split the page. The second read only happens for somebody signed in, which lesson 007 put at about 15 percent of Christmas traffic, so it runs 60 times a second and costs 12 milliseconds of database work a second against the 1.04 seconds lesson 010 measured for the whole shop.
Lesson 010 already priced a book page at that one row read when it built the 1.04 seconds, so half of this is not news. The rest is what no single lesson could say. At the busiest hour of the busiest week of a six year old bookshop, a book page costs Postgres two tenths of a millisecond, and four tenths if the customer is signed in. The bytes are in nine cities. The milliseconds are in Mumbai and there are almost none of them.
Which tells you where the work actually is. Lesson 010 measured the other side of it, that the full text searches are 0.96 of that 1.04 seconds, and lesson 025 turned the pair around: book pages are 96 percent of the traffic and about seven percent of the work, and one request in twenty five carries the other ninety three.
Now count what twenty nine lessons did about each half. The 96 percent got a cache, invalidation on write, a shared cache, a split page, a CDN, an origin shield and a query parameter allowlist. Seven things. The one request in twenty five that carries almost all the work has been moved to the replica (lesson 011) and rate limited per address (lesson 021). Neither made it cheaper. One expensive read did get cheaper, and nobody argued about that one either: lesson 010's composite index took the publisher page from 3,122 milliseconds to 0.38, for two buyers who had been asking since the distributor mess.
That is not a mistake, and it is worth being clear about why. Lesson 001's podcast Friday was a page latency failure, and you fix what is breaking. But a page of twelve boxes with no column for which half of the load each one addresses will let a shop spend six years optimising seven percent.
What an order actually costs
Lesson 027 established that at Christmas one request in two thousand is an order. That request skips nothing.
It starts earlier than you would think, because lesson 019 generates the idempotency key when the checkout page is rendered and carries it in a hidden field. The key exists before the customer has decided anything.
On the POST, in order: claim the key with insert ... on conflict do nothing, so a double submit loses the race rather than charging twice. Commit an order row as pending carrying that key. Close the transaction. Call the card with no transaction open, on the ten second timeout lesson 019 found and lesson 020 argued should be about two. Open a second short transaction and move the row to paid.
Every box on the page is in that sentence. The balancer picked the process. Postgres on box A took the key, then the pending row, then the status change. The payment provider took lesson 015's 300 milliseconds, or twelve seconds on the December evening lesson 019 opened with. The queue lesson 018 built takes the confirmation email. The sweeper lesson 019 added runs every two minutes over anything still pending, which makes it a scheduled job, which puts it in the job_runs table lesson 018 built with a deadline beside its schedule.
Now a question the founder could not answer from the diagram, and which this course has not settled either. Where does the stock decrement happen?
Lesson 015 shipped the single conditional statement after July's flash sale, the one that sold thirteen signed first editions out of twelve copies:
UPDATE books SET stock = stock - 1
WHERE isbn = '9780571358939' AND stock > 0;
Lesson 019 then moved the card call out from between the read and the write. It never said which of its two transactions the decrement rides in, and the two answers are different systems.
Put it with the pending insert and the copy is held at the moment of intent. Lesson 015's whole finding was that thirteen buyers all read a true row and still oversold, because the decision and the write were 300 milliseconds apart; holding at intent closes that for good. What it costs is a copy locked up by an abandoned checkout until the sweeper gets to it, which is up to two minutes plus however long the provider takes to answer, and a refund path that has to put the copy back on the shelf.
Put it in the second transaction and the shelf stays honest about what has actually gone, and July comes back the moment a sale is interesting enough that ninety people arrive in twenty seconds.
I would hold at intent, on lesson 021's reasoning about what a false refusal costs. Refusing a sale wrongly annoys one customer, and the shop has 1.2 million other titles to sell them. Twelve signed first editions and thirteen orders is a letter from a solicitor.
There is a stronger argument than mine, and lesson 019 published the rule for it. A conditional update is idempotent only when its own effect falsifies its condition, and subtracting one where stock is above zero fails that on every copy except the last. Put the decrement in the second transaction and a sweeper retrying it can take a second copy off the shelf. Put it with the key claim and on conflict do nothing covers both in one commit. Lesson 015 built exactly that shape for Stagefront, the ticketing service where two hundred thousand people press the same button at ten in the morning: clicking a seat claims it in one short transaction, and a second one turns the hold into a sale when the payment lands. Nobody ever carried it back to the bookshop.
Which of the two you pick matters less than this: a bookshop with twelve boxes and six years of trading could not look it up, and arrows have no room for it.
The diagram's missing column
What the founder had drawn was components. What the shop is actually made of is copies.
Every outward move in this course created a second copy of something that used to exist once, and the copy is invisible on a diagram, because a diagram draws the machine and not the fact inside it. Count the facts instead. For each one the shop depends on, how many copies exist, and how long two of them are allowed to disagree.
| Fact | Copies | Longest disagreement |
|---|---|---|
| A title's stock | box A's row, the replica's row | 1.1 s daily, 24 min on the first Sunday |
| A session | box A's row, the replica's row | the same |
| Is checkout open | a file on box A, a file on box B | no limit |
| A cover image | the bucket, nine edges | a year |
| A rendered book page | a Redis shell, nine edge copies | a day in Redis, ten minutes at each edge |
The lag figures are lesson 012's, sampled for a week: a 6 millisecond median, 80 at the ninety ninth percentile, a daily peak of 1.1 seconds at the 06:00 distributor import, and 24 minutes on the first Sunday of the month while the payout job holds its snapshot. The year on the covers is lesson 009's expiry header, carried onto every object by December's move. The ten minutes is lesson 022's, and the detail that makes it bite is that each of the nine edges runs that clock from the moment its own city first asked, which is why lesson 023's wrong price decayed city by city from 09:41 to 09:50 instead of ending. The day is lesson 009's, and it is only safe because 009 took the price and the stock out of the thing it is caching.
Read the third row again, because it is the one whose last column is not a number.
Two files, on two boxes, with nothing on earth reconciling them. Lesson 027 chose that deliberately and said why: the switch has to survive the mode where the database is the problem, so it cannot live in the database. Lesson 027 also wrote down the price, in a sentence worth arguing with. Two files can disagree, which is fine for a banner and a disabled button and would not be fine for anything that decides.
That sentence fails its own test. A disabled button decides. It decides whether this customer may give you money, which is the question the stock count answers, and in December, when the two files disagreed, every page load was a coin toss between a shop and a grey button. A grey button is a refused order.
What survives of 027's distinction is the direction, and it is worth keeping apart from the category. A wrong refusal loses one sale and a wrong sale can lose a lawsuit, which is the asymmetry the stock decrement turned on. That asymmetry is why nobody fixed this for three months and why the shop could afford not to. It is not a reason to file the switch under banners.
That is what a copy count gets you that a diagram does not. A row with no limit in the last column is either a fact nobody writes, or a bug nobody has had yet. Marlow's switch has had that row open since September.
What each box stopped being true
Go through the twelve and ask, for each one, not what it bought but what it quietly falsified. Six of them have an answer, and two of the six were not purchases at all.
| The move | Lesson | What stopped being true |
|---|---|---|
| A second box | 005 | state in this process is the state |
| A shared cache | 008 | the page shows the row |
| A replica | 011 | a read is a read |
| A CDN | 022 | the response belongs to whoever asked |
| A file per box | 027 | the shop has one answer |
| One rule at the balancer | 029 | the handler decides |
Every row is an outage in this course. The second box signed customers out, lost cover uploads and ran the payout job twice, and lesson 008 found the casualty nobody noticed, which is that there were now two page caches that never met. The shared cache let an overnight import that predates it sell forty copies of a book the shop had twelve of. The replica answered a customer's read with a table that did not yet hold the review they had just posted, three times in a hundred. The edge served Anand's name and basket to the twenty odd strangers who shared his city.
The last two rows are one outage, which is what it looks like when two of these land on the same Saturday. The file per box made the shop a coin toss, and the rule at the balancer is why somebody who was not the founder was standing in front of the switch at all, having turned "is this the founder" into "is there a cookie" for the eight thousand people lesson 007 counted signed in at the peak.
Six moves. Every one of them correct. Every one of them correct at the radius where the pain was, and wrong one radius further out, where nobody was looking, because nothing there had changed.
The closest this course came to naming it was lesson 023, and it pointed the rule at caches. Lesson 009 split the book page so the price was read live from the row, which was exactly right. Lesson 022 then put the assembled response behind an edge, and the split was still real, still correct and completely invisible, because an edge stores what you sent rather than what you assembled. Lesson 023 did the same split again one radius out. Lesson 027 did it a third time to get the switch into that live response, and December found the seam anyway.
So here is the rule this capstone exists to hand you. It is not in the textbooks, because textbooks describe finished systems rather than six years of Fridays.
A fix is applied where it hurts, and it changes what is true one radius out, where nobody is looking. The question that catches it takes about a minute and nobody asks it. What sentence did I just stop being able to say?
The order you buy things in
The deliverable is not the twelve boxes. It is the measurement that justified each one, because that is the part that cannot rot and the part a list of boxes cannot teach.
Lesson 025 gave the method. Marlow's own history gives the worked example, and it is more useful than any reference architecture, because two of the decisions were wrong.
The second box came after lesson 005 measured a working set of 45 gigabytes against the 32 the machine had. Correct, and it cost forty minutes of downtime because the provider had no capacity in that zone, which is the sort of thing nobody writes down. The balancer came after lesson 006 watched round robin DNS send roughly half its new visitors to a dead box, decaying over the following day as resolvers let go of the record, with every dashboard clean. Correct, and twenty dollars. The shared Redis came after lesson 008 found the two caches that never met.
The replica is the first wrong one, and only halfway. Lesson 011 bought it for five hundred dollars a month and said plainly it was not for throughput, since the shop's entire database load is 6.5 percent of sixteen cores. It was worth buying anyway, because Postgres had never left box A and one disk was still the end of the shop. Then the monthly payout job was moved onto it for a reason that had expired the previous August, when lesson 005's resize took box A from 32 gigabytes to 64 and the job stopped running out of memory. That move cost thirty one minutes of cancelled payout run on the first Sunday in June, and three failures in July.
The second wrong one is lesson 024's fifty lines of Python, which promoted the replica when box A stopped answering for thirty seconds. Written after a Saturday the shop spent dead with a warm copy sitting beside it doing nothing. Deleted inside a fortnight, after a ninety second network blip between two machines produced an hour and fifty minutes of a shop that signed people out and showed stale prices with no expiry to end them.
And then the thing that was never bought at all.
Lesson 016's Saturday went like this. A third nightly database dump ran out of room on a 500 gigabyte volume sitting at 97 percent, Postgres had nowhere to put the next write and shut down rather than pretend it had landed, and because this happened at ten to four in the morning the first customer could not report it until they had woken up. Four hours fifty minutes, beaten in this shop's whole history only by lesson 004's TLS certificate, which expired at half past five on a Sunday morning and left the shop unreachable for six hours twenty one.
The shop still has no alert on disk usage. One threshold and a line of cron would have turned that Saturday into an email days earlier.
Over fifteen hundred dollars a month across the pair, the balancer and the replica, and the cheapest unbought thing on the list caused the second longest stretch this shop has ever spent completely unreachable.
The page the founder ended up with is not a diagram. It is the twelve boxes with what each one removes and what each one made untrue, the copy table with its one row that has no limit in it, and a list at the bottom of the things that were argued for and never built, with the disk alert at the top. It fits on one side of A4 and it is uglier.
They went to the coast in February. The disk alert is still not built, which is the honest ending, because writing a thing down is not the same as doing it and a course that pretended otherwise would be lying to you.
Recap
Follow one request all the way down before you draw anything. A Marlow book page at Christmas costs Postgres two tenths of a millisecond, four tenths if the customer is signed in, because the document is at an edge, the cover usually is too, and only the small live request crosses the country. The bytes moved out and the milliseconds stayed.
The expensive path is almost never the common one. Book pages are 96 percent of Marlow's traffic and seven percent of its database work. Seven of the things this course built went to the 96 percent; the one request in twenty five carrying the rest was moved to a replica and rate limited, and neither made it cheaper.
Count the copies, not the boxes. A diagram draws the machine and hides the fact inside it. For each fact your system depends on, write down how many copies exist and how long they may disagree. A row with no limit in that column is either a fact nobody writes or a bug nobody has had yet, and Marlow's checkout switch has had that row open since September.
One copy decides and the rest are displays, and a display allowed to decide is the bug. It was an editable cookie that cost 47 dollars, a cached stock count that sold forty copies of twelve, a promoted replica that signed out everybody who tried to sign in, and two files that made the shop a coin toss. Four stories, one shape.
A fix is applied where it hurts and changes what is true one radius out. Six moves in this course, six falsified sentences, every fix correct at its own radius. The question costs a minute and nobody asks it: what sentence did I just stop being able to say?
The architecture is the ordered list of measurements, not the list of boxes. Marlow moved a job onto a replica for a reason that had expired ten months earlier, wrote fifty lines of failover that turned a ninety second blip into an hour and fifty minutes, and has still not bought the disk alert that would have prevented nearly five hours of a dead shop. Boxes do not tell you that. Measurements do.
Check your understanding
Write the copy count for the service you work on: every fact it depends on, how many copies of that fact exist, and how long two copies may disagree. Find the row with no limit in the last column and say which of the two things it is.
Marlow has 48 workers per box and a pair that can serve about 260 uncached pages a second, against 384 book page requests a second at Christmas. Redis restarts at 11:00 in Christmas week. Write the first sixty seconds, say what you would shed and in what order, and name the number you would have to be measuring today to do any of it.
Pick the three boxes you would buy first if you were starting Marlow from nothing this morning, in order, and for each one name the single measurement that would make you buy it. Then say which of the twelve you would refuse until something forced your hand.
Galewatch collects readings from nine hundred wind turbines over links that drop for hours and come back with a backlog. Walk one reading from a turbine to an engineer's dashboard, count the copies of it that exist along the way, say which copy is the one that decides, and say where you would differ from Marlow's answer.
Stagefront sells tickets to stadium shows where two hundred thousand people press the same button at 10:00, and an oversold seat is a lawsuit. Apply today's rule to its seat count: name every copy, say which one decides, and name the one thing you would refuse to cache anywhere at all.
Next lesson
031 Consistent Hashing: Adding a Node Without Reshuffling Everything. Phase 1 bought boxes and asked what each one cost; Phase 2 opens them, starting with the question a cache or a store cannot answer gracefully, which is where a key lives when the number of machines holding keys changes.