Marlow Books is an online bookshop run by four people, and it exists only in this course. On Saturday 31 October 2026 its founder closed the shop's checkout on purpose, for two minutes, and twenty two seconds later the shop sold a book.
He spent most of the morning looking for the bug. There wasn't one.
The failover rehearsal lesson 025 put in the calendar belongs to the first Saturday of the month, once the founder measured the replica rebuild at forty one minutes rather than the night the runbook claimed. Lesson 032 noted that this is the one rehearsal he actually does. Promote the replica at nine, rebuild it, be finished before lunch. October's first Saturday was the book fair in lesson 058's hall with the concrete floor and no signal anywhere in it, so the whole morning moved to the end of the month.
What was new was the second line on the list, and lesson 027 asked for it after September's provider outage: flip the degraded mode flag for two minutes at forty requests a second and go and look at your own shop, because a mode nobody has entered is a sentence everybody believes. Lesson 042 pointed out that Marlow can do this at all only because lesson 027 built a two field switch read from a file on each box, which lesson 028 widened to a checkout field holding "open", "closed" or "delayed", changed in thirty seconds over ssh with no deploy and nothing from the database.
Eight weeks before Christmas, he did it.
The price was known in advance, and it is why nobody does this. Lesson 027 derived an order every fifty seconds from an ordinary forty requests a second, carrying lesson 009's Christmas conversion rate onto a quiet day, which nobody at the shop has ever checked. Two minutes shut is 2.4 orders. Call it two books, deliberately not sold, to find out what the shop looks like when it is closed.
At 09:50 he opened a session to box A and set checkout to "closed". Then he opened a session to box B and did the same.
Between those two edits, thirty seven seconds.
The confirmation email came through at 09:50:22, which is the sort of thing you notice when you are sitting there watching for exactly the opposite. He read lesson 026's access log, which carries a path, a status, a byte count and a duration on every request, and the order is in it, a 200, served by box B, while box A had been refusing checkouts for twenty two seconds.
Nothing was broken. Box B had not been told yet.
Thirty seven seconds, priced twice
Lesson 048 has a section called Marlow's two files and the row with no limit, so none of what the founder found is news to this course. Lesson 030 counted the copies of every fact the shop depends on, with a column for how long two copies may disagree, and that row is the one with nothing in its last column: two files, on two boxes, with nothing on earth reconciling them. Lesson 048 picked it up twelve lessons ago, agreed with every one of lesson 027's reasons for the files, refused the textbook answer of a configuration service for lesson 027's own reason, and then wrote down the fix. One writer instead of two edits. A command that writes both boxes and reports which ones it wrote.
Nobody built it. That is what the thirty seven seconds are.
Lesson 048 said the disagreement window becomes the length of that command and left it there, because a window nobody has measured has no number in it either. Saturday supplies the number for the other end, the end the shop is actually running on. Thirty seven seconds, which is how long it takes to type a path into a second ssh session.
Now run the window. Thirty seven seconds at forty requests a second is 1,480 requests, and the balancer lesson 006 installed picks a box by least connections with no stickiness anywhere, so about 740 of them met a box that still thought the shop was open. Half the shop's customers were looking at a working checkout and half at an apology, and which one you got depended on which box answered.
Then look at what a customer could do with that, because the page and the press are separate draws. Somebody served by box B saw a shop that was open, pressed Pay, and had their POST land on box A, which refused it. Somebody served by box A read the banner, pressed Pay anyway, and got a confirmation from box B. Both wrong answers were live at once, in opposite directions, for thirty seven seconds.
No ordering fixes that. Close A first and the customers on B keep buying into a shop that is shutting; close B first and the same thing happens with the letters swapped. For those thirty seven seconds Marlow is two shops.
Which is lesson 024's sentence arriving at a bookshop, thirty six lessons after it was written. Lesson 024 compared a forty three second partition against twenty four hours and eleven minutes of recovery, a ratio of about two thousand to one, and said the length of a partition tells you almost nothing about what it costs, because recovery reconciles decisions rather than packets. Marlow's partition is thirty seven seconds long on Saturday morning and thirty seven seconds long in Christmas week. Same hole, same length. What changes is how many decisions fall into it.
Count them. On Saturday the window held 0.74 orders at lesson 027's rate, and it produced exactly one, which the shop confirmed while its founder believed it was closed, and no refusal of anybody who had been promised a shop, because there was only one order in the window and both of its draws came up box B. Christmas week runs 400 requests a second and lesson 009's conversion rate puts that at an order every five seconds, so the same thirty seven seconds holds 7.4 orders, about half of them refused after the shop had said yes, and 14,800 requests of which about half meet the box that has not been told.
Ten times the traffic, ten times the orders, and the identical hole costs ten times as much.
Lesson 050 published the general warning, that a drill at two in the morning measures the half you were not worried about. The useful refinement is which half, and it falls out of the arithmetic above. A duration rehearses honestly at any volume, because thirty seven seconds is thirty seven seconds whether anybody is shopping or not. A count does not, because a count is that duration multiplied by a rate, and the rate is exactly what you removed to make the rehearsal affordable. So rehearse to measure the duration, then do the multiplication yourself for the hour you will really be in trouble. The founder bought a real number for two books, which is a good trade, and the number he bought was not the one that will hurt him.
Lesson 048's command closes it for four lines of shell. The window becomes a few seconds or a loud error, which was 048's phrase and matters in both halves: call it three seconds, and Christmas week holds 0.6 of an order instead of 7.4, a factor of twelve, while a run that fails after writing one box says so instead of leaving a coin toss in production. Thirty seven seconds to three, for four lines, on the one row of this course's table that had no limit in it.
Then ask lesson 030's question of the fix itself, which takes a minute and which nobody asks. What sentence did the founder just stop being able to say? A command that writes both files together has deleted the only way Marlow had to put a different answer on one box on purpose, and lesson 054's canary at this shop is exactly that: one box of two, serving the new code while the other one holds the control group. The repair for the repair is a second line. Keep the per-box command, add the both-boxes one, and name which is which.
Twenty nine mechanisms
Phase 2 is twenty nine mechanisms and today. A capstone that installed them would be a worse shop and a longer lesson, so here is the sorting instead, because the sorting is what a four person company actually has to do and nobody publishes theirs.
Fourteen of the twenty nine are refusals, and the course priced every one.
Lesson 031's ring wants more than one cache node and Marlow has one. Lesson 032 said in plain words that a human majority of one is the right answer for four people and not the embarrassing one people think, and lesson 033 priced the alternative at three machines to get what you thought two would buy. Lesson 037 said the shop should not run a log and that saying why is the useful half, since replaying it means sending the confirmation email again, which is the bug lesson 036 spent a week removing. Lesson 039 found that moving the pick before the charge is a business change pretending to be an architecture. Lesson 041 found no backpressure story at all in a drainer publishing one row every fifty seconds. Lesson 047 counted what a cache inside each process would cost: a rendered book page goes from ten copies to eighteen, and eight of the eighteen have no delete command. Lesson 048 refused the configuration service. Lesson 049's verdict on a mesh was that it would hand the founder eight more processes to restart on a Friday afternoon and not one new fact about the payment provider. Lesson 050 said not to buy a second region at all, because the shop's outages are a certificate, a Friday deploy, a payout job and one database, and a second region removes none of them while adding a second place for the founder to deploy to on a Friday afternoon. Lesson 051 said to build none of a pipeline, since the two batch jobs it has are built and neither is a project. Lesson 053 said not to encrypt the covers, because a stolen disk full of book covers tells somebody what is in the shop window. Lesson 055 priced an autoscaler at three thousand dollars a month to buy thirty book pages a second. And lesson 058 priced a reservation on the book fair's two phones at two sales and two extra disappointed customers, and called it a bad deal for eight interchangeable books.
| The mechanism | Why the shop refuses | What the refusal costs |
|---|---|---|
| 031 a hash ring | one cache node | nothing |
| 032, 033 a majority | two boxes, no third | no failover at 04:00 |
| 037 a log | one consumer, one email | no replay, and none wanted |
| 047 a cache per process | eighteen copies, eight undeletable | lesson 002's half millisecond |
| 050 a second region | every outage is shared cause | nothing it would miss |
| 055 an autoscaler | the floor is redundancy | thirty pages a second for $3,000 |
Out loud, since the audio skips tables: five of those six cost the shop nothing it would notice, and the one that costs something, the missing majority, costs the shop a dead Saturday morning if box A dies at four in the morning, which has happened once in six years and is the price the founder has decided to keep paying.
Four of the twenty nine are not purchases in the first place. Lesson 034's clocks describe a problem Marlow has never had, because one Postgres decides the order of two writes and a created_at on an order is decoration, and the shop inherits every paragraph of that lesson on the day it has a second place that accepts writes. Lesson 043 explains the engine underneath rather than offering one to pick. Lesson 056 is a way of reading a bill and lesson 057 a way of reading a credential.
Which leaves eleven, ten of them running and one the shop still owes, and the list is less interesting than how they arrived.
Five of the ten went in on purpose, four of them after a morning with a number on it. The cache-aside pattern lesson 046 names went in the week after lesson 001's podcast Friday. Lesson 040's outbox followed nine minutes of broker maintenance that produced eleven orders, paid, logged and never announced. The compatibility discipline lesson 052 argued for followed an email telling a two person poetry press it was owed $128,400. Lesson 054's box by box deploy script was the first one this shop has ever run, on Friday 9 October 2026, eight months after the balancer that made it possible. The fifth had no morning behind it: lesson 038's two reorder consumers were asked for by the two buyers with a finger on lesson 030's diagram, and cost twelve lines each without the founder opening the checkout to add them.
The other five arrived sideways.
Marlow owns a lock, which is lesson 035's subject, and it exists because a crontab got copied onto a second machine and both boxes emailed the publishers a CSV on the first Sunday in October. What went in was lesson 007's ten line claim on a job_runs row, which lesson 018 later gave a finished_at and an outcome for a different reason again. The shop's circuit breaker is the outlier detection on the twenty dollar balancer, bought in January to stop a dead box taking traffic, and lesson 042 observed that nobody there has ever used the words. Its object store was bought because a 500 GB volume filled at ten to four in the morning. Its inverted index sits in a separate book_search table, which lesson 045 found is the reason the shop survived June, because a year earlier the founder did not want a two and a half kilobyte column on the table every book page reads; lesson 045 calls that a decision nobody wrote down. And what made lesson 036's ninety minutes of triple sending survivable was four email workers against a fifth of a message a second, about eighty times more consumer than the shop needs, never bought as a margin at all.
Ten mechanisms, and not one of them was bought because somebody read about the mechanism.
Lesson 030 put the general version as a rule, that the architecture is the ordered list of measurements rather than the list of boxes. The capstone's version is narrower and less flattering: half of Marlow's Phase 2 inventory arrived for a reason other than the one that now makes it matter. Nobody bought it as the thing it turned out to be, nobody wrote down what it was holding up, and lesson 042 found the same thing one company over at Galewatch, which collects a reading every two seconds from each wind turbine on the farms it monitors. Lesson 037's log took Galewatch's blast radius from all nineteen farms down to the four partitions one writer owns, and four is twenty four partitions divided by six writers rather than a blast radius anybody chose.
A side effect is not worth less than a decision. It is worth less than a decision you wrote down, because the next person to touch it will not know it was load bearing.
Lesson 030 is owed one correction while we are here, and it improves both halves. Lesson 030 counted seven fixes aimed at the cheap ninety six percent of Marlow's traffic, against two aimed at the expensive remainder that made it no cheaper, and read that as a shop that spent six years optimising seven percent of its work. Lesson 045 supplies the other half: it took that sixty milliseconds apart and found most of it is ranking, which a posting list cannot answer and core Postgres cannot avoid, so the expensive path was already near its floor. Lesson 010's 0.96 of a busy second is six percent of a sixteen core machine. Nothing moved it in thirty lessons because nothing needed to, and the real criticism of that A4 page is that it cannot tell you either half was ever measured.
What an afternoon buys
A refusal done properly does not end at no. It ends at a cheap substitute, named in the same paragraph. It took sixty lessons to notice that.
Lesson 048 refused the configuration service and left behind one command and two fields on a health endpoint. Lesson 042 refused the payment breaker and left behind ship the fallback first and rehearse it by hand, which is the Saturday. Lesson 053 refused encrypting the covers and left behind spend the attention on the credential that can write to that bucket. Lesson 050 refused a second region as protection for the on-sale at Stagefront, the ticketing service in this course where two hundred thousand people press the same button at ten in the morning, and left behind a load rehearsal that reproduces the window, because the thing that company is scored on cannot be measured at two in the morning.
So the hardening list is not a list of things nobody thought of. Most of it was designed by an earlier lesson that had just decided not to buy the expensive version, and the rest by a lesson that had just finished paying for its absence.
Five items. The shop buys no machine at all.
Lesson 048's one command for the switch. Lesson 048's other half, the switch's value and its file's modification time on the health endpoint lesson 006 already requires of both boxes, so that the question "do my two boxes agree" has an answer somebody can look at. The signed in customer id on lesson 026's access log line, which lesson 059 priced and which would make the December Saturday answerable eleven months late, when one of the temporary Christmas staff pressed Closed at 10:40 and nothing anywhere recorded who. An alert on box A's disk, which lesson 016 earned at four hours fifty minutes and lesson 030 put at the top of its list of things argued for and never built, still unbuilt nearly a year on. And the miss rate as a rate, which lesson 008 asked for because ninety six to eighty eight percent reads as a rounding error while the same week drawn as misses is a line going up threefold, with a line on it at the 260 book pages lesson 005 measured on the pair. Shedding needs a trigger and a trigger is a number.
Notice what is not on it, since a capstone is where somebody usually adds a box. Lesson 055 measured this fleet at about fifteen percent loaded almost all the time: forty requests a second against that same 260. Christmas offers 384, which lesson 008 called 148 percent of the ceiling, and the thing that carries it is the cache lesson 001 shipped in an afternoon rather than any machine. A third box buys five more pages a second by lesson 005's own measurement. The shop is sized, and it is sized by its failure budget rather than by its load, which is lesson 055's finding and the reason there is nothing here to buy.
Each of the five also has to answer lesson 059's question, which is how anybody would later prove it was ever switched on. Three answer it for free: the command is a file in git with a date on it, the health fields are readable by anyone, and the log field is itself the proof. The alert and the graph do not, and lesson 026 gave the move already, which is to fire every signal once on purpose, because a signal that has never moved is untested and a healthy zero looks exactly like a disconnected wire. Put both on the first Saturday, beside the promotion.
Price all five in lesson 056's three shapes and the total is the point.
No rent. Nothing to sign for, nothing added to the $1,598 a month lesson 056 itemised, of which $1,520 is already rent. No meter either, beyond a few more bytes in a log file that lesson 059's fourth question is right to ask about. Every one of the five is lesson 056's third shape, somebody's afternoon, the one that never reaches an invoice and is usually the biggest thing on the list.
Which is why none of them has happened. The disk alert has been outstanding since lesson 016, through two capstones, both of which named it. It is one threshold and one line of cron. Meanwhile the shop pays five hundred dollars a month for a replica, twenty for a balancer and a thousand for the pair of boxes, and every one of those three took somebody signing something.
Rent gets a decision and an afternoon gets a Saturday. Rent arrives as a quote with a number on it, which somebody has to approve, which means somebody has to think. An afternoon has no price, so it never reaches the part of the week where things get approved. At a four person company the Saturday is the scarce resource and the money is not, which is the exact inverse of how the two get discussed.
Lesson 056 found that the cheapest and the simplest architecture are the same thing far more often than a design review admits. Put that beside the sorting above and the honest summary of this shop, after sixty lessons of being broken on purpose, is that it is correctly built, half of it by accident, with five afternoons on its list and no machine to buy.
That reads as failure on every dashboard anybody would point at it.
One of the five did get done on the Saturday. The founder wrote lesson 048's command before lunch, because he was already in the ssh session with both hosts in his shell history and it was quicker than writing it down for later. The other four went onto the A4 page from lesson 030, in the list of things that have been argued for and never built, where the disk alert has been sitting since before the page existed.
The fix that ships is the one your hands are already on. Nobody would put that in a design document, and it has predicted more of this shop's six years than any principle in this course.
Recap
The cost of a partition is not its length. Lesson 024 said so about a forty three second link and a day of recovery, and Marlow's version is thirty seven seconds of two files disagreeing, which is thirty seven seconds in a rehearsal and thirty seven seconds in Christmas week and costs 0.74 orders in one and 7.4 in the other. A hole is priced by the decisions that fall into it, and the decisions arrive at a rate you removed to make the rehearsal affordable.
So split what a rehearsal can tell you. A duration rehearses honestly at any volume, because a duration does not care whether anybody is shopping. A count is that duration times a rate, and the rate is the thing you took away, so rehearse for the duration and do the multiplication for the hour you will really be in trouble. Lesson 050's drill at two in the morning measures the half you were not worried about, and this is which half.
A degraded mode entered by hand has two edges and no safe order for either. Closing box A first lets the customers on box B buy into a shop that is shutting, and closing box B first does the same with the letters swapped, because a page load and the press of Pay are separate draws at a balancer with no stickiness. Only the size of the gap moves, which is why lesson 048's four lines are worth more here than any mechanism in Phase 2.
Twenty nine mechanisms sort into fourteen refusals the course has already priced, four things that are not purchases, ten the shop is running and one field it owes. Write your own sorting down, because the refusals are the half of an architecture nobody records and the half a new engineer will undo.
Half of what a small system runs is load bearing by accident. Marlow's lock exists because a crontab got copied, its breaker is a balancer feature nobody named, its object store was bought by a full disk, its search index sits in its own table because of a decision nobody wrote down, and the margin that absorbed lesson 036's ninety minutes was eighty times more email worker than the shop needs. A side effect is worth less than a decision you wrote down, because the next person to touch it will not know it was holding anything up.
A refusal done properly ends at a cheap substitute rather than at no, and the substitute is the thing nobody builds. Every item on Marlow's hardening list was designed by an earlier lesson, either the one that refused the expensive version or the one that had just paid for its absence, and all five are afternoons: no rent, no meter, lesson 056's third shape. Rent gets a decision and an afternoon gets a Saturday, which is how a disk alert survives two capstones that both named it. The fix that ships is the one your hands are already on.
Check your understanding
The first Saturday in December is the 5th, three weeks before Christmas. Say whether you would run the closure rehearsal again then, what you would change about it, and what you would do instead if your answer is no.
A colleague wants the two files replaced by a row in Postgres, so the switch can never disagree with itself. Price what that row would cost on the specific afternoon the shop needs the switch most, and say what you would build instead if the gap had to reach zero rather than three seconds.
You join a four person company and find a dozen mechanisms running and no record of why any of them is there. Say how you would find out which ones are load bearing, which published number you would look for first, and what you would write down about the ones that turn out to be side effects.
At Stagefront an oversold seat is a lawsuit. Sort today's fourteen Marlow refusals for it instead: name three that flip to purchases, say which figure flips each one, and name one that stays refused for a reason that is not about scale.
Galewatch has a degraded mode nobody has ever entered on purpose. Design the rehearsal: what you would turn off, at what hour, what it would cost, and which of its numbers you would have to calculate rather than observe.
Your own service has a flag, a kill switch or a feature toggle written in more than one place. Work out the gap between the first write and the last, multiply it by your busiest decision rate, and say whether the answer changes what you would build this week.
Take the last three mechanisms your team refused. For each one, say what cheap substitute the refusal left behind, whether it was built, and who would notice if it quietly never was.
Next lesson
061 How to Run a System Design Interview, and a Design Review. Phase 2 ends with a shop whose best decisions were accidents and whose refusals were never written down, which is exactly what a design review exists to ask about out loud.