Marlow Books is a four person online bookshop that exists only in this course. In June its founder rebuilt the database replica from nothing, kicked the copy off some time after dinner, and found it finished when he opened the laptop the next morning. He wrote one word in the runbook. Overnight.
On a Thursday in July he needed that word to be a number.
Lesson 011 bought Marlow a warm copy of its database and noted that the founder had never once practised switching to it. Lesson 024 is what that cost: fifty lines of Python promoted the replica during a ninety second network hiccup, the shop signed customers out for an hour and fifty minutes, and the script went in the bin. What lesson 024 argued for at the end was a rehearsal. Promote the replica on purpose, on a quiet morning, with nobody's checkout in the way, and then rebuild it.
The rebuild is the whole price of the rehearsal. While it runs, the shop has one copy of itself. If that costs a night, you'll rehearse once a year and dread it. If it costs the length of a coffee, you'll do it monthly and stop being afraid of the switch.
So: how long is the rebuild? The only record anyone had said overnight.
He did not have a measurement, and he did have four minutes. Lesson 016 put box A's data directory at 180 gigabytes, six years of orders, reviews, prices and sessions. A base backup is a file copy, so the only other thing needed is a rate, and Marlow has never measured one. Two are lying around in this course. Lesson 013 timed a restore drill at just under 200 megabytes a second for Galewatch, the wind turbine telemetry service this course keeps coming back to, and lesson 002's van full of hard drives managed about 230 over five hundred kilometres of motorway. Both are bulk copy numbers off somebody else's hardware.
Take the smaller one and then insult it. Call it 50 megabytes a second, a quarter of the only measurement anybody has, on the grounds that the estimate should be embarrassing rather than optimistic.
180,000 MB / 50 MB per second = 3,600 s = one hour
180,000 MB / 200 MB per second = 900 s = fifteen minutes
A hundred and eighty gigabytes at fifty megabytes a second is an hour. At the rate lesson 013 actually measured it is a quarter of that. So the envelope says somewhere between fifteen minutes and an hour, and the runbook says eight.
A factor of eight is a disagreement rather than a rounding error, and one of the two parties is wrong.
On the Friday morning he ran it again with a stopwatch. Forty one minutes. That is 73 megabytes a second, and the useful part is what it rules out. Lesson 002's SSD streams at two gigabytes a second, so the disk could have read 180 gigabytes in ninety seconds. Seventy three megabytes a second is 585 megabits: a gigabit link with the headers and the shop's own traffic taken off it. The backup had never been anywhere near the disk. It was limited by the wire between two machines, and it had been sitting there finished since before midnight while everybody was asleep.
Nothing had been slow. The runbook had recorded the time a human opened a laptop.
The rehearsal is in the calendar for the first Saturday of every month now, which is a thing you will actually do when it costs forty minutes on one copy, and a thing you will never do when it costs a night. The payout job lesson 001 put on the first Sunday is a day later, so they do not collide.
Four minutes with a pen beat eight hours of recorded history, and it did not win by being more accurate. It won by being independent. An estimate built out of figures you already trust is a second opinion about your own system, and the valuable thing about a second opinion is never that it is right. It is that when it disagrees with you, you go and look.
The whole list of numbers you have to know
It is shorter than people expect, and the reason it is short matters more than the list.
Seconds in a day: 86,400. Minutes in a year: 525,600. Those two do not change, they divide into everything, and between them they convert almost any rate a design review will throw at you. One of anything per second is 86,400 a day, about 2.6 million a month and about 31.5 million a year. Backwards, a million a day is a bit under twelve a second. Lesson 024 used the minutes to show how little of a year a ticketing business actually happens in.
Powers next, because storage and identifiers live there. Two to the tenth is about a thousand, two to the twentieth is about a million, two to the thirty second is a bit over four billion, and the signed half of that, 2.1 billion, is where auto-incrementing ID columns actually run out, which is the version of the number you want in your head.
Then lesson 002's latency ladder, which you should not memorise as values. Memorise the ratios it ended on: memory is about a thousand times faster than an SSD on a random read, an SSD a hundred times faster than a spinning disk, a data centre round trip about five SSD reads, and a round trip across the world four hundred of those. Hardware moves the values and has barely touched those four ratios in twenty years, and the last one cannot move at all, because it is distance over the speed of light in glass. You cannot buy a shorter Atlantic.
Prices are the part everybody tries to memorise and the part that rots fastest. Lesson 016 paid about two and a half cents a gigabyte a month to keep objects, about ten cents a gigabyte to send them, and about forty cents a million to answer requests for them. Those three numbers will be wrong within a couple of years. The ratio underneath them will not: sending a gigabyte once costs about four times as much as keeping it for a month. Learn the ratio, look up the price.
And then stop, because the best numbers in any estimate are your own. Marlow's 0.2 millisecond primary key read, its 40 requests a second on a Tuesday, its 180 gigabyte data directory: every one of those beats a figure from a blog post, because they came off the machine you are arguing about. A memorised constant is what you reach for when you have nothing. Most of the time you have something, and you have not looked.
One line at a time
Here is the method, and it is almost insultingly simple. Write the question at the top. Then write one line for each dimension the question actually has, in this order: bytes, writes, reads, money, and whatever those divide into. Every line carries its units, because the mistake that actually happens is dividing gigabytes by turbines and reading the answer as machines. Round hard, and round in a direction you picked rather than whichever way the digits fall.
The order is not arbitrary. Bytes are the hardest line to argue with, being arithmetic over a data model somebody already agreed to. Money goes last because it is the only line that belongs to somebody else, and putting it last stops you working backwards from a budget to a fantasy.
Try it on something nobody has sized yet. Galewatch collects a reading from each of its wind turbines every two seconds, and lesson 014 added a two hundred turbine farm that took it to 550 readings a second across eighteen sharded machines. Sales is now talking to a prospect with three thousand turbines of their own. One contract, nearly triple the fleet. What does yes cost?
readings 3,000 / 2 s = 1,500 a second
bytes 3,000 x 5 GB per year = 15 TB a year
machines 15,000 GB / 720 GB = 21
money 21 x $250 a month = $5,250 a month
Carried out in words, because those four lines are the lesson. Three thousand turbines each sending a reading every two seconds is fifteen hundred readings a second, on top of the 550 a second lesson 014 left the fleet at. Lesson 013 worked out five gigabytes per turbine-year once retention went to a full year, so three thousand turbines is fifteen terabytes a year. Lesson 013 also fixed the maximum size of one machine at 720 gigabytes, because a customer contract promises a one hour restore and 720 gigabytes is what a one hour restore can move. Fifteen thousand gigabytes divided by 720 is twenty one machines. At lesson 005's $250 a month for a machine, that is $5,250 a month of new hardware, or double it the day Galewatch's engineers get the per-shard replica lesson 011 left them asking for. No reads line, because the dashboards are a handful of engineers. A machines line instead, which is the bytes line divided by the size of a box.
Now read the four lines and pick the answer, which is a separate skill from producing them.
It is not the write rate. Fifteen hundred readings a second spread over twenty one machines is about seventy a second each, against the five thousand writes a second lesson 005 wrote down and lesson 013 made a per-shard figure, so every new shard runs at one and a half percent of its ingest ceiling. It is not the money either, at least not yet, because $5,250 a month against a contract for three thousand turbines is a question for whoever knows the contract price, and the envelope should hand it over rather than guess.
The answer is the third line. Eighteen machines become thirty nine. Lesson 013 said the disappointing part out loud when it made sixteen: you do not end up with no primary, you end up with sixteen of them, each carrying every problem lessons 011 and 012 described, and nobody can hold the whole system in their head. Thirty nine is that sentence with a worse number in it. The deal does not need a line of new code and it does need a person, and a salesperson can be told that on the phone.
Four minutes. Four lines. One finding, and it was in the line nobody expected.
Get the answer twice, through a different door
The fifteen terabytes came out of a constant: five gigabytes per turbine-year. Where did that come from? Lesson 013 divided four and a half terabytes by nine hundred turbines. If the four and a half is wrong, the fifteen is wrong by exactly the same factor, and no amount of recomputing will show it.
So compute it from the bottom instead. Eighty six thousand four hundred seconds in a day, one reading every two, is 43,200 readings per turbine per day. Times 365 is about 15.8 million a year. At the 200 bytes lesson 002 sized a reading at, that is 3.15 gigabytes of raw rows. Lesson 001 put the fleet at 2.8 terabytes raw and lesson 013 at 4.5 with the indexes on, a multiplier of 1.6, so call it 5.0 gigabytes.
Five gigabytes a turbine-year, arrived at from seconds and bytes rather than from a division somebody else did. That's reassuring, and I want to be careful about how reassuring.
It is not an independent check. It re-derives the same constant from the same model, so it will catch a slipped decimal point and it will catch nothing else. A second route through the same assumption tests your arithmetic. Only a different assumption tests your model.
What would actually test this one is asking what a reading is. Two hundred bytes describes the turbines Galewatch has. A customer whose firmware reports forty sensor channels instead of a handful might send twelve hundred, and then every byte-derived line moves by six at once while the readings-a-second line does not move at all. That single question is worth more than both derivations, and it takes ten seconds to ask.
Lesson 017 has the version of this that works. It needed to know how long a four hour backlog takes to land after a farm's radio link comes back, and it got there twice. Once as plain division: 432,000 buffered readings against a path that stores five thousand a second is 86.4 seconds of work. Once as a queue, filling at seven and a half thousand a second for 34.6 seconds to a peak of 259,000 messages, then draining at 4,970 a second for another 52. Thirty four and a half plus fifty two is about eighty seven seconds.
Less than a second apart, from a division and from a simulation of a buffer filling and emptying. Those two could have disagreed, which is the only reason their agreeing means anything.
Estimate the thing you cannot afford to be slow at
Go back to that 720 gigabyte machine size, because of where it came from.
It did not come from the query rate. Galewatch's writes never moved when retention went from a fortnight to a year: 450 readings a second in October, 450 in December, CPU never above thirty percent, and the flagship farm panel still answering in under ten milliseconds. Every number a capacity planner would reach for said the system was fine, and it was fine, right up until a scheduled restore drill on a Thursday in February ran from nine in the morning to twenty past three against a contract that promised an hour.
Six hours twenty for four and a half terabytes is just under 200 megabytes a second. An hour at that rate is 720 gigabytes. That is where the shard size came from, and every other number in lesson 013 followed it.
Lesson 013 named the rule for its own purposes: you size a cut for the operation you cannot be slow at, and for most companies that operation happens once a year, at night, in a drill nobody watches. Widen it from cutting a database to any estimate at all, and it is the habit people skip most, because the steady state is the easy number and it feels like the important one. Look instead for the rare operation with a deadline attached, and price that. Marlow's disk was sized for a shop and killed by a third copy of a nightly dump, which lesson 016 spent four hours and fifty minutes on. Marlow's payout job reads a year of order lines once a month and is the largest single thing that ever happens to that database. Stagefront is an event ticketing service in this course where a stadium show goes on sale at exactly 10:00 and two hundred thousand people press the same button, and lesson 003 priced its year at forty on-sales of about ten minutes each: four hundred minutes out of 525,600, which is under a tenth of one percent of the year carrying the entire business.
Estimate Stagefront from its average traffic and you will buy a system that cannot sell a ticket. Estimate its storage from those four hundred minutes and you will buy a warehouse for a shed.
Which is the honest complication. The rare operation sets some dimensions and the ordinary one sets others, and part of doing this well is knowing, per line, which clock you are on. Lesson 016 said it plainly when it priced Marlow's covers at 40 requests a second rather than at Christmas, because a bill is a month long and Christmas is a week.
The average is nobody's request
Lesson 014 found the cost of estimating the middle, and it arrives as something you own rather than something you spend.
Galewatch's shards are farms, and farms are not the same size. Spread four and a half terabytes over the sixteen machines lesson 013 sized and the average shard holds 281 gigabytes. Kilmore Sands holds 700. You buy identical machines, and you should, because the alternative is a runbook with an exception in it. Identical means every machine can hold the biggest shard. Sixteen machines able to hold 700 gigabytes each is 11.2 terabytes of disk bought to store four and a half.
Forty percent used. Nobody has ever found that on a dashboard, because every dashboard averages.
You estimate the average and you buy the maximum, and the gap between those two numbers is the bill. So estimate the fattest value on purpose, separately for every dimension you care about, because skew in bytes and skew in writes are different numbers about the same fleet.
Lesson 019 did the opposite thing and was equally right. It needed to know how much disk a table of idempotency keys would take, so it took Marlow's Christmas packing rate, one order every five seconds, and applied it to all twenty four hours of every day. That is the busiest hour of the busiest week of the year, priced as though it were forever. 86,400 over five is 17,280 orders a day, at about three hundred bytes a row, so roughly five megabytes.
Then it said in the text that this was deliberately the wrong way round.
That is the move worth stealing. When the pessimistic number comes out small enough to ignore, being wrong in a named direction ends the conversation instead of starting one. Nobody needs to argue about the order rate, because the answer is five megabytes even if you triple it.
Which is really a rule about writing the assumption into the line. Lesson 013 put twenty three minutes of per-shard restore in its table and then said, underneath, that this holds only if all sixteen shards restore at once and if the 200 megabytes a second belongs to the machine rather than to one pipe they would all be pulling through. Lesson 016 said its one-in-ten cover fetch rate was a guess. Neither lesson got weaker for it.
An estimate gets quoted by people who did not do it, usually in a meeting you are not in. Whether it survives that is the only real test, and a number with its assumption stapled to it survives. A bare number becomes a fact about three months later, and then somebody builds on it.
What the number is actually for
The total is rarely the point. What you want out of an envelope is which term dominates, because that tells you where to spend the next month.
Lesson 016 priced Marlow's 216 gigabytes of cover images in object storage: $5.40 a month to keep them, $4.16 to answer the requests, and $187 to push the bytes down the wire. One of those is thirty five times the storage line and it is the one nobody was discussing.
Take the thirty five apart, since it is two boring ratios multiplied. Marlow ships 1,872 gigabytes a month and stores 216, so it sends 8.7 times its entire library every month. And a gigabyte sent costs four times a gigabyte kept. Eight point seven times four is thirty five. Neither ratio is surprising on its own, and together they are the whole answer. Any conversation about storage cost that stops at the gigabytes is carefully measuring the small term.
The same shape lives inside the database. Lesson 010 worked out that Marlow's Christmas peak needs 1.04 seconds of database work every second, and that the full text searches are 0.96 of it. Book pages are 96 percent of the traffic and about seven percent of the work. So the entire page path, the thing everyone looks at, cannot be optimised into more than a seven percent saving, and one request in twenty five carries the other ninety three. You get that from two multiplications, before anybody opens a profiler.
Lesson 022 got its best result the same way, from one division against an intuition. Everybody knows a popular file on a content delivery network is hot. Marlow's covers get about four fetches a second across the whole shop, spread over 1.2 million covers, which is one request per cover every 300,000 seconds, or three and a half days. Cut that nine ways across the edges that actually serve Marlow and it is about one request per cover per edge per month. A file asked for once a month is not hot anywhere, at any expiry, and that is why a 62 percent hit rate on immutable images was the correct number rather than a bug.
Now the limits, because this tool is worse than people who love it admit.
An envelope cannot see contention. It will happily tell you that 1.04 seconds of work fits in sixteen cores and say nothing about the lock, the pool, or the knee lesson 001 put at around 70 percent of any resource you care to name. Anything that queues is outside its reach, and lesson 002's Little's law is about as much queueing as fits on an envelope at all.
It cannot give you a constant factor either. A factor of two out is noise and you should expect it. A factor of ten is a finding, and you chase it.
And it is a claim about a shape rather than a prediction. The shape is usually right and the total is usually not, which is fine, because you were never going to buy hardware off the back of it. You were going to decide which of four things to look at first.
Then occasionally the shape argues with a number somebody already has, by a factor of eight, and you go and look, and it turns out the number was a story about when somebody opened a laptop.
Recap
Four minutes with a pen, and the point is independence rather than accuracy. An estimate built from figures you already trust is a second opinion about your own system. Its value shows up when it disagrees with something you believed, which is why the answer to a disagreement is to go and measure, not to adjust the estimate.
The list of numbers worth memorising is short on purpose. 86,400 seconds a day, 525,600 minutes a year, the powers of two at a thousand and a million and four billion, and lesson 002's ladder read as ratios rather than values. Prices rot; the ratio under them does not, which is why "a gigabyte sent costs four times a gigabyte kept for a month" outlives the ten cents it came from. Your own system's measured numbers beat all of it.
One line at a time, with units, in the order bytes, writes, reads, money. Bytes first because they are hardest to argue with, money last because it belongs to somebody else. Then read the lines and find the one that is the answer, which for Galewatch's three thousand turbine prospect was not the traffic and not the bill, but eighteen machines becoming thirty nine.
A cross-check through the same door is not a cross-check. Recomputing five gigabytes a turbine-year from seconds and bytes tests the arithmetic and leaves the model untouched. Only a different assumption tests the model, and asking what a reading is made of was worth more than both derivations. Lesson 017's storm timing is the real thing: a division and a queue simulation landing less than a second apart, when they could have disagreed.
Estimate the thing you cannot afford to be slow at. Galewatch's shard size came from a restore drill and a contractual hour, not from 450 readings a second, and everything else in that design followed the number a rare operation set. Stagefront's entire business happens in four hundred minutes out of 525,600, and each line of an estimate has to know which of those two clocks it is on.
The average is nobody's request. You estimate the average and you buy the maximum, which is how sixteen machines sized for a 700 gigabyte shard ended up 40 percent full. Estimating deliberately in a direction you name is the other half of the same habit: lesson 019 priced a day of keys at Christmas rates, said so, and got an answer small enough to end the argument.
The output is which term dominates. Marlow's covers bill is 8.7 times the library shipped each month times four times the price of keeping it, and the $187 was never going to be found by counting gigabytes. Its Christmas database load is 96 percent book pages by count and seven percent book pages by work. An envelope will not see contention, will not give you a constant factor, and is wrong by a factor of two as a matter of routine. A factor of ten is not an error, it is the finding.
Check your understanding
Marlow's rebuild is 41 minutes at 73 megabytes a second over a 180 gigabyte data directory. Name two changes to the shop that would turn that into four hours, say which is likelier to arrive first, and say what belongs in the runbook so nobody redoes this.
The Galewatch prospect turns out to run firmware that reports forty sensor channels, so a reading is about 1,200 bytes rather than 200. Redo the four lines. Say which of them move by six, which one does not move at all, and whether you would still recommend taking the contract.
Marlow is offered an object storage class at a third of the price per gigabyte a month, with egress and request prices unchanged. Work out the new monthly bill from lesson 016's figures, then say how much engineering time the migration is worth.
Pick an operation from a service you have worked on that runs less than once a month and has a deadline attached to it. Estimate how long it takes using only numbers you already have. Then say what single measurement would tell you whether you are wrong by a factor of ten.
Stagefront's four hundred minutes a year carry the business and the other 525,200 do not. Name one capacity number that must be estimated from the window and one that must be estimated from the rest of the year, and say what going the wrong way round costs in each direction.
Next lesson
026 Observability Basics: Logs, Metrics and Traces. Every estimate today was assembled from numbers somebody had already measured, and next lesson is about where those numbers come from, which ones a service should be emitting before anybody thinks to ask, and why the useful ones are almost never the ones on the default dashboard.