Marlow Books is an online bookshop run by four people, and it exists only in this course. At 09:20 on Monday 2 November 2026 one of its publishers sent the founder a two line email: our title sold thirty four copies in the seven days to 18 October and five in the seven days after. Did something change at your end?
Something had. The founder was fairly sure he knew what: the title had been on the gift page, which lists twelve books ordered by what is actually selling, and then it had not been. Half an hour, he thought, and he could tell the publisher exactly which week it dropped off.
Lesson 008 is where that page stopped falling over. The ordering is an aggregation over the last seven days of order lines, it takes about 900 milliseconds, and since the Christmas it fell over a cron job has written the result into a small table every ten minutes so the page can read one row. One row. Overwritten a hundred and forty four times a day, every day since.
So there was no record of what the page had said. That was expected, and it looked survivable, because the query is still in the repository and the order lines are still in Postgres with their dates on them. The founder changed the window by hand, ran it as of 19 October, and got the title in fourth place. On the page, then. He ran it as of 26 October and got it in eleventh. Still on the page.
Which contradicted the only thing he was sure of.
He found the rest of it at about eleven. The gift page does not list what is selling. It lists what is selling and in stock, which is obvious the moment you see it and appears nowhere in the name of the job that builds it. The title had gone out of stock on Monday 19 October and come back on the Thursday. His re-run could not see that, because the stock column holds one number, now, and the distributor file lesson 043 priced rewrites eight thousand of those numbers every morning, and nobody keeps yesterday's file. Three days out of seven, against a fall from thirty four copies to five. The two numbers do not obviously fit, and there was nothing left to check them against.
The order lines were a record. The stock level was a reading. The list was a function of both.
At 11:40 he wrote one line to the colleague who handles the publishers, and it is the sentence this whole lesson is about: we don't know what our own home page said a fortnight ago, and I can't find out.
You are designing the service that would have known.
The brief, and the only thing a reader can detect
Twenty five million people open a feed of posts from the accounts they follow, three times each on a normal day, so seventy five million feed opens. Five million posts a day. The average reader follows four hundred accounts; the largest account on the service has ten million followers. A page is thirty items, text with small attachments, and the feed is ranked rather than chronological, which is a product decision somebody else took and the rest of this is the invoice for it. Those figures are mine and everything below comes out of them.
Seventy five million feed opens a day is 868 a second averaged flat and 2,604 at a three times evening peak. Five million posts is fifty eight a second and a hundred and seventy four at the peak. By the standards of lesson 064's key-value store, which absorbed four hundred thousand operations a second on twelve machines, both of those are quiet.
One division before the architecture, because the memorised answer rests on it. Everybody precomputes a feed because a feed is read-heavy. Seventy five million opens against five million posts is fifteen feed opens per post, and lesson 011 published Marlow's read to write ratio at two hundred to one, two writes a second against four hundred reads at the Christmas peak, with lesson 062 adding that an ordinary Tuesday gives the same ratio because both sides fall by a factor of ten together. So this service is read-heavy by fifteen and a four person bookshop by two hundred, and a feed is thirteen times less read-heavy than the shop everybody in this course has been patching since lesson 001.
Then the question lesson 065 makes it impossible to skip. 065 designed a chat system around a dense sequence per conversation, one, two, three, nothing skipped, because a client holding one to forty that receives forty two knows by subtraction that it is missing forty one. Its sentence was that absence has no shape in a sparse order, and Marlow's own lost message was the price of not having one.
A feed has no sequence anywhere in it, and nobody minds.
Nothing promises you every post from everybody you follow. Nothing could. Four hundred accounts producing five million posts a day between twenty five million people offer you about eighty posts, and the service shows you some of them in an order it chose. If three of the eighty never reached your screen you have no way on earth to notice, and you would not call it a bug if somebody told you. A reader cannot detect a missing post.
They detect a repeat instantly. Scroll, and there is the thing you read this morning, in a different position, as if the service had forgotten. People notice that in the first second and they report it, which almost nobody does for anything else a feed gets wrong.
Which is awkward, because three opens at thirty items is ninety slots against those eighty posts. The supply runs out every day, by about ten slots, so the service has to put something in a gap it creates itself, and the only candidates are posts this reader has already been shown.
So the promise inverts. Chat owes you completeness and gets dedup for free, because each message is delivered once into an order. A feed owes you no completeness at all and has to buy the other thing, never showing a reader something they have already been shown, which everybody shortens to dedup and which is the one property its readers police. Lesson 023 has the vocabulary and the warning in the same sentence: the middle of the consistency range is a promise to one reader, read-your-own-write and monotonic reads are cheap because one reader is cheap, and a person is not a session, so the promise your phone was given does not follow you to your laptop. Dedup across a reader's own devices is exactly that promise, which means it cannot live on the client, which means it is state you own, and state you own has a size.
Four lines on an envelope
Lesson 025's order: bytes, writes, reads, money.
Bytes. Five million posts a day at a kilobyte each, counting the text and the row around it, is five gigabytes a day and fifteen with three copies. Over a year that is 5.5 terabytes, which on the two terabyte disks lesson 062 used is under three machines, one more every four and a half months. Attachments go to object storage on lesson 016's argument and are not in that figure. The posts are the cheap part here, because a post is written once and everything else is written once per reader.
Writes. Here is the number the whole design turns on. Copying every post into a list for each follower at post time, which is fan-out on write, costs five million posts times four hundred followers: two billion rows a day, 23,148 a second averaged and 69,444 at the peak.
Hold that against the structure's own reads. Seventy five million feed opens read those lists; two billion rows a day were written to serve them. Almost twenty seven rows written for every open served, in the one piece of storage the design exists to build. Counted in requests the service is fifteen to one read-heavy, and counted in rows written against the opens that read them, the structure you built because it was read-heavy costs twenty seven writes for every read it answers. Both numbers are true. The multiplier between them is the design.
Reads. A feed open is one range read of the reader's own list, then a fetch of each post it points at, which everybody calls hydrating the list: thirty of those, call it thirty kilobytes on the wire. Each one put in front of somebody is an impression. Seventy five million opens times thirty is 2.25 billion post reads a day, 26,042 a second averaged, which looks alarming beside the 868 and is the easiest number in the lesson. A post never changes, and 2.25 billion reads land on five million distinct posts written that day, so a cache missing once per post misses 0.22 percent of the time. A hit rate of 99.8 percent, which is the best in this course, against the 94 percent lesson 008 measured on Marlow's book pages, and it needs no invalidation at all because there is nothing to invalidate. Three days of posts is fifteen gigabytes and sits in memory on one box.
The tail is the part of the read path worth an argument. Lesson 064 published the shape: at one slow reply in a hundred, a coordinator that sends three requests and takes the first two turns one in a hundred into three in ten thousand. Run it the other way for a page. Thirty independent reads at one percent slow each means the page waits on a slow replica 26 percent of the time, because one minus 0.99 to the thirtieth is 0.26. A page that reads thirty rows turns a one percent row tail into a one in four page tail. With 064's hedge under each read it is 0.9 percent, a factor of twenty nine, and the hedge was free because the bytes were going to those nodes anyway. Everything else about latency is the bound lesson 065 already hit: a few milliseconds of server against fifty to two hundred of mobile network.
Money. Scoring is five milliseconds of processor per open, which the next section derives, and the rest of serving one is about the same, so ten milliseconds. At the 2,604 peak that is 26 core-seconds a second, under two of lesson 005's sixteen core boxes, and you run four for lesson 055's headroom: two thousand dollars. The readers' lists live in memory and cost fourteen nodes, seven thousand. The post log is three of 062's machines after a year, fifteen hundred. Call it eleven thousand a month at the end of the first year, plus lesson 056's afternoon.
The line that grows on its own is the post log, at five hundred dollars every four or five months. The frightening line is not in the bill at all, and it is the one the second decision refuses to buy.
What a copy of a post is allowed to hold
The first irreversible decision, and the textbook asks it in a form that cannot be answered.
Does the list you precompute for a reader hold the posts, or references to them?
That sounds like a question about bytes. What it actually asks is which of a post's properties can be allowed to exist in four hundred places.
Start with what the ranking is made of. A score for one post in front of one reader is some function of who wrote it, how long ago, whether this reader usually stops on this author, and how the post is doing right now: the reactions and replies it has collected since it was written. That last group is the one worth money, because it is the only input that knows anything the author did not know when they pressed send. Lesson 086 designs the model and I am not designing it here.
Now price keeping it in the copies. The envelope's read line put 2.25 billion posts a day in front of somebody. Say one of those in twenty draws a reaction, which is mine: 112 million reactions a day, about twenty two per post. If a post's score lives in each follower's list, every one of those reactions has to update four hundred rows.
A hundred and twelve million times four hundred is forty five billion row updates a day. 518,519 a second. Twenty two times the fan-out that worried you, and the fan-out was the biggest line in the envelope.
So the copy cannot hold the score. It can hold a reference: the post id, a timestamp, four bytes of flags, sixteen bytes in all. The score is computed when somebody asks, over references the reader already has in hand, from counters held once on the post's own row in lesson 064's store, which is the one place a reaction writes to.
Which settles the famous argument by splitting it. You can precompute who is eligible and you cannot precompute the order, because the order's freshest input arrives after the list was written. Fan-out on write against fan-out on read, which stores the post once and makes each reader ask their four hundred accounts what is new, is a question about candidate retrieval and nothing else. Ranking is a separate mechanism that runs on every read whichever way you retrieved, and the two get debated as one thing, which is why the debate never ends.
Scoring at read time is cheap, and this is the number that makes the split affordable rather than merely correct. Five hundred candidates at ten microseconds each of feature lookup and arithmetic, which is mine, is five milliseconds of processor for one feed open. At the 2,604 peak that is 13 cores, under one of 005's boxes. The reason it is cheap is that the candidates arrive as references with their counters in one shared place, so the expensive part of ranking, going and finding out how each post is doing, is one batched read rather than five hundred.
Then lesson 046, which this design has to argue with, politely, because 046 is right and its condition is doing the work. 046 says that if a buffer holds derived data, something you can recompute from a source you still own, losing it costs time and nothing else, and its own example is a ranking that is a group by over the orders table. Marlow's gift page is that ranking exactly. The founder lost it and could not recompute it, and the reason is sitting in 046's own clause: a source you still own. The order lines he still owned. The stock column had been overwritten every morning since. Derived data is recomputable only when every input is a record, and a ranking's best inputs are readings.
Galewatch is the telemetry company in this course that takes a reading every two seconds from each turbine on the wind farms it monitors, and its nightly batch job is the control case. Lesson 051 has it running at 02:00, scanning the previous day off the sharded readings tables and writing one rollup row per turbine per day. Run that job again a year later and the row can move, because 051 tells you what happened to one of those rows: three million readings came home on a Tuesday, after the Sunday's row had already been written. What it cannot do is move somewhere nobody can check. Every reading it scans was written once and nothing ever overwrites one, so the second answer is better than the first and the distance between them is countable. That is the property 046's clause is asking for, and it is why 046's relaxed answer is right about a rollup and wrong about a ranking. The stock column has no second answer and nothing to count.
Start with the size of a list, because it is bought for memory and not for work. Hold seven days: eighty items a day is 560 references, 9 kilobytes a reader, 224 gigabytes for twenty five million of them and 672 with three copies. At the fifty gigabytes a node that lesson 062 called available for data in memory on 005's box, fourteen nodes, seven thousand dollars. The work those nodes do is nothing: 69,444 writes a second at the peak and 2,604 reads, about 15,400 node operations a second each, against the hundred thousand a node 064 priced at half of one of sixteen cores. You are paying for the memory, and the seven days is the only dial on it.
Seven days also means a reader back after nine gets no list at all, which is fine: they get a scan of their four hundred accounts and they have waited nine days. Fan-out on read is not a rejected alternative here. It is the cold path, and every service that fans out on write has one whether its diagram admits it or not.
Now the account with ten million followers. One post becomes ten million references, 480 megabytes with three copies for a one kilobyte post, a factor of 480,000, and it occupies 432 seconds of the entire service's average fan-out rate. Seven minutes of everybody's writes, for one line of text.
Lesson 047 named that shape and its arithmetic runs the other way. A hot key is one key whose read rate alone overloads the node holding it; 047's instrument is rate times value size held against a network card, and its hardest finding is that no partitioning scheme divides one key, because the key is the input to the division. Its worked case was the seat map of Stagefront, the ticketing service in this course where a stadium goes on sale at 10:00 and two hundred thousand people press one button inside a minute: 3,333 requests a second for a one megabyte value is 26.7 gigabits offered to a ten gigabit card.
A post is a kilobyte, so the same rate is a thousandth of that, and the rest of 047's problem does not arrive either. A popular author's recent posts are the same bytes for all ten million of those readers and they never change. 047's replication fix is dangerous because copies disagree, and its own check question walks a reader through a cart total going up and then back down; an immutable value has nothing to disagree about. So this is the one hot key in the course that a cache fixes completely, at lesson 022's nine edges if you like, and the inversion is the finding: fan-out on write is worth most for the accounts nobody follows and worth least for the accounts everybody follows, which is the exact opposite of how its cost falls.
That gives the threshold a derivation instead of a feeling. One post should not cost more than one second of the fleet's whole fan-out budget, and one second is 23,148 rows, so above about twenty thousand followers an account is not fanned out at all. Its posts are fetched at read time, once, shared by everybody.
We fan out on write up to about twenty thousand followers and
not above it, and we never copy a post's score.
Because a reference is sixteen bytes and a score is a reading:
112 million reactions a day times four hundred copies is
518,519 row updates a second, twenty two times the fan-out.
And because above twenty thousand followers one post costs
more than one second of the fleet's entire write rate, while
being the one post every reader was going to see anyway.
Costs a read-time fetch per large account a reader follows,
which is shared, immutable and cacheable at the edge.
Reverses the day the ranking's inputs are all known at write
time, measured as the share of a score that comes from
signals younger than the list.
The list nobody kept
The second decision, and the hook is what it costs to get wrong.
Is the order you served a record, or was it never written down at all?
Notice that the first decision has already answered half of it, badly. The order is computed in flight, over references, from counters that have since moved. It exists in the response for as long as the response takes to serialise. Then it is gone, and it cannot be recovered by rerunning anything, for the reason Marlow found at eleven o'clock on a Monday morning.
Which leaves two things you cannot do, and they have very different prices.
The first is dedup, and the reader polices it. To never show the same post twice you need the set of what you have shown, per reader, and 023 already said the promise belongs to a person rather than a session, so it cannot be a flag on a phone. The cheapest possible form is a post id, eight bytes. The envelope's 2.25 billion servings a day over five million posts means each post is put in front of somebody four hundred and fifty times, so remembering who saw a one kilobyte post costs 3,600 bytes of ids. The cheapest record of what you showed is three and a half times the size of what you showed.
Carried out for a year: eighteen gigabytes of ids a day, fifty four with three copies, 19.7 terabytes after twelve months against the posts' 5.5. In memory, where the serving path needs it, at fifty gigabytes a node, that is 394 nodes and a hundred and ninety seven thousand dollars a month, nearly eighteen times the entire service, growing by about a node a day for ever.
Nobody builds that. What you build is a flag in the reference you already have, inside the seven day list, which costs nothing, because the four bytes were in the sixteen. The list of what a reader might see and the record of what they were shown are one structure.
Read that twice, because it is the trap. The bound you chose for cost, seven days, is now the bound on the only promise your readers check. A reader who comes back after nine days gets a scan with no memory behind it and may be shown something they already read, which is the one failure they can detect, arriving exactly in the case you optimised away. The bound you chose for cost is the whole content of your only promise. I do not know a way out of that which is honestly cheap. You can push the window to thirty days for more than four times the nodes, or keep a smaller second structure holding only the ids a reader actually stopped on, which is a guess and not a measurement, and both of those are the same trade with the dial moved.
The second thing you cannot do is answer the hook. Why did this reader see this post, in this position, on this morning?
Marlow's version cost an afternoon and an email the founder did not want to send. At this scale it arrives from people who can make it expensive: a model ships on a Thursday, engagement falls four percent, and somebody has to say whether the model is worse or Thursday was different. With the list computed in flight and thrown away, nobody can say. You can replay the model, and you will get what Marlow got, a plausible answer built from counters that have moved.
So write the list down. One row an impression: reader, post, score, rank, model version, 32 bytes, 2.25 billion a day, 72 gigabytes a day. Then notice where it goes. It is append-only, nothing on the serving path reads it, and lesson 016 priced the shelf at two and a half cents a gigabyte a month, so a full year of it is 26 terabytes and six hundred and fifty seven dollars a month at the end of the first year. Six percent of the bill.
Put those two numbers next to each other, because they are the same facts about the same 2.25 billion servings. The record of what you showed costs a hundred and ninety seven thousand dollars a month in memory and six hundred and fifty seven on a disk nothing reads. Three hundred times, for the same servings. What you are paying for is not the keeping. It is whether the serving path has to read it, and a gigabyte of memory on these boxes costs four hundred times a gigabyte of object storage, which is 005's five hundred dollars over 062's fifty gigabytes against 016's two and a half cents.
Marlow's row was in a table the page read on every request, so it was sized to be cheap, so it held one row. The question nobody could answer and the structure that could not answer it were the same design decision, taken for a good reason, years before anybody asked.
We log every impression we serve, and we keep the readers'
memory of them for seven days.
Because the order we served is a record whether we keep it or
not: it cannot be recomputed, since its inputs are readings,
and it is what every question about this service is about.
Costs 72 gigabytes a day on object storage, $657 a month after
a year, six percent of the bill, and nothing on the read
path, which is the whole reason it is affordable.
We do NOT keep the dedup set past seven days, which would be
394 memory nodes and $197,000 a month after a year, nearly
eighteen times the service, growing a node a day.
Reverses the day a returning reader's repeat rate is worth
more than those nodes, measured as repeats served per
thousand opens by days since last open.
One thing this design does not get, and lesson 065 did. 065 gave each conversation one home region, because a conversation is an entity and lesson 050's rule is one home per entity. A feed is not an entity. It is an intersection of four hundred of them, so there is nothing to give a home to, and a feed stays a single region service for longer than its traffic suggests.
The author's own post is the exception worth naming, because it is the one piece of completeness anybody checks. A reader who posts and does not see it files a bug in ten seconds. So of the four hundred copies, exactly one is on the serving path, the author's own, written before the response returns. Lesson 012's cheapest fix covers the device that typed it, by handing back what the write already knew, and covers nothing else the author owns.
Recap
A reader cannot detect a missing post and detects a repeat instantly. That inverts lesson 065. Chat owes you a dense sequence because absence has no shape in a sparse order, and a feed owes you nothing about completeness and has to buy dedup, which is the only property its readers police. Everything a feed must store durably follows from that one asymmetry.
Only what cannot change may be copied. A post's bytes may exist in four hundred lists, at nine edges and in every cache you like, because they never change. A post's score may exist in exactly one place, because 112 million reactions a day times four hundred copies is 518,519 row updates a second, twenty two times the fan-out. So the precomputed list holds references, which means it is a candidate set rather than a feed, and you can precompute who is eligible and never the order.
Derived data is recomputable only when every input is a record. Lesson 046's relaxed write-behind over derived data has that condition in its own words, a source you still own, and a ranking's most valuable inputs are readings rather than records. Marlow had the order lines and not the stock, so the query ran and the answer was fiction.
The same facts cost three hundred times more in the structure that serves reads. An impression log is 72 gigabytes a day and $657 a month on object storage. The same rows in memory, where a dedup check needs them, are 394 nodes and $197,000. Memory on these boxes is four hundred times object storage per gigabyte, so the question to ask of anything you are told is too expensive to keep is whether the serving path has to read it.
The bound you chose for cost is the whole content of your only promise. Seven days of references is fourteen nodes and it is also exactly how long your dedup promise lasts, so the one failure a reader can detect is reserved for the readers you see least often.
Check your understanding
A team proposes storing each post's score in the reader's reference row, so that a feed open becomes one range read with no scoring pass. Price their proposal against today's figures, say which of the score's inputs makes it break, and name the one change to the ranking that would make them right.
A product manager asks for never show me the same post twice, ever. Give them the arithmetic in nodes and in dollars, then offer the cheapest substitute you would actually build and say precisely what a reader can still detect after you ship it.
An account crosses twenty thousand followers on a Tuesday afternoon. Say what changes in the system that day, what the author notices, what their followers notice, and which one graph you would watch for a week.
A ranking change ships on a Thursday and engagement falls four percent. Say what you need in order to tell a worse model from a different Thursday, whether today's design has it, and what you would have to add if the impression log had been refused on privacy grounds.
Somebody proposes putting the whole feed response behind lesson 022's CDN with a sixty second max-age, on the grounds that a reader who opens twice in a minute sees the same page anyway. Say what breaks, then name the one thing in this design that genuinely does belong at an edge and why it is allowed there.
The hydration tail is one feed open in four. A colleague proposes fetching fifteen posts instead of thirty and showing a shorter first page. Price their fix against lesson 064's hedge, say which you would ship, and say what each one does to the second page.
Your largest customer is an employer with eighty thousand staff, all of whom follow one internal announcements account that posts twice a day. Work out which parts of today's design serve them correctly, which of the two decisions you would revisit, and what you would measure before touching either.
Next lesson
067 Design a Notification System. Today's feed waits to be asked, which is where every bound above came from: a seven day list, a dedup window, a page of thirty. A notification is sent whether anybody opened the app or not, so it has no such bound and no second chance.