Marlow Books is an online bookshop run by four people, and it exists only in this course. At 08:51 on Tuesday 27 October 2026 its founder typed one line into the group the four of them share in a chat app: hold the newsletter, the boxed set price is wrong.
The newsletter went out at 09:04.
The colleague who sends it never saw that message. Their thread had the founder's 08:37 note about a delivery, then the founder's 09:11 "did you not see what I wrote", and between the two there was nothing. No gap, no spinner, no line of text that had failed to load. The thread was complete and it was wrong.
On the founder's phone that message carried the single tick the app shows when the service has taken something, which is all that app ever tells you.
The Christmas gift guide had gone up on Wednesday 21 October, and the newsletter was built from it, and somewhere in that copying the boxed set came out at 1,450 rupees instead of 2,090. Forty one people bought it at the wrong price before anybody noticed, and the shop honoured all forty one, because a bookshop that argues with a customer about its own typo has made a different and more expensive mistake. Forty one times 640 rupees is 26,240 rupees. A bad Tuesday, not a disaster.
The hour after it was worse than the money. The founder and his colleague had the argument every team has had: I did send it, I never got it, look at my screen. Both of them were right.
That chat app sorts a conversation by the time the sending phone stamped on each message, which is a perfectly reasonable order with exactly one hole in it. A sorted list cannot show you the thing that is not in it. 08:37 sits before 09:11 whether or not anything was ever meant to be between them, and nothing on either phone was in a position to notice. What found it was the founder holding his phone up next to his colleague's and counting.
You are designing the service they were using.
The brief, and the message that carries nothing
Group messaging for teams, one region. Two million people have the app open at the evening peak, fifty million messages a day, an average conversation of five people, and forty conversations each. Text and small attachments, with history kept and readable from any device the customer signs in on. Those figures are mine and they set everything below.
Fifty million a day is 579 messages a second averaged flat, and three times that at the evening peak, call it 1,740. By this course's standards that is nothing. Lesson 010 had Marlow's single Postgres doing a second of database work a second at its Christmas peak of four hundred requests, and lesson 064's store absorbed four hundred thousand operations a second on twelve machines. Seventeen hundred and forty messages a second would not wake anybody up.
Now count what the service actually sends, which is a different question.
Chat needs the server to be able to push, so the client holds a connection open. A WebSocket is an HTTP request that, rather than returning a page, upgrades the connection into a two-way pipe both ends keep open and can write into whenever they like. Lesson 006 has the sentence that decides the shape of this whole tier: a long lived connection cannot move once it is open, which is less stickiness than physics.
An open connection that nothing is happening on still has to be checked. If a phone's network dies without closing anything, the socket on the server stays open and writable while the operating system retransmits into the dark, which on the usual Linux defaults runs well past ten minutes, so the server sends a ping and expects a pong back. Pick thirty seconds for the interval, which is what a presence indicator, the green dot beside somebody's name, wants to feel honest.
Two million connections divided by thirty seconds is 66,667 pings a second.
Hold that against the product. Against 1,740 messages a second it is thirty eight times, and against the 6,944 deliveries a second those messages become once each one reaches four other people, it is still nearly ten times. The busiest message in a chat system carries nothing.
Per box it comes out somewhere familiar. An idle socket costs a few kilobytes of kernel buffers, so a hundred thousand of them is about a gigabyte of lesson 005's sixty four, and ten gigabytes if I am wrong by a factor of ten. A hundred thousand a box is the figure I would start from: not because memory binds, which it plainly does not, but because of what happens when a box's worth of them comes back at once, which the operations line prices below. Two million connections is twenty boxes, call them gateways, and twenty gateways is 3,333 pings a second each.
That is the same arrival rate lesson 047 held against Stagefront's seat map, Stagefront being this course's ticketing service where a stadium show goes on sale at exactly 10:00 and two hundred thousand people press the same button inside the same minute. The difference is that 047's 3,333 requests each carried a megabyte. These carry nothing at all, and at a generous twenty microseconds of socket work each, which is mine, they cost under half a percent of one sixteen core box.
The bytes are the part worth pricing. A ping and its pong are two of the smallest packets a network will carry; call the pair a hundred and twenty eight bytes once the headers and an acknowledgement are counted, which is a floor and not a measurement. 66,667 of those is 8.5 megabytes a second. The messages, at five hundred bytes a delivery including the envelope around the text, are 3.5. The heartbeat moves two and a half times the bytes the product does.
So raise the interval. At a hundred and twenty seconds it is 16,667 pings a second and 2.1 megabytes, comfortably under the messages. What you pay for that is detection time: two missed pings at thirty seconds tells you a device is gone in sixty to ninety seconds, and at a hundred and twenty it takes four to six minutes.
Which raises a question the interval hides: who actually wanted fast detection?
Delivery does not need it. Every delivery is its own probe: write a message into a socket, wait a few seconds for the client to acknowledge it, and a silent client has told you everything the heartbeat would have, at the only moment it matters. The fallback path does not need it either, for the same reason, because the decision to send a push notification instead hangs off that un-acknowledged message rather than off a timer.
What wants thirty seconds is the green dot. Presence sets the heartbeat interval, and the heartbeat interval sets the busiest rate in the system. Lesson 061 said a design review argues about the biggest number and the scariest word, and that the decisions are in neither. Presence arrives at that review as decoration.
Four lines on an envelope
Lesson 025's order. Bytes first, money last.
Bytes. At 579 messages a second averaged and five hundred bytes each, the conversation log grows twenty five gigabytes a day, seventy five with three copies, twenty seven terabytes in a year. On the two terabyte disks lesson 062 used that is fourteen machines after twelve months, and it keeps going up by a machine roughly every month. Cut history to ninety days and the same traffic needs 6.8 terabytes and under four machines. Retention is the only dial on the only line of this bill that grows on its own.
Operations. Reads come in two shapes, and the second one is the problem. A client that has been connected all evening reads nothing; its messages are pushed to it. A client coming back after a gap has to find out what it missed, and lesson 006's sentence decides how many of those arrive together, because a connection cannot move, so a gateway box going away takes its hundred thousand connections with it.
Drain one box and a hundred thousand clients reconnect. If they all come back inside a second, spread over the other nineteen, that is 5,263 TLS handshakes a second a box, and at a millisecond of processor time a handshake, which is mine, a third of one box's sixteen cores for about a second. Survivable.
A fleet-wide event is not. Lesson 020 named the mechanism: a herd comes from a shared cause, not a shared schedule. Two million handshakes inside one second is two thousand core-seconds of cryptography offered to the three hundred and twenty cores the fleet has, six times what exists, so for six seconds the gateways do nothing but handshake, and the clients at the back of that queue time out and start again. Spread the same two million over a minute and it is 33,333 a second, a tenth of the fleet, and 020 already priced the fix in its own words: jitter is what turns four thousand requests in one second into sixty seven a second for a minute.
The uncomfortable part is where that jitter lives. It is in a client you shipped eighteen months ago. Lesson 063 found that the error in a copy-local limiter is chosen by the caller; the same shape arrives here as the worst case of your deploy is chosen by the oldest app version in the field, and no amount of care in the gateway tier moves it.
Latency. The path a message takes, with the server's share of it marked:
phone -> gateway -> assign position -> store
|
recipients' gateways -> their phones
server side: four round trips inside one data centre, 0.5 ms each
phone side: 50 to 200 ms, each way, on a mobile network
Spoken, because the audio skips blocks: the message crosses the gateway, gets its position, is stored, and is pushed out through the recipients' gateways, which is four round trips inside one data centre at lesson 002's half millisecond, so about two milliseconds plus whatever the store itself takes, against fifty to two hundred milliseconds of mobile network on each end. The server is not where chat latency lives. The one thing the server can genuinely ruin is the acknowledgement, which is the tick the sender is staring at.
Money. Twenty gateway boxes at lesson 005's five hundred dollars is ten thousand a month. Lesson 064's store is six thousand and already bought. History is seven thousand after a year and climbs by about five hundred and sixty dollars a month after that. Twenty three thousand a month, plus lesson 056's afternoon, and this one has a name on it: somebody owns the retention number.
Where a message's position comes from
The first irreversible decision, and the hook is the invoice for getting it wrong.
Does a message's position come from a clock or from a counter?
The clock first, because it is what nearly everybody ships. Each message carries a timestamp and the client sorts on it. Lesson 034 spent a whole lesson on what such a stamp is worth, and measured Galewatch's turbines to find out: Galewatch is the telemetry company in this course that takes a reading every two seconds from each turbine on the wind farms it monitors, and across that fleet the clock offset had a median of four tenths of a second, two point one at the ninety fifth percentile, and a worst case of forty one seconds on a turbine that had been reporting cheerfully for two years.
A phone is usually better behaved, because something sets its clock off the network most days, and a server can refuse to trust it either way: stamp on arrival, or use 034's hybrid logical clock, a pair made of a physical part and a small logical counter underneath it that sorts correctly, never goes backwards and stays close enough to real time that a human recognises the afternoon. So the clock's accuracy is a solved problem, and solving it is why this branch feels finished.
It is not finished, and the reason has nothing to do with accuracy.
A timestamp order is sparse. Between any two stamps there is room for anything at all, so a client holding 08:37 and 09:11 cannot tell whether something was meant to be between them, and neither can the server, because the server does not know what the client holds. Absence has no shape in a sparse order. Marlow's Tuesday is that sentence with a price attached.
Now the counter. Give every conversation a dense sequence: one, two, three, no gaps, nothing skipped, assigned by the service and not by the sender. A client holding one to forty that receives forty two knows it is missing forty one, by subtraction, and can ask for it by name. For that to mean anything, a number once handed out can never go unused, so a deleted message keeps its place and answers with a marker saying it was deleted, which is the tombstone lesson 062 invented arriving for a third reason: 062 kept one so a dead key could never be reissued, 064 so its own repair machinery could not run backwards, and here it stops a deletion looking exactly like a loss. Nothing else in the design has to work for the subtraction to hold. The client needs no knowledge of who else is in the room, no agreement with the server about what it has, and no opinion about whether its network was up. It subtracts.
A dense order makes absence visible, and it is the only property here a recipient cannot get some other way.
What it costs is that somebody has to hand out the next number, which is lesson 064's first decision arriving per conversation rather than per key. 064 took the single-owner branch and priced it: every key gets one elected owner, by lesson 033's rule, and you pay the writes to one range while an election runs.
Here it comes much cheaper, and the arithmetic is the reason. Use 064's version check, which 064 priced at one integer: a conditional write on the conversation row, offered against the sequence number you read, and the loser of a race retries against the newer one. No election, no leader, no lease to keep alive. What it costs is that retry, and the retry rate is a multiplication you can do in your head, because a write collides with another write that started inside one round trip of it. Lesson 002 puts a round trip inside one data centre at half a millisecond, so the collision rate is the room's message rate times half a millisecond, treating arrivals as independent, which is mine and is the optimistic reading, because people in a room reply to each other in bunches. A room doing twenty messages a second retries one write in a hundred. Two hundred retries one in ten. At two thousand the model stops being a probability and turns into a ceiling: two writes in three collide, and one row cannot settle more than one write per round trip anyway, which at half a millisecond is two thousand a second and not one more. The retry loop has become a queue whose throughput is no longer rising.
So the question is whether a real room can push past that, and lesson 047's hot key in this design is the group with fifty thousand members. Two thousand messages a second in one room means every one of those fifty thousand people receives two thousand messages a second. Nobody reads that. The ceiling on the cheap implementation is a rate at which the room has stopped being a conversation, which is the honest reason not to build the expensive one.
The expensive one is there when a customer proves otherwise. Elect an owner for the conversation and let it hand the numbers out of memory, in the order the requests reach its queue. There are no collisions at all now, because nobody is guessing, and the ceiling becomes the owner's own throughput. Lesson 063 published that figure for exactly this shape, a single thread doing one small operation a request: five microseconds a dispatch, about two hundred thousand a second. A hundred times the conditional write, and what it costs is an election to run, a lease to renew and lesson 035's whole list of sharp edges. Stay on the cheap one until somebody shows you a room that sustains hundreds of messages a second, which you will see as conditional writes retried per conversation per minute.
We give every conversation a dense sequence, assigned by a
conditional write on the conversation row.
Because a dense order is the only one in which a client can
detect a message it never received, by subtraction, with no
help from the server and no knowledge of the room.
Costs a retry when two messages arrive inside one round trip:
one write in a hundred at twenty messages a second, and a
queue rather than a retry somewhere above two thousand.
Does NOT need an elected owner per conversation, which is the
same promise with a hundred times the ceiling and lesson
035's sharp edges attached.
Reverses the day a real room sustains hundreds of messages a
second, measured as conditional writes retried per
conversation per minute.
One thing this decision does not touch, and lesson 064's second decision is why it is worth saying out loud. 064 asked whether the store can read the value, because a store cannot merge what it cannot read. A position is not a merge. The sequence number lives outside the message, so a service that cannot read a single word of what it is ordering orders it perfectly well, and can still tell a client it is missing number forty one. Order and readability are independent in a way that order and merging are not.
Who remembers what you have seen
The second decision, and the textbook asks it in a form that hides what it is.
Does the server hold a record of what each recipient has received, or does the client hold a record of where it has got to?
Fanning out on write means copying every message into a per-recipient inbox at send time, so a client's reconnect is one read. Fanning out on read means storing the message once in its conversation and letting each client ask its conversations what is new. The usual argument between them is about storage, and storage turns out to be the least interesting half.
At this brief's average of five people a conversation, fanning out on write multiplies the durable log by five, because the sender needs a copy to read on their other devices too: 375 gigabytes a day with copies against seventy five, so the envelope's fourteen machines after a year become sixty eight. That is real money and it is not an argument.
The argument is the group of fifty thousand. One message of two hundred bytes becomes fifty thousand inbox rows, and at the five hundred bytes a row this envelope has been using, that is twenty five megabytes of durable, ordered, replicated writing for one line of text. A factor of a hundred and twenty five thousand. Which is why every service that fans out on write has a group size above which it quietly stops doing so, and why that threshold is a product decision wearing a storage costume.
Underneath the money sits something worse. A per-recipient inbox is an ordered structure with as many writers as that recipient has conversations, and lesson 037 settled what an ordered log can promise: records sharing a key stay in the order they were appended within one partition, and across partitions there is no order at all, for the reason lesson 034 gave, which is that nothing talks so nothing is ordered. A conversation is the obvious partition. A person in forty conversations is not a partition at all. So an inbox that promises its owner an order is promising something its storage cannot give, and it will fall back to ordering by the clock, which is the hook.
The way out is to notice that a reader wants two different things and only one of them is a promise.
Inside one conversation a reader needs a sequence: dense, authoritative, gap-detectable, and it must never silently lose a member. The first decision bought that.
Across conversations a reader needs a ranking: which of my forty rooms moved, roughly in what order, so that the opening screen is useful. A ranking may reorder, may be a second stale, and may be lost outright, because it is recoverable by asking the forty rooms. A sequence is a promise and a ranking is a convenience, and a service that builds one mechanism for both pays sequence prices for convenience data.
The ranking is almost free. One small value per customer, holding a conversation id and its latest position, forty entries at sixteen bytes: 1.28 gigabytes for two million customers, and thirteen if the account base is ten times the peak, which fits in memory on lesson 064's store beside everything it was already holding, and the write on each delivery is sixteen bytes rather than five hundred. Against the inbox's twenty five megabytes, one message to the big group costs eight hundred kilobytes, thirty times less, and the saving that matters is not the bytes but that those bytes may be lost.
Then the number that makes the ranking compulsory rather than tidy. A client coming back cold without one has to ask each of its forty conversations where things stand. One box draining is a hundred thousand clients over a jittered minute, 1,667 clients a second, 66,667 reads a second, a sixth of the four hundred thousand operations 064's store was sized for. Fine. A fleet-wide event is two million clients over that same minute: 33,333 a second times forty conversations is 1.33 million reads a second, three and a third times that entire brief, arriving while the store is also carrying the messages. With the ranking it is one read each, 33,333 a second, a twelfth of it.
Be careful with that comparison, because four hundred thousand is a load 064 chose rather than a ceiling it hit: 064 priced those operations at half of one of sixteen cores a node, so one and a third million is nearer one and two thirds of them, and the store probably absorbs the bad version without complaining. Which is the argument for the ranking rather than against it. Nothing refuses that burst and nothing pages anybody, and it arrives multiplied by a number nobody on your side chose: forty conversations a customer is three and a third times the store's whole brief, eighty is six and two thirds, and your own design did not move. The factor between those two is the conversation count, and the conversation count is a number your customers choose. Headroom is no defence against a multiplier somebody else sets, and a gigabyte of state you are allowed to lose removes it.
The green dot, and the receipt I would refuse
Presence has the same shape as the ranking and takes the same fix. Two million people flipping between present and away twice an hour is 1,111 changes a second, and telling each of a customer's two hundred contacts about every change is 222,222 notifications a second, a hundred and twenty eight times the message rate the service exists for. Nobody pays that. So presence goes out as a subscription rather than a broadcast: a client hears about the people whose conversation is open on its screen right now, call it two of the two hundred, and 1,111 changes a second become 2,222 notifications. A factor of a hundred, for a restriction no customer can detect, because somebody whose name you cannot see cannot show you a dot either.
Which leaves the feature I would refuse, and the hook is going to argue with me about it.
A read receipt is itself a message. In a room of five, one message produces four receipts, and even shown to the sender alone that exactly doubles the deliveries the service makes, which is affordable. The doubling never gets worse as the room grows, which is the trap in it. What gets worse is where the receipts land: all of them on one phone, and a list of two hundred names is already past the point where the sender reads it. In a room of fifty thousand, one line of text produces fifty thousand receipts, and if every member is shown every receipt it is fifty thousand squared, 2.5 billion deliveries for two hundred bytes of typing. That curve is why the feature quietly vanishes in big rooms everywhere you have ever seen it, and that threshold is arithmetic rather than taste.
Marlow's Tuesday is the argument against me, because a per-recipient receipt in a group of four is exactly what would have shown the founder that one of his three colleagues never got the message. It would have, if he had looked. And that is the difference worth keeping: a receipt is a question the sender has to remember to ask, and a sequence is an answer the recipient's own client gets whether anybody is looking or not. At 09:11 the colleague's client would have seen a number two higher than the one it held and fetched the missing message before either of them typed a word. Ship both in small rooms. Only one of them survives a large one.
We do not show who has read a message in groups above about
two hundred members.
Because a read receipt is itself a message: in a group of n,
one message produces n receipts, and showing them to
everybody makes it n squared. Fifty thousand members is
fifty thousand receipts for one line of text, and 2.5
billion deliveries if the whole room sees them.
Costs the feature in exactly the rooms that ask for it most,
which are the announcement channels.
Reverses the day somebody wants it for a specific room,
measured as receipts a second against that room's message
rate, with the sender-only form costed first.
One home per conversation
Multi-region, which lesson 064 refused outright, this design can actually afford, and the reason is the tick. A dense sequence has one assigner, lesson 002 puts Mumbai to Virginia at two hundred milliseconds, four hundred times the local round trip, and lesson 050 already published the rule: give every entity one home and route writes by entity. The conversation is the entity. A sender on the wrong side of the ocean waits two hundred milliseconds for their sequence number, on top of the fifty to two hundred their own mobile network was charging anyway, and receives no slower than they did when everything was in one region, because the fan-out leaves the conversation's home whichever region the sender was in. What you buy with that is the thing to say out loud, in lesson 024's terms: a region that loses its conversations loses them completely for the duration, and that is one of two answers with no third.
Recap
Absence has no shape in a sparse order. A timestamp, however accurate, leaves room between any two stamps for anything at all, so neither the client nor the server can tell a quiet conversation from a conversation with a hole in it. A dense sequence per conversation makes the hole arithmetic: hold one to forty, receive forty two, ask for forty one. Lesson 034's clocks are fixable and fixing them does not touch this, because the problem was never accuracy.
A dense sequence has to be assigned, and the cheap assignment is good enough for every room a human is in. Lesson 064's version check, one integer, as a conditional write on the conversation row. The retry rate is the room's message rate times one round trip, so a write in a hundred at twenty messages a second, and two thousand a second, where it stops working, is a rate at which nobody is reading. An elected owner handing numbers out of memory has no collisions and a hundred times the ceiling, with lesson 035's sharp edges attached.
The busiest message in a chat system carries nothing, and presence is what sets its rate. Two million connections pinged every thirty seconds is 66,667 heartbeats a second against 1,740 messages, thirty eight times, and two and a half times their bytes. Delivery never needed that interval, because every delivery is its own probe and a client that fails to acknowledge one has told you more than a timer would. The green dot needed it, and the green dot reaches a design review as decoration.
A sequence is a promise and a ranking is a convenience. Inside a conversation the order must be dense, authoritative and never quietly short a member. Across a customer's forty conversations, which room moved last is allowed to be stale, allowed to reorder and allowed to be lost, because forty reads rebuild it. Building one durable ordered inbox per person to serve both is how a service ends up paying twenty five megabytes to deliver two hundred bytes, and lesson 037 is why the inbox cannot keep the order it promises anyway: across partitions there is no order at all.
Order and readability are independent in a way order and merging are not. Lesson 064's second decision was whether the store can read the value, because nothing can merge what it cannot read. A position sits outside the message, so a service that can read none of what it carries still orders it exactly and still reports the gap.
Check your understanding
A team ships a client that sorts a conversation by the server's hybrid logical clock stamp and ignores the sequence number, on the grounds that the stamps sort identically and the code is simpler. They are right about the sorting. Say what they have given up, construct the user-visible failure, and say what you would ask for in the review.
Your heartbeat is every thirty seconds with two misses allowed. A colleague proposes five seconds, so that presence feels instant. Give them the three numbers that change, say which one you would put in front of the person who asked for instant presence, and name the one thing that genuinely gets better.
A new deploy process restarts all twenty gateway boxes inside ten seconds, because it is faster and the old process took twenty minutes. Describe the first ninety seconds, say which graph moves first, and say what you would change given that the reconnect behaviour is in an app you cannot update today.
Somebody proposes dropping the per-customer ranking and having clients ask each conversation on reconnect, pointing out correctly that it removes a whole piece of state and a whole class of bug. Price their proposal for a normal evening and for a fleet restart, then say under what change to the brief they would be right.
A customer wants a channel with every one of their employees in it, eighty thousand people, used about four times a day for announcements. Walk through which parts of today's design are fine, which break, and which of the two decisions you would revisit, with the arithmetic for each.
Two conversations in different regions both have a participant who travels constantly, and that person complains their messages take a quarter of a second to show a tick while everybody else's are instant. Explain what they are measuring, say whether it is worth fixing, and name the lesson whose rule you would cite in either direction.
A reader of the conversation log notices that sequence number 2,044 is missing from one room and has been for three weeks, with 2,043 and 2,045 both present. Say what could have produced that, which of today's mechanisms should have reported it and did not, and what you would add.
Next lesson
066 Design a News Feed. Today's second object, the ranking that is allowed to be stale and allowed to be lost, is the whole product of a feed rather than a convenience beside it, and a feed has no sequence anywhere in it to fall back on.