Galewatch is a telemetry company that exists only in this course. It takes a reading every two seconds from each wind turbine on the farms it monitors and sells the owners dashboards. At 02:48 on Wednesday 23 September 2026 the wireless link to one of those farms went down in bad weather, and ten minutes later a phone on a bedside table buzzed sixty times in about four seconds.
Tarrow Ridge is the farm lesson 002 measured: sixty turbines along a ridge, on a fixed wireless link doing about 20 megabits a second. Lesson 026 asked every system for an alert that fires when something stops arriving, because lesson 018 had established that the absence of an event is not an event, and Galewatch had built exactly that. Per turbine. Ten minutes of silence and it fires.
One link. Sixty turbines. Sixty notifications.
The engineer on call took three minutes to work out that this was one link rather than sixty failures, because the sixty arrived together, and went back to bed. The link came back at 05:41, and sixty more notifications said so. A hundred and twenty interruptions in under three hours, for the kind of weather nobody at Galewatch worries about. Three hours is well inside what a turbine can hold: lesson 003 settled that the turbines buffer their readings locally and replay them when the link comes back, so availability breaks and durability holds. Lesson 058 is where that confidence runs out, at nine hours and six minutes a turbine, and that is not this story.
So at ten past six that morning the engineer opened the phone's settings and turned the app's notifications off. Not snoozed until nine. Off, in a screen Galewatch cannot see, cannot read and cannot ask about.
Thirty one days later, at 06:20 on Saturday 24 October, the same link went down again, and that is the weekend lessons 058 and 059 are about: twenty seven and a half hours of it, a damaged blade, and an insurer who wanted the two second record. Sixty notifications fired at half past six that morning. The rota is short and the same engineer was on it.
On the Wednesday, during the afternoon lesson 059 tells, somebody working back through that weekend asked a smaller question. Did anybody get told at the time? Two answers came back and both were true. The engineer had not seen a notification from Galewatch since the Wednesday morning in September. The notification service had delivered sixty of them to that engineer on the Saturday, every one accepted by the push provider, none failed, none retried.
Both of those are measurements. Neither is a measurement of whether anybody was told.
The brief, and the three things a person can detect
Twenty million people have the app installed. Eleven product teams inside the company can send them something, and all eleven believe their thing matters. The product generates two hundred million notifiable actions a day, somebody comments or replies or follows or finishes a job or changes a price, and each action has on average four people with a defensible claim to hear about it. A person opens the app four times a day. Those figures are mine, including the four, and everything below is built out of them.
Two hundred million actions a day is 2,315 a second and ten actions per person. Multiply by four claimants and you get eight hundred million candidate notifications a day, 9,259 a second averaged and 27,778 at a three times evening peak. Forty candidates per person per day.
Now the other side of the brief, which is the side with no machine in it. How many interruptions will a person accept in a day before they do what the engineer did in September? Nobody knows exactly, and everybody in the business knows the order of magnitude, which is a handful. Call it five, mine, and hold it as a hard ceiling rather than a target: twenty million people times five is a hundred million sends a day, 1,157 a second and 3,472 at the peak.
Forty candidates against five slots. Eight candidates for every notification this service is allowed to send, so its largest output by volume is refusals: seven hundred million a day, 8,102 a second against 1,157 sends. The forty is a mean over a skew, and the verdict survives the skew being severe: if only the busiest tenth of people clear five candidates a day, that is still two million people who need refusing, and they are the people who use the product most and who generate most of the volume.
Lesson 066 designed a feed on the same kind of arithmetic and got the opposite answer: three feed opens a day at thirty items demanded ninety slots while four hundred followees supplied about eighty posts, so the supply ran out by ten slots a day and the service had to repeat itself. Here the supply is eight times the demand. A feed's hard problem is finding enough; this one's is throwing away seven eighths of what it has.
What makes the throwing away hard is the recipient. 066 found that a reader cannot detect a missing post and detects a repeat instantly, which is one detectable failure, and its whole design followed from that asymmetry. Count them here.
A person detects a notification that never came, because they find out about the thing later, from somebody else, and they are angry in a way nobody is about a missing post. They detect a duplicate instantly, the same way a feed reader does. And they detect one that arrives at 03:00, which is the new one: a notification has a time of arrival that is part of its correctness, and nothing else in this course has had that.
Three detectable failures against a feed's one, and two of them pull in opposite directions. Send less and you create the first. Send more and you create the second and the third. No setting is safe in all three directions, which is why this design's correctness is defined by a person's tolerance rather than by a data structure. Lesson 026 already put that in hard form: an alert comes from a tolerance, not from a number somebody liked. 026 aimed it at the threshold. It governs the delivery too.
One division before the architecture, because it decides which tier to worry about. Count what the service stores against the times a person comes to look: a hundred million rows a day written against eighty million list reads, so 0.8 reads per write. Lesson 011 published Marlow Books' two hundred reads per write, Marlow Books being the four person online bookshop this course has been patching since lesson 001. A bookshop is two hundred and fifty times more read-heavy than a notification system, and this is the first Phase 3 design that is write-heavy in the structure it exists to keep. Count every operation instead and it inverts, because deciding reads a recipient's preferences and budget several times for each row it writes, on one small record per person that never leaves its own tier. The storage is write-heavy, the decision path is read-heavy, and confusing the two is how this system ends up on the wrong database.
Four lines on an envelope
Lesson 025's order: bytes, writes, reads, money.
Bytes. A row says who is being told, about what, by whom, when, and whether they have seen it: recipient, event id, actor, target, kind, a timestamp and a few flags is forty six bytes, call it sixty four with the row around it. At the hundred million ceiling that is 6.4 gigabytes a day and 19.2 with three copies. Keep ninety days and each copy is 576 gigabytes, 1.73 terabytes in all, which is three of the two terabyte machines lesson 062 used, each a third full, fifteen hundred dollars; a year would be 2.3 terabytes a copy and six machines. The same ninety days in memory, at the fifty gigabytes a node 062 called available on one of lesson 005's boxes, is thirty five nodes and seventeen thousand five hundred dollars a month. Nothing here needs memory.
Writes. A hundred million rows a day at the ceiling, 1,157 a second, 3,472 at the peak, and that is the whole write path. There is no fan-out multiplier on it, which is worth saying out loud, because the memorised answer for a notification system borrows a feed's diagram and a feed's two billion rows a day. A notification already names its recipient when it arrives. The work is deciding, not copying.
Reads. Eighty million opens a day, 926 a second and 2,778 at the peak. An open is one range read of the recipient's recent rows, and the unread count comes out of that same read. The rows hold ids, so rendering a page of ten hydrates an actor and a target each, twenty lookups against 066's thirty, on the same better-than-99-percent cache; 066 owns that arithmetic and I am not redoing it.
Money. Deciding costs about a millisecond of processor per candidate, mine, so 27,778 candidates a second at the peak is 27.8 core-seconds a second, under two of 005's sixteen core boxes and four for lesson 055's headroom: two thousand dollars. Serving is a third of one box; run two, a thousand. The store is fifteen hundred. The log in front of the deciders is lesson 037's and I am not re-pricing it, so call the fleet five thousand dollars a month.
Then the line that is not rent.
A push through a platform push service costs nothing per message. An email costs a hundredth of a cent. A text message costs about a cent, and those last two are my own rounded figures, low for some countries and high for others. Multiply by the ceiling. Every one of the hundred million by email is ten thousand dollars a day, three hundred thousand a month, sixty times the entire fleet. Text messages on a tenth of them is a hundred thousand dollars a day, three million a month, six hundred times the whole platform.
Lesson 056 sorted a bill into rent, a meter and somebody's afternoon, and said a loop in a template can cost as much as the fleet underneath it. This is that sentence with three more zeros on it: the architecture is almost entirely rent, and one product decision about channels attaches a meter the fleet cannot be seen next to. The only channel with no meter is the one you do not own. Push is free at the margin because the platform carrying it is selling something else, and free at the margin is exactly why all eleven teams want it and why the budget has to belong to you.
Is a notification a message, or a view of state
The first irreversible decision, and it sounds like a question about wire formats.
When you notify somebody, have you sent them a message, or changed some state they can come and read?
The answer is forced by what you do not own, so write down the hops. Your service composes something and hands it to a platform push service, a third party. The platform hands it to an operating system. The operating system decides whether and how to put it on a screen. A person looks at the screen or does not. Four hops, and underneath all four is a settings screen where that person can switch the arrangement off, which is what happened in September. You own the first, none of the rest, and the one that decides whether the notification worked is the last.
Lesson 065 left this lesson a sentence and the sentence does not survive the trip. 065 designed a chat system where delivery needs no heartbeat because every delivery is its own probe: write the message into the socket, wait a few seconds for the client to acknowledge it, and a silent client has told you everything a heartbeat would, at the only moment it matters. That works because 065 owned the socket. Here the acknowledgement comes from the platform, about the hop between you and the platform, and it says the platform accepted the bytes. The last hop reports nothing. This is the one delivery in the course with no probe on it, which is the gap Galewatch's two true answers sat in.
The silent failure underneath is worse than the missing probe. A device token is a credential the platform issues, scoped to one installation, and it goes stale when somebody reinstalls the app or restores a backup. Lesson 053's rotation arithmetic covers the keys you authenticate to the platform with, and this token is a different animal, because it names a device rather than a caller. A dead token is the kind case: send to it and the platform will usually tell you, in its own time, that nobody is there, and you prune it. The case that matters is September's. The token is live, the device is awake, the person has switched you off in a screen you cannot read, and the platform accepts the send and the device drops it. On your dashboard that is indistinguishable from somebody who saw the notification and did not care. Lesson 026 named the shape in a line: a missing measurement and a measurement of zero look identical and mean opposite things.
So the push cannot carry the guarantee, because nothing you can measure says it arrived.
Which is lesson 040's sentence, and I am reusing its shape deliberately. 040 was arguing about Postgres NOTIFY over a polling drainer: a notification is not durable, so if nothing is listening when the commit lands it is gone, and therefore a notification is an optimisation and the poll is the guarantee. Change one noun. The push is the optimisation. The list is the guarantee, and the list is rows in a store you own, keyed by the recipient, which the app reads when it opens.
A lost push stops being a lost notification. The row landed. The person sees it the next time they open the app, which is four times a day, so the worst case for the failure nobody notices until later is a few hours rather than for ever.
A duplicate push stops mattering, because two pushes pointing at one row are two taps that land on the same thing. The service still owes idempotency, and lesson 019 gives it free from the named pile: the key is the intent and not the attempt, the intent here is telling this person about this event, so a unique constraint on recipient and event id refuses the second row without any code at all. 036's at-least-once is why you will get the second copy; 019's constraint is why it costs nothing.
The unread badge stops oscillating, and this is the small problem nobody warns you about. A count on the icon is a number you put in the push payload, computed when the message was composed, which makes it a reading rather than a record in exactly 066's sense: two pushes in flight can arrive in either order, and a person who reads everything on a laptop leaves the number on the phone knowing nothing about it. Implementations that fix that by incrementing and decrementing a stored counter rediscover two things this course published two phases ago. Lesson 019: atomic is not idempotent, so a redelivered message corrupts a counter permanently while it leaves a flag on a row exactly as it found it. And lesson 047, which replicated a hot key ten ways, watched a seat map go backwards in time by up to a full expiry, and said in a clause that the same trick aimed at an account balance or an unread count gives you a value that oscillates. A badge is a count, never a counter. Derive it from the rows not marked seen, treat the payload's number as a hint, and let the store be the authority.
And quiet hours become possible at all. A notification composed at 14:00 in one place arrives at 03:00 in another, and the usual fix, hold it until morning, is a bad trade when the thing is time sensitive, because a held interruption about something that already happened is worse than none. If a notification is a message you have to choose. If it is a view of state you do not: the row lands immediately and only the interruption waits. Open the app at 03:00 because you could not sleep and it is there.
A provider outage comes free with the same split. Lesson 042 asked for graceful degradation and lesson 048 offered fail static, which cannot help here because you cannot cache a delivery. What you can do is let the rows carry on landing while the pushes queue behind the breaker, with the bound written in seconds rather than items the way 041 insists, depth over drain rate being the longest a fresh item waits behind a stale one. Past that bound you drop. Drop hints, never rows.
Retention is where this design escapes a trap 066 walked into. 066's closing finding was that the bound you chose for cost is the whole content of your only promise, because a feed's seven day list of references was also its dedup record, so a reader returning after nine days lost the one guarantee the design made. Ninety days of rows here is three machines a third full, and the dedup key is an event id. Events do not recur. Something that happened once cannot be shown to you twice ninety days apart, so the bound and the promise have come apart, and the reason is the identity of the thing being deduplicated rather than anything clever about storage. 066's trap bites whenever the dedup record is the same structure as the working set. Check which one you have.
A notification is a durable row we own, and the push is a
hint that one has appeared.
Because we own one of the four hops to a person's attention
and the last one reports nothing: a send to a device whose
owner switched us off is accepted, delivered, dropped, and
indistinguishable from somebody who did not care.
Costs ninety days of rows, 1.73 TB across three machines at
$1,500 a month, plus one read on every app open at 926 a
second.
Buys a lost push recovered in hours, a harmless duplicate, a
badge that cannot oscillate, quiet hours that withhold
nothing, and an outage that costs only the buzz.
Reverses the day a platform reports true per-device delivery,
measured as the share of sends with a confirmed outcome.
Who decides not to send
The second irreversible decision. Eight candidates, one slot, eleven teams.
Who refuses the other seven: the service, the device, or the person?
The device is what you get by default, and the September morning is what it looks like. An operating system will group, summarise and delay on your behalf by rules you cannot see, and the person has one more instrument underneath that, which is the switch. Neither refusal is visible to you, so leaving the decision there sets your real send rate by a stranger's defaults, and your most engaged users turn you off first because they get the most candidates. The person is what the product team reaches for, and it is a preferences screen with forty checkboxes on it that hardly anybody opens. Preferences say which kinds are welcome and nothing about how many, which makes them necessary and not a budget.
Which leaves the service, and it needs an instrument it does not have yet.
Each of the eleven teams has a reasonable case. The comments team's notification is useful, and so is the approvals team's and the billing team's and the one about a long job finishing. No team can see the other ten, so no team can see the total, and the total is the only quantity the recipient experiences. Nobody owns the sum. That is an organisational fact with an engineering answer: the platform keeps a per-recipient budget and spends it on the senders' behalf, refusing anything over it.
Lesson 021 built that instrument and pointed it the other way. 021's limiter protects you from a caller, refusing a request because of who asked and how often, and its recap says a limiter is the only thing in your architecture that can say no, which can fire on a perfectly healthy morning. Every limiter in this course so far has refused somebody outside the building. This limiter's subject is the recipient, not the caller, and the callers are all your colleagues. Same bucket, same burst question, same cheap refusal path, different beneficiary, and a political problem where 021 had a technical one.
Size the refusal path the way 021 says, by the traffic it refuses. Stagefront is the ticketing service in this course where a stadium show goes on sale at 10:00 and two hundred thousand people press the same button in the same minute, and 021 published its five hundred admissions a second against six thousand refusals once you count the retries, twelve to one. This service refuses seven for every one it allows. A kinder ratio and a worse situation, because Stagefront's refusals went to strangers who retried and these go to colleagues who will come and find you.
So the refusal has to be worth something, and this is where the two decisions hold each other up. A refused candidate is not dropped. It writes its row, which the first decision already paid for, and simply does not buy an interruption; the person sees it on their next open, or in one digest a day, which is lesson 046's write-behind pointed at a person instead of a database: many small items coalesced and flushed on an interval somebody chose. You can only afford to refuse seven interruptions out of eight because not one of the seven was the only copy. Take the first decision away and the budget becomes data loss. Put it back and the budget is a scheduling policy. Neither decision is defensible without the other, which is the thing I would hope somebody noticed in the review.
Something still has to decide which candidate gets the slot, and it cannot be a priority field the sender fills in, because a priority the sender chooses is always high. The decay class belongs to the platform, from a short fixed list agreed with the senders: a failed payment does not decay at all, a mention decays over hours, somebody liking a thing decays over minutes and is a digest line rather than a push. That is a product taxonomy with an engineering consequence, since 041 says drop whatever decays fastest and this is where you write down what that means. Ranking the survivors against each other is a model, lesson 086 designs it, and I am refusing it here on purpose: a budget with decay classes behind it can be defended with arithmetic and a learned importance score cannot, which is where a platform stops being able to explain itself to the teams it serves.
The digest path and the interrupt path then need separate queues, because 041 settled that trap already: you cannot prioritise upstream of a shared buffer, a queue serving in order being what made lesson 017's two lanes decoration in August. Two drain rates, the interrupt lane sized for the peak and the digest lane for the day.
Now the case that breaks the budget, and it is the one that actually arrives on a Tuesday. A product team wants to announce something to everybody: twenty million candidates in a single request.
The bytes are nothing. The time is not. At the fleet's average decision rate of 9,259 a second, twenty million candidates take 2,160 seconds, thirty six minutes of everything the platform does, and twelve minutes if you hand it the entire peak capacity, during which the comments team is queued behind an announcement. 066 found the same shape in a feed and priced one post from a ten million follower account at 432 seconds of the whole service's fan-out. The difference is what comes next.
Twenty million candidates at one slot each spend twenty million of the hundred million slots the whole company has that day. One team, one afternoon, one fifth of everybody's daily budget. Every other team's notifications for the rest of that day are competing for what is left, and the losers are refused by a rule none of them can see. A broadcast spends a budget it does not own, and no amount of queue tuning fixes it, because it is not a throughput problem. So a broadcast is a different product with its own lane, a lower ceiling and an allowance that cannot be borrowed from the per-person five. My own view is that most of them should not be pushes at all: let them land as rows and show up on the next open, where they cost nothing and nobody loses a slot.
One thing this design gets cheaply that 066 could not get at all. 066 refused multi-region because a feed is an intersection of four hundred accounts and lesson 050's one home per entity rule had nothing to attach to. A notification list is an entity, belonging to one person, written by other people's actions and read only by them, so 050 applies without an argument. The recipient is also the shard key 013 and 014 taught you to agonise over, the subject of the budget, and the unit lesson 037's per-partition ordering has to hold for. One key doing four jobs is rare enough to say out loud, and it is why this design is simpler than its reputation.
The platform refuses on the recipient's behalf, against a
per-person budget of five interruptions a day, and
broadcasts get a separate allowance.
Because nobody else can see the sum: eleven teams each with a
defensible case, forty candidates a person a day, and the
only instrument under us is a switch we cannot read.
Costs 8,102 refusals a second against 1,157 sends, an argument
with every team that measures its own open rate, and a decay
taxonomy somebody has to maintain.
Affordable only because a refused candidate still writes its
row, so a refusal is a scheduling decision and not a loss.
Reverses the day a measured budget appears: opt-out rate by
sends per person per day, which tells us five is wrong and
in which direction.
Recap
A notification has three detectable failures where a feed has one. A person notices one that never came, one that came twice, and one that came at 03:00, and two of those pull against each other, so no setting is safe. Lesson 026 said an alert comes from a tolerance rather than from a number somebody liked, and that governs the delivery as well as the threshold.
The push is an optimisation and the list is the guarantee. Lesson 040's sentence with one noun changed. You own one of the four hops to somebody's attention and the last one reports nothing, so the durable object is a row in your own store and the push is a hint that one appeared. That choice recovers a lost push, neutralises a duplicate, makes quiet hours free, and turns a provider outage into a missing buzz.
A badge is a count, never a counter. Lesson 019's atomic is not idempotent means a redelivered message corrupts a stored counter for good, and lesson 047 had already named the value that oscillates. Derive the badge from unseen rows and let the store be the authority.
The limiter's subject is the recipient, not the caller. Lesson 021's instrument turned around. Nobody inside the building can see the sum a person experiences, so the platform spends a per-person budget on the senders' behalf and refuses seven candidates out of eight: 8,102 refusals a second against 1,157 sends.
You can only afford to refuse an interruption because it was never the only copy. The two decisions hold each other up, since the budget is a scheduling policy if the row is durable and data loss if it is not. And unlike 066, the bound you chose for cost is not the content of your promise here, because a notification's identity is an event that happened once.
Check your understanding
The approvals team shows you that their notifications have a 61 percent open rate, far above the platform average, and asks to be exempt from the budget. Give them your answer with arithmetic in it, say which measurement would change your mind, and say what you would do if their director overrules you.
A platform launches genuine per-device delivery and read receipts. Say which of today's two decisions it reverses, which parts of the design you would delete, and which you would keep anyway and why.
Your badge is derived from unseen rows on every open. A client team says that costs them a round trip before the first paint and asks you to put the count in the push payload and trust it. Price both, say what a person can detect under each, and name the hybrid you would ship.
A marketing team books a broadcast to all twenty million people for 09:00 on a Monday. Work out what it costs in minutes of the fleet and in slots, say what you would offer them instead, and say what you would need to have built before you could offer it.
Galewatch wants its per turbine alert kept but never again wants sixty of them in four seconds. Design the change using only mechanisms in this course, say where it lives, and say what the engineer on call loses by it.
The billing team wants a text message when a payment fails, on the grounds that it is the one notification nobody may miss. Do the arithmetic at a failure rate you choose, say whether you agree, and name the cheaper mechanism that gets most of what they want.
Six months in, a graph shows that people who turned notifications off at the device level had received 2.1 times the median number of sends in the fortnight before. Say what that does and does not prove, what you would change on Monday, and the one experiment you would run before changing anything bigger.
Next lesson
068 Design Search Autocomplete. Today's service gets a whole day to decide and spends most of its effort refusing; autocomplete has to answer in the gap between two keystrokes, where refusing is not available and being slightly wrong is free.