The publisher payout job at Marlow Books, a four person online bookshop that exists only in this course, runs at two in the morning on the first Sunday of the month. It loads a year of order lines, groups them by publisher, and emails each one a CSV. Lesson 011 moved it onto the read replica and then spent a long section on the ways that went wrong.
On the first Sunday in December it did not pay anybody.
It started on time. Four minutes in it reached an order line whose publisher row was gone, because in November the founder had deleted a duplicate publisher at a psql prompt at eleven at night, which is the fifth of the five write paths lesson 009 counted and the one no code review reaches. The script did what scripts do with an unexpected nothing. It raised, printed a traceback, and exited non-zero.
Cron did what it has always done with a job's output. It mailed it to the local user on box A, where the cron line lives, and there it joined years of other mail nobody has opened.
Eleven days later a publisher emailed the shop to ask when November's money was coming.
The founder found the traceback in about four minutes. The eleven days is the part worth a lesson.
Marlow is not an unmonitored shop any more. There is lesson 003's uptime monitor, nine dollars a month, fetching the home page once a minute from a machine in Singapore. There is the slow query log lesson 010 turned on at 200 milliseconds after the crawler weekend. There is lesson 012's replication lag graph, and there are lesson 006's health checks on both boxes.
Every one of those instruments watches something happen.
A job that does not run does not happen. No log line, no metric, no error, no CPU, no request, no row. The absence of an event is not an event, and there was nothing there to catch.
Two kinds of later
Lesson 017 put work on a shelf and called it the buffer with a memory. Everything from here is about getting it back off again, and the first thing to notice is that "later" turns up in two quite different shapes.
Some work is handed to you by a caller. A customer checks out and a confirmation email has to go. A turbine sends a reading and a row has to be written. There is a message, somebody upstream knows the work exists, the broker is holding evidence of it, and that evidence has an age. Which is exactly why lesson 017 could tell you to alert on it.
Some work is handed to you by a clock. Pay the publishers on the first Sunday. Expire the abandoned seat holds every minute. Nobody asked for it, nothing arrived, and the only record that the work was ever due is a line in a crontab plus a belief in somebody's head.
A queue forgets nothing. A schedule has nothing to forget.
That difference decides where the bugs live. For work a caller handed you the record already exists, so the hard questions all come afterwards: what happens when the far end is down, how long you keep trying, where the work goes when you stop. For work a clock handed you the hard question comes first, and it is embarrassingly basic. Did it start, and did exactly one of you start it?
Marlow has now got both of those wrong, in the same script.
The job that has to run once
Lesson 005 established the first half. The payout job lived in the application's crontab, the crontab was copied along with the machine image when the second box went in, and on the first Sunday of October both boxes woke up, both loaded a year of order lines, and both emailed a CSV. Nobody was paid twice. The founder established that by spending Sunday morning proving it.
Lesson 007 wrote the fix and nobody built it, for the most ordinary reason there is: the bug had already stopped happening. The founder had taken the cheapest of lesson 007's four answers and deleted the cron line from box B, which for four people is a legitimate answer and works right up until the day somebody rebuilds box B. Lesson 007 priced that answer in a clause that reads differently this morning: if box A is down on the first Sunday, nobody is paid and nothing tells you.
So here it is at last, with two columns lesson 007 did not have.
create table job_runs (
job_name text not null,
run_date date not null,
started_at timestamptz not null default now(),
finished_at timestamptz,
outcome text,
primary key (job_name, run_date)
);
The claim itself is lesson 007's and has not changed. Every box tries to insert a row for today at 02:00, on conflict do nothing, and whoever gets a row back owns tonight's run, because deciding who wins is the one thing a database is genuinely excellent at. What is new is that the job stamps finished_at and an outcome on its way out of the door.
Lesson 007 was careful about that column and it is worth being careful about it again, for a different reason. Its warning was that finished_at is what you need if a second box may take over a run the first one abandoned, and that the moment somebody can retake an abandoned job, the job has to be safe to run twice, which is lesson 019 and is a harder problem than the lock.
Every word of that is true, and every word of it is about using the column as a lease. Nobody has to retake anything for this to earn its place:
select job_name, started_at
from job_runs
where run_date = current_date and finished_at is null;
One query over a table you already have, asking which of today's runs has a start and no finish. No leader, no lease, no new failure modes, nothing that can run anything twice. It is also the whole of the eleven days. The cheapest half of a hard problem is usually the read only half, and it gets skipped because people go looking for the hard half first, find it, and put the ticket in the backlog.
Then the limit, which is the one that sends the rest of this lesson somewhere else.
A row that never appears is invisible to a query over the table. If cron does not fire at all, because the box is down, or somebody rebuilt it without the crontab, or the line has a typo in it, there is no row with a null finished_at. There is no row. The table can tell you a run started and did not finish. It cannot tell you a run was due.
Knowing what was due is the schedule's job, and the schedule is the thing that just failed.
Nothing in your building watches for absence
Write down where your evidence actually comes from and this stops being surprising. Logs are written by code that ran. Metrics are incremented by code that ran. A trace exists because a request existed. An alert fires because a number somebody is publishing crossed a line, and when nobody is publishing the number, the alert is not false. It is absent.
You can't see a silence from inside it. You can only compare it against something you wrote down beforehand, somewhere that doesn't share the failure.
The shape has a name and it is a good one: a dead man's switch. The job's last act is to tell something outside itself that it finished. That something holds a deadline and exists to complain when the deadline passes with no word. You aren't alerting on an error. You're alerting on the ping that did not arrive.
Marlow has owned most of one for two years and has never thought of it that way. Lesson 003's nine dollar monitor sits in Singapore, fetches the home page once a minute, and sends an email when it doesn't come back. It goes and asks rather than waiting to be told, so the plumbing is not identical. The part that matters is: the expectation lives outside the thing it is watching. Nothing running on box A can hold an expectation about box A being gone.
Which raises the obvious question, which is who watches the watcher, and lesson 006 has already shown you how this kind of question ends. Put a balancer in front of your balancers and you have bought a new single point of failure, so the chain stops at DNS or anycast, outside your control on purpose. The chain of watchers stops the same way. The last one has to be something whose failure mode is telling you about a problem that isn't there, rather than saying nothing about one that is. A watcher that fails quiet is not a watcher.
Now the number that makes any of this real, and it isn't a technical number at all.
A deadline says how late this work may be before somebody should be woken up, which is the same question as what one missed run costs per hour. For Marlow's payout: a publisher waiting a day for money is annoyed, and a publisher waiting eleven days is on the phone asking whether this shop is solvent. So the answer is hours, not days. Call it six. Then set the check earlier than the deadline by however long a fix takes, which puts it at 06:00 the same morning.
Ask it again at Stagefront, the ticketing service where a stadium show goes on sale at exactly 10:00 and oversold seats are a lawsuit. Lesson 015 established that clicking a seat commits a hold, and that a hold nobody confirms has to expire, so Stagefront owns a timer and a sweeper, and every "your seats are reserved for 4:59" you have ever seen is that timer made visible. Stop the sweeper and nothing errors anywhere. Every buyer who wanders off with a hold takes a seat out of the sale permanently, so the show sells out with rows of empty seats in it, and the first instrument to notice is a photograph of the stadium.
Marlow's payout may be six hours late. Stagefront's sweeper may be ninety seconds late. In a crontab those two jobs are indistinguishable.
The schedule is written down. The deadline almost never is.
Marlow's story ends like this. The job_runs table and a deadline went in that week. On the first Sunday in January the job died again at 02:06, on a different bad row, because the founder had fixed the publisher and not the script. At 06:00 the watcher noticed there was no finished run and the phone buzzed. The CSVs went out at 07:36.
Five and a half hours late instead of eleven days. The bug is still in that script. What changed is how long it costs, which is lesson 003's argument about MTTR arriving somewhere that lesson never thought to look.
When the far end is down
Marlow's order confirmation email, which lesson 017 argued for and did not build, went onto a queue in the middle of December. That is not when you ship things, and the founder shipped it anyway, so that an email provider's bad afternoon would stop multiplying into the shop's own availability. Lesson 017 sized that queue at a fifth of a message a second, with a depth of zero on every ordinary day of the year.
On the Tuesday of Christmas week the provider went down at 10:00 and came back at 12:00.
The consumer took a message, called the provider, got a 503 and had a decision to make. Retrying straight away in a tight loop is the obvious thing to write and is how you turn somebody else's outage into your own; lesson 020 owns that properly. What it did instead was exponential backoff: wait a minute, then on the next failure two, then four, eight, sixteen, thirty two, sixty four, a hundred and twenty eight, each failure doubling the sleep before the next attempt. With a little randomness on each wait, for the same reason lesson 008 jittered its cache expiries.
Eight retries, doubling, so nine attempts counting the first one. Now add the sleeps up.
One plus two plus four plus eight plus sixteen plus thirty two plus sixty four plus a hundred and twenty eight is 255 minutes, which is four and a quarter hours. That sum is your policy. It is how long Marlow has decided its email provider is allowed to be down before the shop starts throwing confirmation emails away, and nobody at Marlow decided it. It fell out of "retry eight times, doubling", which is what everybody writes.
Hold that figure against the shop's own record for a second. Lesson 003 put a typical Marlow outage at three and a half hours. Lesson 016's Saturday, when the disk filled and Postgres shut itself down, ran four hours fifty. Marlow has set its patience for somebody else's failure at slightly less than its own worst outage of the year, by accident.
Two hours is comfortably inside four and a quarter, so nothing was discarded that Tuesday. The interesting part is when the emails actually went out.
one order, placed 10:56, provider down 10:00 to 12:00
attempt 1 10:56 fail then sleep 1 minute
attempt 2 10:57 fail then sleep 2
attempt 3 10:59 fail then sleep 4
attempt 4 11:03 fail then sleep 8
attempt 5 11:11 fail then sleep 16
attempt 6 11:27 fail then sleep 32
attempt 7 11:59 fail then sleep 64
attempt 8 13:03 sent
That customer ordered at 10:56 and was emailed at 13:03. Their seventh attempt failed at 11:59, one minute before the provider came back, and the next sleep in the schedule is sixty four minutes long, so the message slept straight through the recovery and most of lunch.
The customer who ordered at 11:58 was emailed at 12:01. They failed at 11:58, failed again at 11:59, waited two minutes, and got through on the third try.
Sixty two minutes later into the queue, sixty two minutes earlier into the inbox.
Backoff punishes the oldest work. By the time a far end recovers, the messages that have been failing longest are the ones sleeping longest, so they come back last, and they come back in roughly the reverse of the order they arrived. Lesson 017 told you first in, first out is a policy rather than a law. Retries make a stronger claim than that: once anything is being retried, the order work happens in stops resembling the order it arrived in at all.
The incident is worth stating as one number. The provider was down for two hours, and Marlow's last delayed confirmation went out at 13:03, so from a customer's side the thing lasted three hours and three minutes. An hour and three minutes of that was Marlow's own retry schedule, waiting politely for a service that had already come back.
The obvious fix is a cap. Never sleep longer than five minutes and everything is out within five minutes of recovery. Do that and then add the sleeps up again: one, two, four, then fives, and the same eight retries now carry thirty two minutes of patience. The tolerance you never chose has gone from four and a quarter hours to half an hour, which you also did not choose.
So stop counting attempts. Retry to a deadline, not to a count. Back off to a cap of five minutes, keep going while the message is less than four hours old, and give up when it isn't. Four hours is a number a founder can defend to a customer, to a publisher, or to themselves at two in the morning. Eight retries is a number that quietly produces a duration nobody in the building has ever worked out.
Notice that it is the same kind of number as the watcher's deadline in the first half of this lesson. Both of them come down to a question that has nothing to do with crontabs or brokers: how late may this be before somebody has to know?
Where the work goes when you give up
Give up and the message has to go somewhere, and that somewhere is a dead letter queue: a second queue that the broker moves a message onto once it has failed too many times or run past its deadline.
It fixes nothing. It is a place. What it buys you is that the work was not silently deleted, and that one bad message is no longer blocking the pipe. Everything after that is a person.
Which is why a dead letter queue nobody reads is the mailbox on box A with better branding. It needs exactly what the payout job needed: an owner, an alert on it being non-empty at all rather than on some threshold, and a way to put a message back once the problem is fixed. That last one always gets written at three in the morning by somebody who had not planned to write it.
What lands there is not always a retry that ran out of patience. Galewatch, the telemetry service that collects a reading from each of nine hundred turbines every two seconds, has a harder version. Lesson 017 named the dial: once the ingest box has answered 200, the turbine deletes its local copy, so from that instant the broker holds the only copy of that reading in the world. A dead-lettered reading is not a failed attempt to do something. It is the only surviving record of what a machine on a hillside was doing at half past three yesterday afternoon, and that moment is not coming back. Draining that queue with a delete is a decision about the product, and it should be made by somebody who knows that.
Then there is the case a retry schedule cannot help with at all.
A poison message fails every single time it is tried, for reasons that have nothing to do with the far end. Patience is useless against it. The far end is fine and the message is wrong.
Marlow met one on the Friday of Christmas week at ten to five, the founder's usual slot. The change added a field to the confirmation email's payload and went out in the producer and the consumer at the same moment, which feels like the careful way round and is precisely the mistake. The queue was not empty: the provider had been sluggish that afternoon and forty messages written by the previous version were still sitting there. The new consumer picked up the first of them and raised a KeyError on a field that had not existed on Tuesday. Lesson 017 flagged this exact row in its own table and handed the contract to lesson 052, because a message is an interface between two programs that are never running at the same time. The order that works is to teach the consumer to read both shapes, ship that, wait for the queue to drain, and only then change the producer.
There are two versions of what happens next and the gap between them is large. If the consumer catches the error, tells the broker the message failed and carries on, the broker offers that message to somebody else in thirty seconds and hands out other work meanwhile, and forty poisoned messages cost you some noise in the logs. If the consumer does not catch it, the process exits, the supervisor starts a fresh one, the broker hands the new process the same message, and it exits again. Nothing else on that queue moves at all. It is lesson 004's head of line blocking, at the front of a queue instead of a TCP connection.
Marlow's consumer was about thirty lines long with nothing catching exceptions around the handler, so it was the second one.
It lasted eleven minutes. The age of the oldest unprocessed message crossed ten, which is the alert lesson 017 argued for, and the founder's phone buzzed. At a fifth of a message a second, eleven minutes put around a hundred and thirty more customers in the queue behind those forty, which is nothing, which is the entire point. The number stayed small because something was watching an age.
A deploy that is safe because the queue is empty is a deploy that is unsafe on your worst day. That consumer had gone out three times since it was written, every one of them to an empty queue.
The founder added a delivery limit that afternoon, so that a message which has been handed out three times moves aside instead of coming back. The forty went there, a five line script gave them their missing field, and they were back on the main queue before six.
Put the two halves of the month side by side, because it is the same shop and the same founder. Work that had been written down was caught in ten minutes. Work that had never been written down took eleven days. The instruments were not better in the first case. There was simply something there to instrument.
Recap
Later arrives in two shapes. Work a caller handed you leaves a message behind, and a message has an age, which is what makes lesson 017's alert possible. Work a clock handed you leaves nothing behind at all. A queue forgets nothing; a schedule has nothing to forget.
Whoever writes the row first runs the job, and whoever reads the table afterwards finds out it never finished. Lesson 007's insert on the job name and the run date is still the cheapest single-owner mechanism in the course. Adding finished_at as a lease is lesson 019's problem. Adding it as a question you can ask costs nothing and is most of the value.
Nothing inside your system watches for absence. Every instrument you own is evidence that something happened. Seeing a silence means writing the expectation down in advance and checking it from somewhere that does not share the failure, which is a dead man's switch, and a watcher that fails quiet is not a watcher at all.
The schedule is written down and the deadline is not. How late may this job be before somebody is woken up? Marlow's payout can be six hours late. Stagefront's hold sweeper cannot be ninety seconds late without seats quietly leaving the sale. In a crontab the two lines look the same.
The sum of your backoff schedule is the outage you have decided to tolerate. Doubling from one minute for eight retries is 255 minutes, four and a quarter hours, sitting just under Marlow's own worst outage of the year and chosen by nobody. Pick the deadline, then let the schedule fall out of it.
Backoff punishes the oldest work. The customer who ordered at 10:56 got their email sixty two minutes after the customer who ordered at 11:58, because the longest sleep belongs to whoever has been failing longest. Capping the sleep fixes that and shortens your patience in the same stroke, which is why the deadline has to be the thing you set.
A dead letter queue nobody reads is the mailbox on box A with better branding. It is a place rather than a fix, and it needs an owner, an alert on being non-empty, and a way back onto the queue.
Check your understanding
The payout job now pings the watcher when it finishes. One month it finishes in four seconds having paid nobody, because a change to the query started returning zero rows. Say what the watcher does, what a row in
job_runsshows, and what you would add so this case is not invisible as well.Marlow's email consumer retries with a five minute cap and gives up when a message is four hours old. The provider goes down for six hours during Christmas week. Using lesson 017's rate for that queue, work out roughly how many messages reach the dead letter queue and which orders they belong to, then say what you would want done with them.
Stagefront's hold sweeper runs every minute and the team wants to know how late it is allowed to be. Describe what you would measure during a single on-sale to turn "seats quietly leave the sale" into a number, and what deadline you would then put on the watcher.
A colleague proposes deleting the dead letter queue and having the consumer log the failed message body at error level instead, on the grounds that the logs are searchable and nobody was reading the dead letter queue anyway. Give the strongest version of their argument, then the reason you would still refuse it.
Galewatch dead-letters a reading its writer could not store. Say who at Galewatch owns that queue, what deadline belongs on it, and why your answer differs from the one you gave for Marlow's confirmation email.
Next lesson
019 Idempotency: Safe to Retry. Today's retries, the abandoned run another box takes over, and the message that comes back after a half-finished attempt all rest on the same quiet assumption, that doing the work twice is harmless; next lesson is about how you actually earn it.