Ardnave Point is a wind farm of three hundred turbines off the west coast, and the largest site belonging to Galewatch, a company that exists only in this course and takes a reading every two seconds from each turbine on the farms it monitors. Its link went down at twenty to six on the first Saturday in August, in a storm that had been on the forecast all week.
Nobody rang anybody. Lesson 003 settled this: when a farm's link drops, the turbines hold their readings in flash and replay them when it returns, so availability breaks and durability holds. Lesson 017 watched four hours of that arrive from a sixty turbine farm, and lesson 037 rebuilt the whole ingest path around a log so the next one would be boring.
The link came back at ten past one in the afternoon.
Seven and a half hours, three hundred turbines, a hundred and fifty readings a second: four million and fifty thousand readings, 810 megabytes at the two hundred bytes lesson 002 sized one at. Ardnave Point is new and offshore and came with a hundred megabit link, so the bytes crossed it in about sixty five seconds.
The log took them without noticing, which is what lesson 037 built it for. Every one of those readings carried a recorded_at hours behind its received_at, so lesson 037's rule sent all of them to the backfill topic rather than the live one.
Then the part nobody had designed.
At 13:11 the writer process holding that partition started draining it. Lesson 037 left six of those, splitting the log's twenty four partitions between them and inserting readings into the nineteen Postgres machines behind lesson 014's routing table. At 13:13 and 38 seconds the kernel killed it.
Lesson 001 has the mechanism: the out of memory killer wakes up, scores every process largely on how much memory it is holding, and kills the one that scores worst. The two biggest things on that box were two writer processes, and the one that died was the one that had read three million readings out of the log and put them in a queue inside itself with no limit on it.
What made the afternoon expensive is that those three million readings had already been marked as handled. The writers run with enable.auto.commit at its default, which has been true for as long as the Java consumer has existed, and an automatic commit follows the poll rather than the write. Lesson 036 named those two placements and said there is no third. Reading moved the offset. Writing happened later, in another thread, out of a queue the broker had never heard of.
So the replacement writer resumed from the committed offset, which was past the three million, read the remaining 890,000 records, and finished the backlog by half past one looking entirely successful.
The lag graph lesson 037 built was green throughout. Every committed offset sat within a second of the end of the log, which was true and was the wrong question, because the records those offsets had walked past were in a dead process's heap rather than in Postgres. Lesson 017's age of the oldest message reads that same offset, so nothing else alerted either.
On the Tuesday an engineer at Ardnave Point pulled up Saturday's power curve, which is exactly how lesson 031's dotted line was found. This hole was not dotted. Five hours and thirty three minutes of the Saturday morning, 05:58 to 11:31, was absent.
It was recoverable, with four days to spare. Lesson 037 set the log's retention at seven days, so the records were still there. Rewinding a group is done by time, which is how 037 did it in July, and the time on a record is the one the ingest box stamped when it accepted the reading, which for all four million of these is ten past one: a backfill partition has no 05:58 to seek to, because the whole Saturday arrived inside a minute. So they put the writer group back to the start of that minute, re-read all of it, and lesson 019's unique constraint on (turbine_id, recorded_at) refused the million readings that were already in Postgres. Retention is how long you are allowed to be wrong for, said lesson 037, and this is the first time anybody at Galewatch has spent days of it rather than minutes.
Two things never came back. A replay goes to one group's consumers, which is lesson 037's rule and the reason 038 tells you to ask the broker who is listening, so the alarm processes watching for a turbine sitting at zero have never seen those five and a half hours. And eighteen and a half thousand live readings from the four other farms that writer was serving went with them, two and a half minutes missing from four more power curves.
The fix shipped the Monday after. It was one argument: a bound of two thousand on that queue.
Twenty thousand in, one thousand out
Three numbers explain the afternoon, and two of them had never been written down.
the writer, 13:11, draining a backfill partition
reads from the log at 20,000 a second (parse, enqueue, poll again)
inserts into Postgres 1,000 a second (20 workers / 20 ms)
so the queue grows at 19,000 a second
The insert rate is the one everybody knows. Lesson 002 priced a Galewatch write at 20 milliseconds holding a connection, so twenty database workers in one writer complete a thousand a second, and six writers give the fleet six thousand against its ordinary 620. That headroom is what lesson 037 meant when it said July's replay ran at about five times the rate the farms were filling the log, a dial they chose rather than a limit they hit.
The read rate is the one nobody measures, because reading is not work. A loop that polls a partition, parses two hundred bytes and puts an object on a queue manages about twenty thousand a second on that box, and I would not defend that figure to two significant figures. What matters is the ratio: a loop that only copies beats a loop that does something by a factor of ten or more, and the gap is the rate your buffer fills.
Then the number I would genuinely go and measure. A reading is two hundred bytes on the wire and something larger once it is an object with a timestamp and a decimal in it, two to five times that depending on your runtime. Call it a kilobyte and treat it as suspect. Nineteen thousand a second at a kilobyte is 19 megabytes a second of growth, and the writers run two to a box on three boxes of 8 gigabytes, so one of them has about three gigabytes before the box is in trouble.
Three gigabytes at 19 megabytes a second is a hundred and fifty eight seconds.
That is the whole mechanism. 13:11 plus two minutes thirty eight is 13:13:38, and a process that had been healthy for the six weeks since lesson 037's deploy died because somebody three hundred miles away switched a link back on.
A buffer you never look at is correct until it is full. On an ordinary day that queue holds nothing: 620 readings a second arrive at a fleet that writes six thousand, so it is never more than a handful deep. Lesson 037 priced an ordinary deploy at twenty four seconds of rebalances and said nothing is lost, the lag just drains. That sentence is true of the log and it was also a claim about the consumer, and it had been right for six weeks for a reason nobody chose: a queue holding four records loses four records. July's replay, the one lesson 037 tells, put 226,800 records through the writers' queues, 227 megabytes at that kilobyte and spread across all six of them, and killed nothing. The storm did not create the bug. It supplied the depth.
Every buffer has a bound, and you chose it only if you typed it
None of this is really about Kafka.
That queue had a bound. It was three gigabytes, it was set by a purchase order, it was discovered at runtime, and it was enforced by a kernel feature whose job is keeping the machine alive rather than keeping your data. Every buffer in every system you will own has a bound. The only question is whether you wrote it down.
When a bounded buffer is full there are four things it can do, and one of them is not a choice.
| A full buffer can | Who finds out | What it costs |
|---|---|---|
| block the producer | the producer, at once | lesson 020's timeout |
| drop the oldest | the reader, later | history, silently |
| drop the newest | the producer, as an error | lesson 021's refusal |
| grow | nobody, then everybody | the hook |
Blocking is backpressure proper: the consumer's inability to keep up becomes a signal that travels back up the chain, and the producer slows down. Nothing is lost and nothing is refused. Arrival comes down to service because the producer is physically prevented from going faster.
Dropping is loss, and it is sometimes right. Which end you drop from is a product decision with a rule attached: drop the records whose value decays fastest. Galewatch sells dashboards on being current, so a panel's feed that had to shed should throw away the oldest unwritten reading and show the newest. The year long report lesson 008 cached indefinitely wants the opposite end, having no opinion about recency and every opinion about completeness. Same records, same buffer, opposite policies, which is why nobody can ship you a default drop policy.
Refusing is lesson 021 and I am not rebuilding it. The one thing to add is that refusal is the only one of the four that says anything at all to a producer you do not own.
Growing is the hook. It looks like the kind option and it is the other three applied later by something that does not know what your data is worth.
Blocking is the real thing, and a timeout turns it into load
Blocking has a reputation problem and deserves about half of it.
Give a producer a bounded queue and it will sit there with work in its hands, which feels like a bug and is the mechanism working. The question is what sits behind the producer. A thread with nothing better to do gives you the whole benefit for free. A request holding a socket open to a caller with a timeout does not, and lesson 020 published what happens next: the caller gives up, retries, and your offered load climbs while you are drowning. Lesson 020 called that the multiplier switching on at the worst possible moment.
A timeout turns backpressure into load. That sentence decides whether blocking is available to you at all, because backpressure needs a producer that can wait, and a producer with a client timeout can wait exactly that long.
Galewatch's turbines are the sharpest case. Lesson 021 worked out why rate limiting the ingest path protects nothing: a refused turbine either sends the reading again a minute later, which is the load arriving anyway, or drops it, which loses the product. Blocking has the same shape with worse manners, because a turbine holding a connection open is a turbine not sending the reading it took two seconds afterwards. Lesson 017's October afternoon is the full version, where four ingest boxes blocked on one farm's connection pool and eighteen other farms' turbines timed out and resent, which is where most of those twenty two minutes went.
So blocking is excellent inside a process and expensive across a network boundary. Inside the writer it costs nothing: the read loop is not holding anybody's request, and the records it declines to read are on the broker's disk where they already were. That is the entire fix from the hook.
There is a name for the arrangement that gets this right without being asked. A pull consumer fetches work when it is ready for more, so its own pace is the signal and nobody has to send one. A push producer sets the pace and leaves the consumer to object. Lesson 037's log is pull the whole way down: nothing is ever delivered to a writer, the writer polls. The August writer had backpressure for free and gave it away, which is the usual way this goes.
Which gives the rule I would put on a wall. Push the backpressure to the last place that can wait for free. Between a writer's heap and a broker's disk the disk wins, and 810 megabytes against the 75 gigabytes lesson 037 sized for a week of the whole fleet is one percent of a buffer that already exists, is already backed up and already has a graph on it.
The oldest backpressure in the building
You have been relying on a bounded buffer with a blocking producer since lesson 004, and that lesson never named it.
Lesson 004 explained slow start: a fresh TCP connection opens at about ten packets and doubles every round trip, because the sender has no idea what the network can absorb. That is congestion control and it protects the network. There is a second window, set by the other end, and it protects the receiver.
Every acknowledgement a TCP receiver sends carries a receive window, the number of bytes it currently has room for, and the sender may have that many bytes unacknowledged and not one more. If the application on the receiving side stops calling read, the kernel's buffer fills, the advertised window shrinks, and at zero the sender stops sending and sits there probing politely until there is room. That is flow control: blocking backpressure, implemented once, in a place nobody configures, and the reason a slow reader of a TCP socket does not lose anything. It only makes the sender wait.
The same shape has a library version you may already have imported. The consumer sends a number, the producer may send that many items and no more, then waits for the next number. That is request(n) in Reactive Streams, and it is what HTTP/2 does per stream with a window of its own: pull, with the credit written down.
Which gives the hook a second reading that I like more than the first.
The ingest boxes answer a turbine in two milliseconds because they append and return. If one of them slowed down, TCP would have squeezed the turbines without being asked. The writer did the opposite: it emptied its input as fast as it possibly could into a queue of its own. Copy data out of a socket faster than you can process it and you have turned the kernel's backpressure off. Every framework that hands you a parsed message and an unbounded work queue has done that on your behalf, and the commonest example is not obscure. Java's Executors.newFixedThreadPool puts a LinkedBlockingQueue with no capacity argument in front of its threads, which is a queue bounded only by the heap.
Build the pool by hand with a ThreadPoolExecutor, give the queue a size, and you get to pick one of the four behaviours. There is even a policy for today's: CallerRunsPolicy makes the thread that submitted the work run it, so a full queue slows the submitter down by handing it the job. Backpressure in one line, in the standard library of the language most of this runs in.
How deep, in seconds
Two thousand. Where does a number like that come from?
Not from memory, which is how everybody picks it. Lesson 002's Little's law gives the honest version read backwards: concurrency equals throughput times latency, so a queue's depth divided by its drain rate is the delay waiting at the back of it. A bound in items is unreadable. The same bound in seconds is a product decision anybody can argue about.
Galewatch sells dashboards that lag by seconds. The writer drains a thousand a second. Two thousand deep is two seconds, which fits inside the promise with room for the rest of the path. Type a hundred thousand, feeling generous, and you have chosen a hundred seconds of staleness without writing the word staleness anywhere.
A buffer's bound is the longest you will let a fresh item wait behind a stale one. That is the sentence I wish somebody had handed me years earlier, because it dissolves the question. You are not sizing a buffer to absorb a burst. You are naming the worst delay you will admit to, then dividing by the drain rate.
Lesson 017's two lanes get sharper under that rule, and so does their limit. The ingest boxes split live records from replayed ones into two topics, and lesson 037 has the writer pausing its backfill partitions whenever the live ones hand back a full batch. That priority rule is real, and in August it was undone by something behind it: both lanes fed one queue. Three million backfill records queued ahead of a live reading get written first, whatever the read loop decided, because a queue is in order. You cannot prioritise upstream of a shared buffer. Either the lanes keep separate buffers the whole way to the consumer, or the priority is decoration.
A bound of two thousand makes it real again, for a reason worth seeing. A live reading can now only queue behind two thousand backfill records, which is two seconds. The four farms on that writer stay inside the product's promise through a seven hour backlog, not because anything got faster, but because the unfairness the buffer can hold fell from three million records to two thousand.
Price the bounded Saturday against the real one. A partition is read by exactly one process, which lesson 037 named as the hard ceiling on parallelism, so the backlog drains at that writer's thousand a second, less the hundred and twenty the four live farms are still sending: 4,050,000 records, seventy six minutes. Seventy six minutes of one farm's panel being behind, on the lag graph the whole time, with no crash, nothing lost and nobody else affected. The afternoon that actually happened cost five and a half hours of one farm's history, three days of nobody knowing, and four other farms stale behind a green dashboard.
The bound made nothing faster. Lesson 017 said the same of the queue itself: the total work never moves, only who waits and where. What a bound buys is that the waiting happens somewhere you can see it, and that the commit still sitting in the placement 036 said loses work can only ever be wrong about two thousand readings.
The chain ends at something that cannot slow down
Backpressure is a signal, and a signal travels until something absorbs it. Walk Galewatch's chain from the bottom and ask at each link who gets told.
Postgres shard <- the writer waits on a connection
the writer <- the read loop blocks on a bounded queue
the log <- 75 GB of disk, seven days, nobody waits
the ingest box <- TCP squeezes the turbine
the link <- a radio that does not negotiate
the turbine <- flash, and then nothing
the wind
Every arrow is a buffer and a party that can be made to wait, and following them far enough gets you to the wind, which has never slowed down because a database was busy. One link before the end is the turbine's flash, and lesson 017 said what that is: a queue, with a producer, a consumer, a depth and a drain rate, and no graph anybody in the office can see. It held four hours in lesson 017 and seven and a half in August, and nobody at Galewatch knows what it does in the eighth hour, because the firmware author picked one of the four behaviours and nobody has read that code. Lesson 058 owns what that chip is really doing.
Backpressure travels up the chain until it reaches something that cannot slow down, and whatever that is pays. At Galewatch it is a flash chip on a hillside, which is the only reason the ingest path can afford to be rude to its producers.
Then the case where the chain crosses a boundary you do not own. Lesson 021 raised it and left it open and lesson 029 answered the credentials half: three hundred turbines belonging to a customer who runs their own hardware and their own firmware, which Galewatch cannot change. The backpressure half has an answer nobody likes. You cannot block them, because blocking a client whose timeout and retry behaviour you have never read is a slower way of being retried at. You cannot bound their buffer, because it is in a building you have no keys to. What is left is lesson 021's refusal, sent early enough to be useful and loudly enough to be noticed, plus the admission that a refusal to firmware that treats every error as retry in a minute is a drop with extra steps.
Which is lesson 021's own sentence arriving from the far side. A limiter is for callers you do not control and backpressure is for the ones you do, and the reason is not politeness: only one of them has somewhere to put the work.
Stagefront, an event ticketing service that exists only in this course, where a stadium show goes on sale at exactly ten in the morning and an oversold seat is a lawsuit, is the case where the producer is a person. Lesson 017 argued for a waiting room admitting five hundred a second, which is backpressure built deliberately and in public: the 2,833 a second who cannot be served are made to wait, with a position and a closing time, rather than refused one at a time.
Its bound comes out of the same rule. Forty thousand seats at five hundred a second is lesson 017's eighty seconds of inventory, and since plenty of people who get in will not buy, admit two or three times that: the room stops at about a hundred thousand, which is three minutes of waiting at five hundred a second, and tells everybody after that no straight away. The alternative is the one lesson 021 confirmed Stagefront still has not fixed, walking a hundred and sixty thousand people to the front of a queue one at a time to be told the show is sold out.
Marlow's version costs one row
Marlow Books is an online bookshop run by four people, and it exists only in this course. At lesson 027's rate of an order every fifty seconds nothing in that shop has ever been behind, and lesson 040's new outbox drainer publishes one row every fifty seconds to a broker that accepts in two milliseconds. There is no backpressure story in that drainer.
The shop has had one for months somewhere else. Lesson 019's sweeper runs every two minutes over orders pending for longer than two minutes, asking the payment provider what happened to each. Lesson 027 watched a provider stop answering, with a dozen questions at a ten second timeout coming to a hundred and twenty seconds, which is the sweeper's entire schedule, and lesson 035 is the Wednesday evening two copies ran at once and the shelf came out wrong.
That is arrival exceeding service with a service rate of zero, and the buffer is a where status = 'pending' clause nobody thinks of as a buffer. Lesson 035's fix was exclusion, a row on (job_name, run_slot) inserted with on conflict do nothing, and it is backpressure in work clothes: a full buffer that drops the newest. The second run arrives, finds the slot taken, and leaves the work in the table.
What lesson 035 did not need and this one does is the other bound. A sweep should take the oldest fifty rows and no more, because a job whose work per run is unbounded has the hook's problem with a cron entry instead of a heap. Cap it and the list grows on the days the provider is dark, which is the point: a backlog that is a row count in Postgres sits on the same graph as everything else, instead of being a run time quietly crossing its own schedule at ten to nine on a Wednesday evening.
A cron job's backpressure is a cap on how much work one run accepts. Two words in a SQL statement, and almost nobody writes them.
Recap
Every buffer has a bound, and you chose it only if you typed it. The writer's queue was bounded at three gigabytes by the size of a box, enforced by lesson 001's out of memory killer, and discovered on the first afternoon anything filled it. An unbounded queue is not a kind one: its limit and its behaviour at that limit were both chosen by somebody who has never seen your data.
A full buffer can block, drop the oldest, drop the newest, or grow. Blocking is backpressure, dropping the oldest is loss, dropping the newest is lesson 021's refusal, and growing is the other three applied later by the kernel. If you drop, drop whatever decays fastest, which is the opposite end for a dashboard and for a year end report.
A timeout turns backpressure into load. Blocking only works where the producer can afford to wait. It is free inside a process, where nothing is holding a request, and it buys you lesson 020's retry multiplier the moment you do it across a network to a client that gives up.
Copy data out of a socket faster than you can process it and you have turned the kernel's backpressure off. TCP's receive window gets this right: the application stops reading, the window shrinks to zero, the sender stops. A framework that hands you a parsed message and an unbounded work queue has switched that off for you.
A buffer's bound is the longest you will let a fresh item wait behind a stale one. Divide depth by drain rate and the bound is a number in seconds anybody can argue about: two thousand in front of a thousand a second is two seconds. And you cannot prioritise upstream of a shared buffer, because a queue serves in order, which is what made lesson 017's two lanes decoration in August.
Backpressure travels up the chain until it reaches something that cannot slow down, and whatever that is pays. At Galewatch it is a flash chip on a hillside, which is the only reason the ingest path can be rude. When that thing belongs to somebody else, you are left with lesson 021's refusal and the knowledge that firmware you cannot read will probably treat it as a retry.
Check your understanding
Your service reads from a queue and writes to a search index through a thread pool whose queue you have just discovered is unbounded. Nobody can tell you the right bound. Say what you would measure to derive it, and what you would ship while the measurement runs.
A colleague proposes fixing the Saturday by giving the writer boxes 64 gigabytes each. Say exactly what that buys, what it makes worse, and which number from today you would show them first.
Galewatch is asked to add a consumer group that forwards every reading to a customer's own HTTP endpoint, which accepts about fifty a second and times out at thirty. Work out what that group's lag does on an ordinary day, say which of the four behaviours you would choose when its buffer fills, and say how the customer finds out.
Stagefront's waiting room gets built, admits five hundred a second, and on the first on-sale the room itself fills: more people arrive than it will hold. Say what the room should do, what a person should see, and which earlier lesson owns the part you cannot solve.
Marlow's founder caps the sweeper at fifty rows a run, and a fortnight later the pending list holds nine hundred rows and nobody has noticed. Say what alert you wanted, which quantity it measures, and which lesson already argued for that quantity instead of the obvious one.
A team argues their service needs no backpressure because every call it makes is synchronous and has a timeout, so nothing can ever queue up inside it. Say where they are right, then find the two buffers they have not counted.
Next lesson
042 Circuit Breakers, Bulkheads and Graceful Degradation. Today the answer was making the pressure travel backwards; next lesson is the dependency that is never going to answer at all, and what to do about a signal that says stop calling me altogether.