Message Queues: The Buffer With a Memory

The link to Tarrow Ridge, a wind farm of sixty turbines at the end of a long access road, went down at 10:30 on a Wednesday morning in October, in weather that had been on the forecast for two days. Nobody at Galewatch, the telemetry service that collects a reading from each of nine hundred turbines every two seconds, was worried about it, because lesson 003 already declared this a solved problem. When a farm's link drops, the turbines buffer their readings locally and replay them when it comes back. Availability breaks, durability holds, not one reading is lost.

The link came back at 14:30.

At 14:31 the dashboards for the other eighteen farms stopped updating.

Here is what happened in between. Lesson 002 measured Tarrow Ridge's site link at about 20 megabits a second. Four hours of readings had been sitting in flash on those sixty turbines, and at 14:30 every one of them started pushing as fast as the link allowed. The readings arrived at Galewatch's ingest boxes, which did what they have always done: take the reading, write it to Postgres, return a 200.

A Galewatch write takes 20 milliseconds and holds a database connection while it does. That is lesson 002's arithmetic, and lesson 005 turned it into a ceiling: one connection completes fifty writes a second, a hundred connections complete five thousand. Since lesson 013 cut the database by farm, Tarrow Ridge has a machine of its own, with its own hundred connections and its own five thousand a second.

Twelve and a half thousand readings a second arrived.

The ingest boxes ran out of database connections in under a second. Their request workers then sat waiting for one. Lesson 013 gave each farm a database of its own, but those four ingest boxes share a single pool of request workers across all nineteen farms, which is how everybody writes it the first time, and a worker blocked on Tarrow Ridge's connections is a worker that Kilmore Sands, the largest of the nineteen farms at a hundred and forty turbines, cannot have. Nineteen farms, one pool of workers, one farm's storm. The turbines at the other eighteen farms started timing out, and doing what turbines do when a write times out, which is send it again in a minute. Lesson 020 owns that part properly.

The fleet was current again at 14:52. Twenty two minutes, for a backlog that represented eighty six seconds of database work.

Most of those twenty two minutes were not Tarrow Ridge's readings at all. They were eighteen other farms' readings, buffered and resent while the workers were busy, arriving in their own small storms behind the first one.

Nothing was broken. No disk filled, no process died, no query was slow, no machine was undersized. Work arrived faster than the place it was going could accept it, and there was nowhere to put it in the meantime, so it piled up in the worst available place: inside request handlers, on connections, in front of everybody else's traffic.

The arithmetic nobody had done

Write down the three numbers and the shape of the afternoon becomes obvious.

Tarrow Ridge, 14:30, the link comes back

  buffered on 60 turbines     432,000 readings   (4 h x 30 a second)
  the link will deliver at     12,500 a second   (20 Mbps, 200 bytes each)
  the shard can store at        5,000 a second   (100 conns / 20 ms)

Sixty turbines at one reading every two seconds is thirty a second, which is lesson 002's figure for that farm, and four hours of it is 432,000 readings. At the 200 bytes a reading lesson 002 sized them at, that is 86 megabytes. A 20 megabit link moves about 2.5 megabytes a second, which at that size is 12,500 readings a second, so the bytes themselves take just under 35 seconds to cross, assuming the turbines ship their buffers in batches rather than one reading per request, which they do.

Twelve and a half thousand a second against a ceiling of five thousand. Arriving two and a half times faster than they can be stored.

A burst is not an overload. Twelve and a half thousand a second sounds like a system that needs to be two and a half times bigger, and it is nothing of the kind, because the arrival is finite. There are 432,000 readings and there will never be a 432,001st. Divided by five thousand a second, the entire storm is eighty six seconds of database work. The machine was never too small. It was too small for thirty five seconds, which is a completely different problem with a completely different fix.

When arrival rate genuinely exceeds service rate and stays there, lesson 001 already gave you the answer: the denominator goes negative, there is no steady state at all, and the line just grows for as long as the traffic lasts. Note what lesson 001 called that line. A queue. It was already talking about one; it just happened to be an invisible queue made of waiting requests, which is the worst kind to own. When arrival exceeds service for half a minute, all you need is somewhere better to put the difference.

A queue is a place to write down that something has to happen

That somewhere is a queue, and the definitions are mercifully small.

A producer is whoever has work that needs doing. A message is the work, written down: a few hundred bytes describing one reading, or one email to send, or one thumbnail to generate. A broker is the thing that holds messages, usually a separate piece of software with a name you have heard. A consumer reads messages from the broker and does the actual work, at whatever rate suits it, and tells the broker when it is done.

The change at the ingest boxes is one line long and it changes the shape of the failure completely. Instead of "write this reading to Postgres, then answer", the handler does "append this reading to the queue, then answer".

before                          after

turbine -> ingest -> Postgres   turbine -> ingest -> queue -> writer -> Postgres
             |                                |               |
      answers when the           answers when it has     drains at whatever
      write has landed           written it down         rate Postgres allows
        (20 ms, or never)           (2 ms)                 (5,000 a second)

Call the accept a couple of milliseconds. The honest version is that it depends entirely on whether the broker waits for a disk before it says yes, which is a dial I will open in a moment, and which most people turn the wrong way.

Now redo the afternoon with a queue in the middle.

Readings arrive at 12,500 a second and leave at 5,000, so the queue's depth grows at the difference, 7,500 a second, for as long as the link is busy. Seven and a half thousand a second for 34.6 seconds is a peak depth of about 259,000 messages. At 200 bytes each that is 52 megabytes, which is less memory than your terminal is using. When the turbines finish, arrivals drop back to the farm's ordinary thirty a second, the depth falls at 5,000 minus 30, so 4,970 a second, and 259,000 of them take another 52 seconds.

Thirty five seconds of filling, fifty two of draining, eighty seven all told. Which is the same eighty six seconds of database work from before, plus the trickle that arrived while it drained. That is worth sitting with: the queue did not make Postgres faster. The total work never moved. What moved is who waits, and where. Instead of every request worker on four ingest boxes parked on one farm's connection pool with nineteen farms stacked up behind them, 52 megabytes sat in a broker and everybody else carried on.

Fifty two megabytes, against twenty two minutes of fleetwide blindness. That is the trade, and it is not a close one.

Fresh work waiting behind stale work

There is a catch in that eighty seven seconds, and it is the sort of thing you only find by watching the screen instead of the graph.

The queue is first in, first out. The first messages out of it are the ones the turbines buffered at 10:30, four hours old. So for the first eighty seven seconds after the link comes back, the newest row in Tarrow Ridge's table is not a reading from now. It is a reading from the storm, and it moves forward through the afternoon as the queue drains.

Lesson 003 established that the Galewatch dashboard shows the last reading it holds. So an engineer who opens Tarrow Ridge a few seconds after 14:30 is looking at a reading from half past ten, and then watches four hours of weather replay in about a minute and a half: the wind rises, a turbine feathers, the wind drops, all in fast forward, all of it long over. Every tile is green, every number is hours old, and nothing on the page says so, which is lesson 003's healthy green tile arriving by a completely new route.

The fix is not a bigger queue. It is two of them. A reading from a turbine that is currently connected goes in the live lane; a reading a turbine is replaying out of its own flash goes in the backfill lane; the writer drains live first and backfill with whatever is left. The dashboard is current within seconds, the backfill takes a little longer than eighty seven seconds because it now yields, and the year-long report that lesson 008 said can be cached indefinitely does not care in the slightest what order its rows arrived in.

First in, first out is a policy, not a law. It is a good default precisely because it is the one that needs no thought, and the moment your readers care about recency more than about order, it is the wrong one.

The queue you already had

Galewatch did not install its first queue that October. It installed its second.

The first one is the flash chip inside each turbine, holding four hours of readings on a hill in bad weather, and lesson 007 already told you what it is: for those hours, the turbine holds the only copy of that data in the world. That is a queue. It has a producer, a consumer, a depth, and a drain rate. What it does not have is a graph anybody in the office can look at.

This is the general case, and it is worth being blunt about it. Every system that survives a dependency being unavailable already has a queue somewhere, whether or not anybody chose one. It might be a retry loop, a file on a disk, a pending column in a table, or a customer pressing the button again. The only question is whether it is somewhere you can see it, size it, and drain it on purpose.

Which brings up the dial. A broker that keeps messages in memory and acknowledges instantly is fast and is lying to you about durability. Once the ingest box has answered 200 and the turbine has deleted its copy, the queue holds the only copy of that reading, exactly as the flash chip did, and lesson 016 already priced what that means: durability is a promise somebody has to sell you, and it is sold separately from everything else. If the broker's process dies with 259,000 readings in memory, those readings are gone, and the turbines that produced them threw theirs away on the strength of your 200.

So the queue is a shelf, in lesson 016's sense, and it gets the same test. Marlow's covers went to object storage because nothing on the page that takes money depends on them. Galewatch's readings are the product.

A queue only helps when the caller does not need the answer

Marlow Books, the four person online bookshop that exists only in this course, has an obvious candidate for a queue sitting in the middle of its checkout, and it is the wrong one.

Lesson 015 opened that checkout: read the stock row, charge the card, decrement the stock and insert the order. It also priced the gap between reading the stock and acting on it: two tenths of a millisecond for the read by primary key, about half a millisecond of round trip to reach it, and then 300 milliseconds of somebody else's time for the card. The card is better than ninety nine percent of that gap. Queue it.

Marlow cannot queue it. The customer is standing there, and the answer to "did the card work" is the product. Anything you write down and answer later turns "your order is confirmed" into "we will let you know", which is a business decision wearing an engineering costume.

It is a real option, mind. Lesson 003 named it as one of the two ways to stop the payment provider's 8.8 hours a year from being yours, and plenty of shops take it. Marlow's founder will not, because a four person shop that emails a stranger the next morning to say the card failed has spent more goodwill than the outage would have cost. So the shape that works here is the one lesson 015 already gave for slow third parties: claim the thing in one fast transaction, do the slow unreliable part with nothing locked, confirm in a second transaction.

Here is the rule underneath that. A queue only helps when the caller does not need the answer. Slow work does not qualify. Unimportant work does not qualify. The test is whether anybody upstream is waiting on the outcome to decide what to render.

Which Marlow does have, right next to the card charge. The confirmation email. Nobody has ever refreshed an order page to find out whether an email was sent, and an email provider is a third party with its own bad afternoons. Send it inside the checkout request and that provider's availability multiplies into the shop's, which is the multiplication lesson 003 ran down a ticketing checkout's five component purchase path, not this one: 99.78%, with the payment provider alone eating 45% of the budget. Write the email onto a queue and it multiplies into nothing.

The size of that queue is the part worth keeping. Lesson 009 measured the Christmas peak packing room at about one order every five seconds, which is the order rate too, near enough, so the email queue takes a fifth of a message a second. If the email provider is down for two hours, the queue holds 1,440 messages, which is not a number anybody needs to plan for. On every other day of the year its depth is zero.

That is what most queues are. Empty, almost always, taking a fifth of a message a second, costing nothing, and quietly worth the whole afternoon the far end is down. You are not buying throughput. You are buying room. Lesson 018 takes the rest of this thread, including how the work gets retried and how you know it finished.

The waiting room

Stagefront, the ticketing service where a stadium show goes on sale at exactly 10:00 and oversold seats are a lawsuit, has the other kind of queue, and it holds people rather than messages.

Two hundred thousand of them press the button inside a minute, which is 3,333 a second. Lesson 004 published the rate at which the purchase path converts them: 500 payment calls a second at peak. Call the stadium forty thousand seats, which is what lesson 009's "tens of thousands" comes to for a big show, and the arithmetic moves with that number if you pick a different one.

Forty thousand seats at five hundred a second is eighty seconds.

The on-sale is over at 10:01:20, and a hundred and sixty thousand people were never going to get a ticket. Lesson 006 has already shown you what that minute does to twenty application boxes with nothing in front of them: 75% CPU by 10:00:20, a two second health check with a one second timeout, and five boxes taken out of rotation in thirty seconds by a checker that was measuring a checkout p99 that is over a second on purpose. None of those five were broken.

A queue in front of that changes one thing. The twenty boxes are asked for five hundred a second instead of 3,333, which is a load they can serve with a p99 a health check can live with, and the other 2,833 a second wait in a room. The boxes stop being the thing that breaks.

What the queue does not do is create a single extra seat. It cannot. A hundred and sixty thousand people are going to be disappointed either way, and the only thing left to decide is how they find out.

That is where a queue holding people is genuinely different from a queue holding messages, and the difference is not sentiment. A message does not care that it waited three hundred seconds. A person waiting three hundred seconds is forming an opinion the entire time, and a queue with no visible position is indistinguishable from a hang, which is lesson 004's failure with no status code. So a waiting room owes them two things a message queue never owes anybody: where they are in it, and an honest closing time. When the inventory is gone at 10:01:20, the room should stop admitting and say so, rather than walking a hundred and sixty thousand people to the front one at a time to be told no. Lesson 021 owns saying no gracefully; lesson 041 owns what to do when the queue itself is full.

The memory is the expensive half

A queue is easy to add and permanent to own, and everything that makes it worth having is the same property: it remembers.

That third row is the one nobody sees coming. A message written by the version of the code you deployed on Tuesday will be read by the version you deploy on Friday, so the message format is now an interface between two programs that are never running at the same time, which is a harder promise than any API you have written. Lesson 052 owns it.

The second row has a sharper edge than it looks. A consumer that takes a message, does half the work and then dies has left you a question with no good answer: does the message come back? If it does not, the work vanished. If it does, the work runs twice, and running twice has to be safe, which is lesson 019 and is the reason that lesson exists. Lesson 036 has the delivery guarantees underneath.

Then the ordinary tax. The queue is state. It needs backing up, monitoring, credentials, and capacity, and it is one more dependency whose availability multiplies into yours, which is the price lesson 016 put on every extra shelf.

Monitoring is where I would spend the effort, because the obvious metric is the wrong alert. Depth is what every broker shows you first, and depth on its own means nothing. Tarrow Ridge's 259,000 messages at 14:30:35 was a perfectly healthy system doing exactly its job. Five thousand messages that have not moved in ten minutes is an outage. Same graph, opposite meanings.

What tells them apart is the age of the oldest unprocessed message, because that is the number a human at the other end actually experiences. Age has a unit everybody understands and it maps straight onto the promise you made: the dashboard is eleven seconds behind, the emails are four minutes behind. Alert on age. Graph depth next to it so you can see which way the difference is pointing.

And that matters more than it sounds, because of the last row of the table, which is the real bill for the memory.

Before the queue, a slow database made the ingest API slow, the turbines got timeouts, and the failure announced itself on nineteen dashboards inside a minute. After the queue, a slow database makes the ingest API keep returning 200 in two milliseconds while the data quietly gets older. The queue moves the failure from the caller to the clock. It is lesson 003's stale stock count served in eight milliseconds, and it is lesson 012's argument about replicas in a new setting: a number on a screen can be old because the thing behind it is behind, and nothing on the page has a colour for that.

Which leaves the case a queue cannot fix and is very good at hiding. If the consumer is permanently slower than the producer, the depth graph climbs a straight line forever and the tempting fix is a bigger queue. A queue that never empties is not a buffer. It is a system that is too slow, with a memory bolted on, and the memory is only postponing the conversation about the arithmetic.

That climbing line is the only warning you get, and it is a generous one. It starts long before anybody outside the team notices the data is old.

Recap

A queue is a place to write down that something has to happen. The caller's job changes from doing the work to recording it, and the doer gets to be slower, absent, or briefly on fire without anybody upstream noticing. That is the buffer with a memory.

A burst is not an overload. Tarrow Ridge's 432,000 readings arrived two and a half times faster than they could be stored and still only amounted to eighty six seconds of database work. Before sizing anything for the peak rate, check whether the arrival is finite, because the fixes are completely different.

The queue does not make anything faster. Depth grows at arrival minus service, drains at service minus arrival, and the total work is whatever it always was. All a queue changes is who waits and where, and the answer to "where" should not be inside a request handler holding a connection.

A queue only helps when the caller does not need the answer. Marlow cannot queue the card charge, because the answer is the product. It can queue the confirmation email, and that one change takes an email provider's bad afternoon out of the shop's availability arithmetic for a fifth of a message a second.

First in, first out is a policy, not a law. Drain a four hour backlog in order and the newest reading in your database stays hours old the whole time the queue is emptying. Two lanes and a rule about which drains first cost an afternoon and fix it.

Alert on the age of the oldest message, not on depth. A quarter of a million messages can be healthy and five thousand can be an outage; only age knows which. And watch for the graph that climbs forever, because a queue that never empties is not a buffer, it is a system that is too slow with a memory bolted on.

The queue moves the failure from the caller to the clock. Before, a jammed writer broke loudly upstream. After, every instrument reads 200 and two milliseconds while the data gets old. That silence is what the memory costs.

Check your understanding

1. A second Galewatch farm, also sixty turbines, loses its link in the same storm, and both replay at 14:30. Work out the peak queue depth and the time to catch up, remembering which parts of the path the two farms share and which they do not, then say which of those two numbers belongs in a capacity plan and which belongs in an alert.

2. Marlow's founder proposes putting the checkout's stock decrement on a queue, so that checkout returns in two milliseconds and a consumer does the database work. Describe the first Christmas Eve this goes wrong, using lesson 015's arithmetic for the oversell.

3. Stagefront's waiting room admits buyers at five hundred a second and a colleague suggests raising that to a thousand, on the grounds that the queue is just throttling a system that could go faster. Say what you would measure before agreeing, and what you expect lesson 006's health checks to do if the colleague is wrong.

4. Your queue's depth has been flat at about eight thousand messages for three days and nobody has complained. Give two completely different explanations that both fit that graph, and say which single measurement separates them.

5. Galewatch's broker acknowledges a reading in two milliseconds by holding it in memory and writing it to disk shortly afterwards. The turbine deletes its local copy on that acknowledgement. Say what is now the only copy of a reading and for how long, and describe the smallest change that makes the pair safe.

Next lesson

018 Asynchronous Work: Doing It Later Without Forgetting. Today put the work on a shelf; next lesson is about everything that has to be true for it to come back off, including the job that has to run exactly once, the failure nobody is watching for, and how a system tells you that work it accepted three hours ago has still not happened.