Marlow Books is a four person online bookshop that exists only in this course, and at eleven seconds past six on the morning of Tuesday 13 October, its newest improvement put the wrong author on fourteen hundred book pages and gave each one a twenty four hour lease.
The improvement was three months old and it was the right thing to build. Lesson 009 is the April morning this shop sold forty copies of a novel it had twelve of, and the cause was never a bug in the invalidation code. It was a write path the invalidation code had never heard of: the distributor's overnight import, written before the shop owned a cache. Lesson 009 counted every piece of code that touches a book row and found five. Three called the cache. Two did not.
In July the founder went back to that list. The import got the cache treatment, and not the cheap version: instead of deleting each changed title's entry and letting the next reader rebuild it, it renders the page shell itself and writes it straight into Redis. No manufactured miss, and a corrected blurb live in seconds rather than in a day. It ran every morning for three months and did exactly that.
Then the file was wrong.
The same 06:00 feed that lesson 045 blames for three thousand changed search documents a day arrived with one publisher's block shifted by a column, so for fourteen hundred titles the author field held the series name. Nothing validates that file, which is a conversation the shop has been not having for six years. At 06:00:04 the rows were written. At 06:00:11 the shells were.
At 08:17 a customer emailed to ask why the novel she had bought on Sunday was now by somebody called The Thornfield Chronicles. The founder found the shifted column in seven minutes and put the authors back at 08:24 with one UPDATE in psql, off the previous night's copy of the file.
The pages did not change.
Nothing in that psql session has ever called Redis, and lesson 009 counted it as the fifth write path for exactly that reason: no code review reaches a founder's terminal at twenty four minutes past eight in the morning. The import had already written fourteen hundred shells with twenty four hours on each, and the correction went into the rows behind them, where nothing would read it until the next morning's import came round. Twenty one and a half hours of a wrong author over a row that had been right since half past eight.
He flushed the cache at 08:31 and the shop was correct inside a minute. Lesson 009 called that button a load bearing decision wearing a friendly label, and it was free that morning only because twenty past eight on an ordinary morning is about forty requests a second, against the two hundred and sixty a second lesson 005 measured for the pair with no cache at all. The same click at eleven on the Tuesday before Christmas is an outage.
The invalidation was not missed. The cache was written by the one piece of code in the building that definitely knew the value had changed, immediately, deliberately, with the freshest thing it could see. That is write-through working.
Who writes the store, and when
Three patterns, three answers to one question: who puts the value into the slow durable thing, and at what moment relative to the client getting its answer.
Cache-aside, also called lazy loading, is what almost everybody has. The application owns both stores and treats them as separate: a read looks in the cache, and on a miss goes to the database, uses the answer and writes it back. A write goes to the database, and the entry is dealt with afterwards by deleting it or leaving it to expire. The cache learns by missing.
Write-through puts the cache in the write path. The write goes to the cache, and the cache writes it on to the store before the call returns. The cache is never behind the store, and every write pays the store's latency plus the cache's.
In the textbook drawing a cache library does that forwarding, and almost nobody runs it that way. What teams call write-through is the application writing the row and then the entry, in that order, seconds apart, which is exactly what Marlow's import does at 06:00:04 and 06:00:11. Close enough on the property that matters, with one failure mode the drawing does not have: if the second write loses its race or its connection, the old entry sits there on its original lease with nothing left to retire it.
Write-behind, also called write-back, returns the moment the cache has it, and the store is updated later: on a timer, when the buffer fills, or when something gets round to it. The writer gets its latency back and the durable copy lags.
| Pattern | Who writes the store | When |
|---|---|---|
| Cache-aside | the application, around the cache | its own business |
| Write-through | the cache | before the write returns |
| Write-behind | the cache | later, and "later" is a design |
Now the thing the textbook version gets wrong, which matters because it changes what you are choosing between. Write-through is not an alternative to cache-aside. The read path in a write-through system is still a cache-aside read: look, miss, fetch, fill. What write-through replaces is the line after the write, the one that deletes the key. Lesson 009 filed it in precisely that place, as the second of three ways to retire an entry, and left the word there with a note saying today would own it.
So the real question is narrower than three patterns. It is what you do about the entry you just made wrong, and whether anything is allowed to sit in the cache that is not yet in the store.
You will also meet read-through, where the cache fetches on a miss and your code never sees the database. That is packaging rather than design, and its price is that the handler can no longer see which reads are expensive.
The reader writes the cache, and most of cache-aside follows
In cache-aside, entries exist because somebody asked for something. Nobody chose the contents, so the cache is a picture of the traffic, which is lesson 008's whole explanation for a 94 percent hit rate over 1.2 million titles. The good news in that is the half nobody says: a cache-aside cache cannot be wrong about something nobody has read. The lie needs a reader to create it.
The bad news is the herd, and the herd belongs to cache-aside specifically. Lesson 008 has the Tuesday: one key expires, every request that wanted it finds the same absence in the same moment, and none of them can see the others already on their way. Ten requests a second to each box against a nine hundred millisecond aggregation gave nine copies of one query, then thirty nine, then every worker the box had.
A write-through cache does not have that failure on the key it just wrote, because the writer filled it. It has the identical failure on everything it evicted, everything it lost in a restart and everything whose TTL ran out, because the read path is unchanged.
The fix everybody names is single flight: the first miss takes a lock, computes, writes and releases, and the rest wait for it instead of starting their own. What nobody says is where the lock lives, and the arithmetic turns on that.
An in-process lock, a mutex in the application, is the version you get in four lines. Lesson 006 put four application processes of twelve workers each on every Marlow box, so a box's ten requests a second arrive at four separate processes at two and a half a second each. During the nine hundred millisecond rebuild each process sees two or three misses and starts one aggregation. The herd of nine becomes four.
Four, not one. That may well be enough, because what took the shop down was nine concurrent copies of one aggregation fighting over the same memory and stretching the rebuild to seconds, and four might not stretch it. If you want one, the lock has to be visible to all four processes on both boxes, which makes it a key in Redis with an expiry: lesson 035's subject exactly, with lesson 034's clock underneath it.
This is the one place in these lessons so far where 035's sharp edges are blunt. Every horror in that lesson comes from a holder who lost the lock and carried on doing something, and here the something is rebuilding a cache entry, which sits in lesson 019's free pile as an absolute value written again. Two processes writing the same bytes have wasted work and changed nothing, so the worst case of a broken cache lock is the stampede you had before you added it. Lose a lock around a payout and you have paid twice. Two rebuilds can still land out of order, which is lesson 009's race: the lock bounds your load, not your correctness.
Then the other fix, and these two get bundled into one sentence constantly. Single flight protects the origin and does nothing for the customer, who now waits nine hundred milliseconds rather than running their own nine hundred millisecond query. What protects the customer is handing back the copy that just expired and refreshing behind them, which lesson 022 has in its mature form as stale-while-revalidate. In your own cache you build it with two expiry times per entry: a soft one, after which you serve the value and start a refresh, and a hard one, after which you stop believing it.
Decide which victim you are saving. Most teams write the lock, declare the stampede solved, and keep a p99 that is still the rebuild.
Write-through renews the lease
Start with the objection everybody raises, because it is wrong at this scale and knowing that frees you to see the real one.
Lesson 002's ladder puts a round trip inside one data centre at half a millisecond. Lesson 030 walked Marlow's checkout end to end, and the card call in the middle of it is lesson 015's three hundred milliseconds. Adding half a millisecond to that write is one part in six hundred, which is not a number anybody is going to defend in a planning meeting.
Where it bites is distance. The same ladder puts a round trip across the world at two hundred milliseconds. Put the cache in one region, leave the store and the code that writes to it in another, and every write now waits a fifth of a second on the cache, which is the one machine in that path the writer needs no answer from. Lesson 050 owns that. Inside one data centre, latency is not why you are refusing write-through.
Two real bills. The first is lesson 009's: the writer has to be able to build the cached value. Marlow's value is twelve kilobytes of rendered page, so write-through meant putting the page renderer inside the overnight import, which is most of why a fix that reads like one line took from April to July. A value that is one row needs nothing; an assembled document drags the assembler into every write path that wants to keep it fresh.
The second bill is the one this lesson is named for, and I have not seen it written down.
Every write-through is a lease renewal. A delete lets the TTL fire on schedule. A write-through resets it to full, which is fine when the value is right and is the whole of October's story when it is not. Lesson 009 called the TTL the backstop and said that raising it is a statement about how wrong you are willing to be. Write-through is sneakier than raising it: the number in the config file never moves, and the backstop stops arriving on the entries that need it.
Do the counterfactual, because it is the part that surprised me. Suppose the import had never been fixed and sat there in April's state, writing rows and ignoring Redis. The bad author goes into the rows at 06:00:04, and every cached shell written before that moment still holds last night's correct author until its own twenty four hours run out. The correction lands at 08:24. So the only entries that could ever pick up the bad value are the ones whose lease happened to expire inside those two hours and twenty four minutes. Spread an entry's write time roughly evenly around the clock, which is what fourteen hundred titles first missed at fourteen hundred arbitrary moments gives you, and that is a tenth of them.
A hundred and forty instead of fourteen hundred. And the hundred and forty are wrong for just as long. Nothing invalidates them either, so each one carries the bad author for its own full day, same as the fourteen hundred did. The fix did not make the wrongness last longer. It made it ten times wider, in seven seconds, at six in the morning.
It bought one thing, and the one thing is what ended the morning. Fourteen hundred wrong pages produce a customer email by 08:17. A hundred and forty scattered through a catalogue of 1.2 million might not produce one for a week.
The rule is not "do not write through". It is that a cache with write-through on some paths and nothing on others is worse than a cache with nothing on any of them, because the paths that have it renew the lease on whatever the paths without it got wrong. Partial coverage is the dangerous state, and partial coverage is where every real system lives, because the path you forget is always the oldest script and the human at a terminal.
Lesson 009 pointed at the way out, and today is when it stops sounding like architecture for its own sake. Stop asking writers to remember: read the database's own write log and invalidate from that, so psql is covered and so is next year's script, and lessons 037 and 040 have the machinery.
Be clear about what that buys, though. It is not write-through, which means the cache has the value before the write returns. A consumer reading a log is behind by however long the log takes, and lesson 012 drew that distribution for this shop: a median of six milliseconds, and a second every morning while the import runs, which is precisely the hour a feed is rewriting fourteen hundred titles. A write path can forget; a change stream cannot; the change stream is never quite now.
One more trap in the same family, and it is lesson 044's no partial write arriving indoors. If your cached value is a composite record and a writer patches one field, it has renewed the lease on every field it did not look at, including the one another path got wrong an hour ago. Marlow is accidentally safe, because a rendered page cannot be patched: the only write-through available was to rebuild it whole from the row, and the expensive option was the correct one.
Write-behind, and which side of the durability line the buffer sits on
Write-behind is usually introduced as a trade of durability for latency, which is true enough to be useless. The version that decides designs is that a write-behind buffer sells whatever its position lets it sell, and its position is defined by one line: the edge of the durable store.
Lesson 045 handed one over by name. Postgres GIN keeps a small unsorted pending list inside the index, fastupdate on by default and gin_pending_list_limit at four megabytes, around a hundred and fifty thousand entries, which get sorted into the index proper in bulk later. The writer gets its latency back and every searcher pays, because a query has to scan that list too, and on a replica you cannot even clean it.
What that buffer cannot do is lose anything. It lives inside the database, under the write ahead log, so a crash replays it like any other page. Inside the durability boundary the only currency it has is read work, and that is the entire reason it ships on by default on millions of installations.
Move the same buffer four inches outward, into your application's heap or into Redis in front of Postgres, and it is a different animal with the same name. Now there is an interval in which the only copy of a committed-looking write is in memory on one machine, and the arithmetic is one line:
what you lose = flush interval x write rate
Pick the flush interval and you have picked how much data you are willing to lose, which is not what anybody says out loud when they set it to five seconds because five felt small.
A size bound behaves worse than a time bound, in a direction people find backwards. A buffer that flushes at four megabytes is exposed for four megabytes divided by your write rate, so the window stretches as traffic falls. Your longest exposure is your quietest hour, which is also the hour when a maintenance restart looks free.
Lesson 044's no partial write deserves one more turn here, and it is a sharp one. In front of an object store there is no partial write, so a flush cannot be "this field": the unit is the whole object, so the buffer holds the whole object until it goes and five coalesced revisions of one object are one object. In front of a database the flush can be a single column. The store's write granularity decides whether your buffer coalesces for free or has to carry everything, and nobody checks that before choosing the buffer.
A cache answers "full" by throwing data away
Here is the failure that kills write-behind caches in production, and it is not the one in the diagrams.
A cache's job, when it runs out of room, is to discard something. Lesson 008 has what that looks like when a crawler writes twenty four thousand entries into a two hundred and fifty six megabyte cache over ten minutes: LRU reads a sequential walk as wildly popular new content, nothing has aged, and the shop's bestsellers go out one at a time all week. It cost hit rate and nothing else, because Marlow's cache holds no only copies.
Make that cache write-behind and the same eviction is a lost write. An LRU policy has no concept of a dirty entry. Redis evicts under maxmemory-policy by recency or frequency, never by whether the store has seen the value, and the one setting that refuses to discard anything is noeviction, which answers a write with an error instead. Read that twice, because it is lesson 041 in a configuration value: noeviction is backpressure, and every other policy is silent loss under memory pressure.
Which gives the distinction to carry out of here. A queue bounds its depth and makes somebody wait. A cache bounds its memory and throws something away. Same buffer, same producer, opposite answer to being full, and the word "cache" is what hides it.
Galewatch, which collects a reading every two seconds from each of its wind turbines, has the clearest write-behind buffer in these lessons that holds original data and gets away with it, and it gets away with it by being a queue. Lesson 017 sat through four hours of Tarrow Ridge, a farm of sixty turbines at the end of a long access road, losing its link on a Wednesday morning in October, and did the division: thirty readings a second for four hours is four hundred and thirty two thousand held in flash on the hillside, and all of them came back. The same lesson named the mechanism in a clause, that the turbine deletes its local copy once the ingest box has answered 200. The buffer discards on an acknowledgement, not on a clock and not on a memory ceiling, and that is the difference between a buffer you can trust with the only copy and one you cannot.
The ordering failure is the other half of write-behind, and it reaches production with a passing test suite. Two writers, one key, a buffer in front of one of them, and the flush lands after the later write. Lesson 015 has the cure, and this is the half of that lesson the oversell story left on the floor: a relative statement, SET views = views + 4 rather than SET views = 11731, survives arriving late because it never claimed to know the old value. A read-modify-write flush is a lost update with a delay fitted. Then the cure brings back 015's other bug, read backwards: correct a count to zero at 09:02 and flush + 4 at 09:05, and the relative statement applies perfectly to a correction it was never told about.
A third thing goes quietly missing. A buffer that collapses five updates into one store write has made the store's history lossy, which nobody notices while everything downstream reads current state. The moment something reads changes instead, which is lesson 037's log and lesson 040's outbox, four of those five events never happened.
Who could prove you lost one
What may a write-behind cache hold, then? Lesson 007 asked the question that settles it, of any shared store: what is in there that exists nowhere else.
If the buffer holds derived data, something you can recompute from a source you still own, losing it costs time and nothing else. Lesson 008's answer for Marlow's cache was exactly that: the data safe to lose and the capacity not. An index, a rendered page, a ranking that is a group by over the orders table. Write-behind over derived data is a slow rebuild wearing a frightening name, and you can be relaxed about it.
If the buffer holds original data, something that happened and was written down nowhere else first, it is a database with no durability, and the only question worth asking is who could prove you lost one.
That has an answer per value and the answers are not close. Twelve signed first editions can be counted in the back room, and in lesson 015 the founder counted them twice before typing the number in. A card charge has a second copy at the provider, which is what lesson 027's sweeper is for. A turbine reading has nothing on the other side at all, and lesson 018 put it better than I can: it is the only surviving record of what a machine on a hillside was doing at half past three yesterday afternoon, and that moment is not coming back. Nobody will ever email Galewatch to say a reading is missing, which makes it the most dangerous of the three rather than the safest.
Stagefront, the ticketing service where two hundred thousand people press the same button at ten in the morning, is where this stops being a judgment call. A seat claim is original data whose audit is a solicitor's letter, and lessons 015 and 035 settled it: a committed row in the database that owns the seat, in one short transaction, with no lock and no buffer near it. No flush interval is small enough there, because the acceptable number is zero and a buffer exists to not be zero.
The received advice is to use write-behind for non-critical data, which is the same rule phrased so it cannot be used, because every team believes its data is critical and no team can rank its own tables. Who could prove we lost one is a question a room of engineers can answer in four minutes, per value, with names.
Recap
Three patterns, one question: who writes the store and when. Cache-aside has the application write around the cache and the cache learn by missing. Write-through puts the cache in the write path. Write-behind returns when the cache has it and settles up later.
Write-through replaces the delete, not cache-aside. The read path is a cache-aside read in all of them, which is why write-through removes only the herd caused by your own invalidation and leaves the ones caused by an eviction or an expiry where they were.
The reader writes the cache. Nobody chose the contents, so the cache is a picture of the traffic, and that is the whole reason 1.2 million titles live behind a 94 percent hit rate on a few thousand entries.
Single flight saves the origin and stale saves the customer. An in-process lock makes the herd equal to your process count, four per box at Marlow and not one. Serving the expired copy while you refresh is a different fix for a different victim, and it wants two expiry times per entry.
Every write-through is a lease renewal. It resets the TTL to full without changing the number in the config file. October's bad value outlived its own row's correction by twenty one and a half hours, and the design that ignored the cache entirely would have been wrong for just as long on a tenth as many pages.
Partial coverage is the dangerous state. A write path can forget, a change stream off the write log cannot, and the change stream is always a little behind. Nothing is both immediate and unforgettable, so choose which you can defend.
Which side of the durability line the buffer sits on decides what it can sell. Inside a durable store, under the write ahead log, it can only sell read work, which is why GIN's pending list ships on by default. Outside one it sells durability, at flush interval times write rate, and a size bound hands you your longest exposure in your quietest hour.
A cache answers "full" by throwing data away; a queue answers by saying no. LRU has no notion of dirty, and the only eviction policy that will not lose a write is the one that refuses the write. Galewatch's hillside flash survives four hours of darkness because it discards on an acknowledgement.
Who could prove you lost one. Derived data in a buffer costs time when it dies. Original data makes the buffer a database with no durability, and a reading from a hillside is more dangerous than a card charge because nobody will ever write in to report it missing.
Check your understanding
Marlow's founder, having had October's morning, wants to take the write-through out of the import and go back to a plain delete. Say what that costs on an ordinary morning using lesson 008's figures, what it costs in Christmas week, and whether you would do it or something else.
You inherit a write-through cache in front of a store in another region, and writes are slow. A colleague proposes write-behind with a two second flush, on the grounds that two seconds is nothing. Say what you need to know about the values in that cache first, and name the one answer that ends the conversation.
A team adds single flight to a cache-aside read path, measures the database's load falling as expected, and reports the stampede fixed. Three weeks later the p99 on that endpoint has not moved. Say what they measured, what they did not, and what you would ship next.
Galewatch's engineers want the live panel fed from a write-behind cache: the ingest box writes the reading into Redis, answers the turbine, and flushes to Postgres every five seconds. Work out what five seconds is worth in readings, say which earlier lesson's guarantee it breaks, and what you would change on the turbine's side if it shipped anyway.
Somebody at Stagefront points out that a seat map rebuild is expensive and proposes write-through on the seat rows so the map is never stale. Argue for it as well as you can, then give the number from lesson 009 that kills it, and say what that number implies about the ten second TTL the map has today.
Your cache holds a composite customer record, and two services write through to it: one updates the address, one the marketing preferences, each writing the whole record. Describe the bug, say what would make it frequent rather than rare, and give two fixes that differ in what they cost the reader.
Next lesson
047 Multi-Level Caches and Hot Keys. Today treated the cache as one box with one policy; next lesson stacks them, in the process and in the shared store and at the edge, then puts one key in front of all of them that everybody wants in the same instant, which is where today's eviction arithmetic stops being an average.