Marlow Books, the four person online bookshop that exists only in this course, went dark at ten to four in the morning on the last Saturday in November because somebody backed it up.
The somebody was cron. Every night at 03:00 a pg_dump runs on box A, gzips itself into /var/backups, and the founder keeps the last two: write the new one, then delete the oldest, which is the correct order and the reason the disk has to have room for three. That dump had been 46 GB for a while, small against the database because a dump carries no indexes and prose compresses well. On the last Saturday in November it ran out of room 12 GB in, the next write-ahead log record had nowhere to go, and Postgres did the only honest thing available to it. It panicked and shut down rather than pretend the write had landed.
At ten to four on a Saturday there is nobody to notice. Lesson 003 put 95 minutes on Marlow's time to hear about an outage from a customer email; the customer also has to wake up, so the email arrived at 08:24. Deleting the half written dump bought back the 12 gigabytes, Postgres started, and the shop was serving by 08:40. Four hours and fifty minutes of nothing, on a volume that had sat at 97% for weeks with no alert on it, three weeks before the week that makes the year.
Then the founder ran du, which is the part of this story worth keeping.
box A, Saturday 09:10
216G /var/marlow/covers 1.2 M JPEGs at 180 KB each
180G /var/lib/postgresql the database, six years of it
92G /var/backups two nightly dumps at 46G
----
488G of a 500G volume
Six years of trading, which you can date from lesson 004: the certificate that expired one Sunday in March had been renewed by hand every year since launch, six times. Six years of orders, reviews, prices and sessions came to 180 gigabytes. The largest single thing on the shop's database disk was none of it. It was image files.
Lesson 001 counted them on day one: 1.2 million covers at about 180 KB, on the same volume as the database. Lesson 005 watched them break when the second box arrived and uploads started landing on whichever machine served the form. Lesson 007 named the fix, said 016 and 044 owned it, and moved on. Nobody moved them, so they were still there, and they took the database down by being next to it.
And not one byte of those 216 gigabytes was in the backup that filled the disk. pg_dump dumps a database. The covers are files; the database has never heard of them. Nor does 011's streaming replica carry them, because physical replication ships the write-ahead log and the covers were never in it. So the shop holds four copies of every order it has ever taken, counting box A, the replica and the two dumps, and exactly one copy of 1.2 million images, on the disk that just filled up.
That is not a storage problem. It is a shelving problem, and the founder had never once asked which shelf anything belonged on.
A store is a bundle of promises
"Which database should we use" is a bad question, and it is bad in a specific way: it has an answer, the answer is whatever the person asking used last, and nothing about the data ever enters the conversation.
Every store makes a small set of promises, charges you for them whether you use them or not, and refuses the rest. Name the promises your data actually needs and the shelf usually picks itself.
Five of them do most of the work. The smallest thing you can read and write in one go, and whether it matches the thing your application actually thinks about. Who wins when two writers race, and who finds out, which is all of lesson 015 restated as a shopping question. What happens to the bytes when the machine dies, and how many copies then exist in how many places. Which questions you can ask without having planned for them a year earlier. And what the bytes cost, not to hold but to write, replicate, back up and read back.
Notice that "how fast is it" is not on that list. Speed is a number you can usually buy. Promises are not, and the ones you gave up are the ones you find out about at four in the morning.
Call it the shelf question: not which store is best, but which promises this particular data needs, priced against what the store will charge you for all the others. A shop's orders and a shop's cover images have almost nothing in common except that the founder happened to own one disk.
The relational shelf, and what it charges
Start with what Marlow actually gets from Postgres, because most of it is invisible until it is gone.
Rows with a declared shape, so stock is an integer everywhere forever. Indexes, which lesson 010 spent a Friday afternoon proving are the difference between 0.38 milliseconds and 3.1 seconds. Joins, so the buyers' reorder view can ask about books and order lines in one question. Lesson 015's transaction, a boundary drawn around a decision, which is the only reason the shop can sell the last copy of a book to exactly one person.
And one more that engineers undervalue until they lose it: the schema is the only code every writer runs.
Lesson 009 counted Marlow's write paths one April. Checkout. The price import. The admin form. The distributor's overnight CSV, written in year one, which has never called Redis and never will. And the founder's own psql session at eleven at night. Five writers, four of them written by the same person in different years and different moods. When 015 finally added CHECK (stock >= 0), it became true for all five, including the one that runs at 06:00 and knows nothing about anything. You cannot put a rule in the application and have it hold for the CSV. You can put it in the table and have it hold for the founder.
That is the promise worth paying for. Now the bill.
Every byte you put in a row gets written down five times on this shop's hardware. Once into an 8 KB heap page on box A's disk. Again into the write-ahead log, because that is how Postgres makes a commit durable. A third and fourth time on 011's replica, which receives the log and applies it to its own pages. And a fifth into the nightly dump. Then it has to be read back through whatever restores it on the worst day of the founder's year, and index entries ride along on top for every indexed column.
So run the thought the founder nearly had that Saturday: put the covers in Postgres. It is one column, it solves the box A against box B problem in an afternoon, and both boxes can then read any cover. Consider what it costs.
A 180 KB value does not sit in the row. Postgres moves anything over about 2 KB out of line into a side table, in chunks of roughly 2,000 bytes, so each cover becomes about ninety rows in a companion table with its own index. Times 1.2 million covers, that is about 108 million extra rows, and at roughly the twenty eight bytes an entry that 010 used for a different index, the index that finds those chunks is about 3 GB on its own. Postgres will also try to compress the value before it chunks it, and give up, because a JPEG is already compressed. This is true of your dump as well, which is the punchline: 46 GB plus 216 GB of incompressible image data is a 262 GB dump, and three of those on a 500 GB volume is 786 GB. The backup that broke the shop in November becomes arithmetically impossible.
Reads are the subtler half, and I want to be careful because the obvious version of this argument is overstated. If you stream a cover out of Postgres straight to the customer's socket, you hold a pooled database connection for as long as the transfer takes. A cover is 180 KB, call it 1.44 megabits; on a 4 Mbps phone that is 0.36 seconds. Lesson 008 put Christmas week at 384 book page requests a second, and 009's one year expires header means most visitors already have the image, so call it a tenth of them fetching one, 38 a second. That tenth is a guess and not a measurement, and every number in the rest of this section moves with it. Lesson 002's arithmetic gives 38 x 0.36, about 14 connections held open at all times, out of the 96 that 010 says the fleet reserves of Postgres's 100. Fourteen percent of the shop's entire connection budget, pushing JPEGs at the speed of somebody's phone.
The tail is where it gets funny. Put a tenth of those customers on a bad link at 400 kbps and each of them holds its connection for 3.6 seconds instead of 0.36. The 34 fast ones now hold 12 between them; the 3.8 slow ones hold 3.8 x 3.6, which is 14. A tenth of the customers cost more than the other nine tenths put together, the total goes from 14 to 26, and more than a quarter of the connection budget is gone. That is 002's tail-becomes-typical turning up where nobody looks for it.
You can dodge all of that by buffering the whole cover in the application and releasing the connection first, and the cost is then 38 x 180 KB, about 7 MB of memory, which is nothing. Streaming is simply the code people write, because it is the obvious code. Buffering fixes the connection problem and leaves the write, replicate, dump and restore bill exactly where it was.
None of which means Marlow has outgrown Postgres, and a reader could take this section that way. Lesson 013 argued at length that this shop should not be sharded, and that a working set fitting in memory is an indexing problem or a caching problem wearing a scary hat. The orders, the stock, the sessions and the reviews are exactly where they belong. It is the pictures that are on the wrong shelf.
What "NoSQL" actually names
The name is a negation, which is how you can tell it was chosen for a conference rather than for a taxonomy. It covers products with less in common with each other than any of them has with Postgres. Worth knowing, then, as three shapes rather than one word.
A key-value store is a dictionary you cannot ask questions of. You hand it an exact name, it hands back the bytes, in well under a millisecond. Marlow has had one since February: 008's page cache, now a small managed Redis. It is very good at the one thing and completely useless for "all the titles from publisher 41", which is why 010's story was about an index and not about a cache. Anything you did not put in the key, you cannot ask.
A document store keeps a whole object together, nested, and lets you read and write all of it in one go. The trade is that the object is also the unit of everything else. Every document store guarantees that a single document is written atomically. Several of them added transactions across documents later, and their own documentation still tells you to design so that you rarely need them.
The unit of atomicity is the unit of design.
If Marlow's stock count lives inside the book document, 015's UPDATE books SET stock = stock - 1 WHERE isbn = ? AND stock > 0 has a direct equivalent and works, because the decision fits in one document. The day the founder wants "decrement stock and add loyalty points and mark the gift voucher used, all or nothing", the shape of the store is the shape of the problem. In Postgres that is a transaction someone writes in an afternoon. In a store whose atom is a document, it is either a redesign of the documents or lesson 039's sagas, and 039 is thirty lessons away for good reasons.
A wide-column store is the third shape: rows grouped into partitions, ordered within a partition, built to take writes faster than one machine can. That is the shape at Galewatch, which collects a reading from each of nine hundred wind turbines every two seconds. Those 450 rows a second are written once, never updated, and read back as time ranges. Look at what that data needs from the list above: no cross-row decision, no join, no constraint anybody could violate, because nothing is ever changed. Lessons 013 and 014 put those readings on eighteen Postgres shards, and neither lesson ever argued the store itself: what decided it was the queries the engineers ask and the tooling the team already has, not the promises. Galewatch had a real case for a different shelf and paid for Postgres in a different currency. Lesson 043 owns the engine underneath that difference and 088 owns time series properly.
The history helps here too. People reached for these products around 2010 mostly because a single relational box could not be cut in half and these could cut themselves. Lesson 013 supplied the other half of that story: Foursquare in October 2010, on MongoDB sharded across two machines by user, where one shard grew until its working set no longer fitted in that machine's memory and the site went down for about eleven hours on the Monday, and again on the Tuesday while they were still repairing it. Having the product shard for you is not free either.
And "schemaless" deserves one sentence of scepticism. There is always a schema. The only question is whether it lives in a table definition, where it applies to the CSV import at 06:00, or in the application, where it is really one schema per version of the code that ever wrote a document, all of them still in the store. Lesson 052 owns the migration either way. I would rather have the migration.
| Shelf | What it promises | What it refuses | Marlow's use |
|---|---|---|---|
| Relational | a rule every writer obeys, 015's boundary | a cheap home for 180 KB of bytes | orders, stock, sessions, reviews |
| Key-value | one value by its exact name, in under a millisecond | anything you did not put in the key | 008's page cache |
| Document | the whole aggregate in one read, one atomic write | a decision spanning two documents | none, and that is correct |
| Object storage | durability for anything file shaped | queries, joins, partial updates | 1.2 M covers, from December |
The blob shelf, and the row that outlives the object
Object storage is the plainest shelf in the building. You give it a key, which is a string, and a blob of bytes. You get them back by that key over HTTP. There are no partial updates, so changing one pixel means writing the whole object again. The namespace is flat: the slashes in covers/9780140449136.jpg are a convention you and your tooling agree on, nothing more. Listing is paginated at a thousand keys a call, so finding out what you have means 1,200 round trips for Marlow's covers, which is a thing you do once.
In exchange it makes the one promise no disk on box A can. S3's published numbers, which lesson 003 already used, are eleven nines of durability against a 99.9% availability SLA. Run the first half against Marlow: 1.2 million objects times 10^-11 is 1.2 x 10^-5 expected losses a year, one cover about every 83,000 years. Then the second half, which 003 already worked out in another context: 99.9% is up to 8.8 hours a year of not being able to reach them. On 28 February 2017 that was one afternoon for a large part of the internet, when a mistyped playbook command in S3's us-east-1 region removed far more capacity than intended for hours.
Durability and availability are different promises, they are sold separately, and the store that will keep your bytes for eighty three thousand years is also the one that will be unreachable for nine hours next year. That is a fine trade for a book cover and a catastrophic one for the row that decides whether the book is in stock.
So the covers move, and the database keeps a row about each one.
create table covers (
isbn text primary key references books(isbn),
object_key text not null,
bytes int not null,
content_type text not null,
sha256 text not null,
uploaded_at timestamptz not null default now()
);
Lesson 007 argued for exactly this split and handed the mechanism here, so give it a name: the row that outlives the object. The pointer lives on the shelf that can enforce rules, and the bytes live on the shelf that is cheap and durable. The buyers' upload form writes both, the book page reads the row and renders a URL pointing at the object store, 022's CDN will later sit in front of that URL, and box A's disk drops 216 gigabytes overnight.
Now the part that gets shipped wrong, because two stores can disagree in two directions and they are not symmetric.
row, then object object, then row
------------------- -------------------
insert covers row PUT the bytes
PUT the bytes <-- fails insert covers row <-- fails
= page promises a = 180 KB nobody
picture, 404s will ever ask for
A row with no object is a broken promise: the page says there is a cover, the browser asks, and a customer sees a hole. An object with no row is 180 KB of litter you pay for monthly, forever, and which nothing will ever notice. One of those is a bug. The other is a thousand orphans adding up to 180 megabytes, which at the prices below is half a cent a month.
So the ordering rule is: write the object first, commit the row second. The sweeper that eventually deletes orphans needs a grace period, or it will delete the bytes of an upload whose row is still in flight; a day is plenty, and Marlow runs it monthly.
Deletes run the same way backwards. Delete the row, leave the object. There is no rush, and 009 already proved there is no point: covers are served with a one year expires header, so copies of that image are sitting in strangers' browsers and no purge on earth reaches them. When a publisher sends a new cover you write a new key, covers/9780140449136-b41f9c.jpg, and change the row. Which is 009's rename trick, except that now the rename is an insert into a table instead of a file operation on a disk that only one of your two boxes has.
The bill, because this is where people guess wrong. Object storage costs about two and a half cents a gigabyte a month at the standard class, so 216 GB is $5.40. Requests are about forty cents a million. Egress is the one that matters: around ten cents a gigabyte.
Price it at 001's ordinary forty requests a second rather than Christmas, because a bill is a month long and Christmas is a week. Borrowing 006's Christmas mix for want of an ordinary one, that is 38 book pages a second, a tenth of them fetching an image, so about 4 cover fetches a second: 10.4 million requests a month, $4.16, and 1,872 GB of egress, about $187. Five dollars to keep them, four dollars to answer the requests, one hundred and eighty seven to push the bytes down the wire.
For file shaped data the storage is never the bill. And Marlow already pays that egress today, because the same bytes leave box A's network card. Moving the covers does not create the cost, it makes it visible and puts it somewhere an edge cache can eat it, which is 022's job and is easy here, because an immutable file with a hashed name is the most cacheable object in computing.
Three systems, six shelves
Count Marlow's shelves after December. Postgres for orders, stock, sessions, reviews and now the cover rows. Redis for 008's page cache. Object storage for 216 gigabytes of JPEGs. Three stores in a shop run by four people, and the founder made a conscious decision about exactly none of them: the cache arrived because a page was slow, and the covers are moving because a backup filled a disk.
That is the normal case, not a failure of discipline. Stagefront, the ticketing service where a stadium show goes on sale at 10:00 and oversold seats are a lawsuit, has the same split for sharper reasons: seats and holds live in a relational database because 015's transaction is the product, while 008's seat map is a cached value with a ten second TTL because it changes faster than anyone can read it. Galewatch has its eighteen sharded Postgres machines and a real case for something else underneath them. Six shelves across the three systems, and the only pair anybody argued about in advance is Stagefront's, because there the cost of the wrong shelf is a lawsuit.
The price of each extra shelf is worth naming, and it is rarely the monthly bill. What you buy is a consistency problem you now own in application code, forever: the row and the object, the cache and the row, 009's invalidation problem in a new costume. Then one more thing to back up, one more thing to monitor, one more set of credentials, and one more dependency whose availability multiplies into yours. Lesson 003 did that multiplication for Stagefront's purchase path and got 99.78%, with the payment provider alone eating 45% of the budget.
Which gives the one test I would actually apply. Is the new shelf on the path that takes money? Marlow's covers are not: if object storage is unreachable for nine hours, every page renders, search works, checkout takes money, and 1.2 million books look sad. That is a cheap shelf. If the founder had put the sessions there, the same nine hours would log out every customer in the shop.
Same store. Same nine hours. Completely different Tuesday.
Recap
A store is a bundle of promises, not a brand. Name what the data needs from the five questions: the unit you read and write, who wins when two writers race, what happens when the machine dies, what you can ask without having planned for it, and what the bytes cost to write, replicate, back up and hand out. Then the shelf usually picks itself.
The schema is the only code every writer runs. Marlow has five write paths and one of them is a CSV import written in year one that knows nothing about anything. A rule in the application holds for the paths you remembered. A rule in the table holds for the founder's psql session at eleven at night.
A byte in a row gets written down five times. The heap page, the write-ahead log, the log again on 011's replica, the replica's own pages, the nightly dump, and then a read back through a restore on the worst day of your year. Do that multiplication before you put 216 gigabytes of JPEG into a column; the dump is where it stops being survivable.
The unit of atomicity is the unit of design. Every document store writes one document atomically, and every decision that fits inside one document is therefore fine. The question is never whether the store has transactions; it is whether your decision fits in its atom.
Durability and availability are sold separately. Eleven nines and a 99.9% SLA on the same shelf means one lost cover every 83,000 years and up to 8.8 hours next year when you cannot reach any of them. Decide which of those two numbers your data cares about before you pick.
The row that outlives the object. Pointer on the shelf that enforces rules, bytes on the shelf that is cheap and durable. Write the object first and commit the row second, because a thousand orphaned covers cost half a cent a month and a broken pointer is a hole in your page.
For file shaped data the storage is never the bill. Marlow's 216 gigabytes cost $5.40 a month to keep and about $187 a month to hand out. Every conversation about storage cost that stops at the gigabytes is measuring the wrong thing.
Check your understanding
The founder's first instinct on that Saturday was to put the covers into a
byteacolumn so both boxes could read them. Give the two cheapest arguments against it, one about the nightly dump and one about what happens on a Tuesday in March when a publisher sends 4,000 new covers at once.Marlow's sweeper deletes objects that have no row in
covers. Write the rule it needs so that it never deletes an upload that is still in progress, then say what goes wrong if the grace period is one minute instead of a day.Galewatch keeps a year of turbine readings, 450 rows a second, never updated, read back as time ranges. Go through the five questions from the shelf section and say, for each one, whether a relational database is earning its money on that data or just charging for it.
Stagefront wants to store a PDF ticket for every seat sold, generated at purchase and downloaded once or twice. Say which shelf each piece goes on, what the row about each PDF has to contain, and what happens if the object store is unreachable during the on-sale minute.
A colleague proposes moving Marlow's
sessionstable out of Postgres and into object storage, on the grounds that a session is small and object storage is cheap and durable. Give the arithmetic that kills it, using 007's figures for how many sessions exist and how often they are read.
Next lesson
017 Message Queues: The Buffer With a Memory. Today was about where data sits still; next lesson is about the shelf you put work on when the thing that has to happen cannot happen right now, which is the first honest answer this course has offered to the problem of a dependency that is slower than you are.