Galewatch is a company that exists only in this course and collects a reading every two seconds from each of twelve hundred wind turbines, selling the farm owners dashboards. Its backups live in an object storage bucket, which is where lesson 013's restore drill pulled them from: one base backup of each of its nineteen Postgres shards, every Sunday night, with a lifecycle rule that deletes anything older than twenty eight days.
Four weeks of nineteen shards. Lesson 031 put the fleet at 5.5 terabytes, so that is seventy six objects and about twenty two terabytes, and at the two and a half cents a gigabyte a month lesson 016 priced, about $550.
September's storage line said $683.
The engineer who noticed listed the bucket. Seventy six objects, 22 terabytes, exactly as expected, summed by the command anybody would reach for. The bill was a hundred and thirty three dollars above a bucket whose contents added up correctly.
Those hundred and thirty three dollars are 5,300 gigabytes, and they were sitting in twenty seven multipart uploads that had been started and never finished. The oldest was from the second Sunday in May.
Here is how they got there. A base backup of a Galewatch shard is far too big for one HTTP request, so the script sends it as a multipart upload: open an upload, push the object in numbered parts, then one final call stitches them together. The link to the bucket is good but not perfect, and roughly one upload in fifteen hits a network error partway through. When that happens the script does the sensible thing and retries the whole backup from the top, which works, and the Monday morning log says "attempt 2 succeeded" and nobody has ever read it.
The parts from attempt 1 stay exactly where they were. Twenty one Sundays since May, nineteen shards a Sunday, is 399 uploads; one in fifteen of those is about twenty seven abandoned attempts, holding an average of 196 gigabytes each.
That average is worth a second look, because the average Galewatch shard is 290 gigabytes and an upload cut in half at a random moment should strand 145. The bias is not an accident. Your chance of being interrupted goes with how long you are exposed, which goes with the object's size, and the bytes you lose when it happens go with that size again. Kilmore Sands, the hundred and forty turbine farm that lessons 013 and 014 established as Galewatch's fattest shard at seven hundred gigabytes, is about two and a half times the average, which is roughly the skew factor lesson 014 printed when the fleet was sixteen machines. So it is exposed two and a half times as long and loses two and a half times as much when it is interrupted, which is six times the garbage out of one shard. The rule generalises and is worth keeping: the mess a failed transfer leaves behind goes with the square of the object's size.
And the twenty eight day rule did not touch a single byte of it, because expiring objects and abandoning half-finished uploads are two different lifecycle rules, and the bucket only had the first.
Two commands fixed it. One to abort the twenty seven, one to add the rule that aborts them after seven days from now on. Add up what the company paid while it was accumulating, 252 gigabytes a week for twenty one weeks, and it is a little over three hundred dollars for bytes that no listing in the world would have shown anybody.
The money is trivial. What a bucket holds and what the one command everybody runs will tell you about it are two different quantities, and only one of them reaches the invoice.
Two systems wearing one name
An object store looks like the simplest thing in your architecture. A key, some bytes, HTTP. That is where lesson 016 left it, at the depth Phase 1 needed.
Underneath, it is two systems.
One holds bytes: storage nodes with disks that have no idea what a bucket is and address their contents by something closer to a fragment id than a filename. The other is an index: a sorted, sharded key-value store whose keys are the pair of bucket name and object key, and whose values are the small record that says how big the object is, when it was written, what its checksum is, and where on the fleet its bytes actually live.
A GET is an index lookup and then a byte fetch. A PUT writes the bytes, waits until they are durable, and only then writes the index record. Almost everything surprising about object storage is one of those two halves showing through.
GET <bucket>/<key>
|
v
[ metadata index ] sorted by (bucket, key), cut into key ranges
| holds size, checksum, and where the fragments are
v
[ storage nodes ] fragments, no idea what a bucket is
|
v
the bytes
That index is sorted by key and partitioned by ranges of key. Which makes the thing lesson 016 called a convention more than one. Marlow Books is a four person online bookshop that exists only in this course, and covers/9780140449136.jpg is its key for one cover image. The slashes really are just characters, and the namespace really is flat, with no directories in it anywhere. But a run of keys that share a leading string is a contiguous run in a sorted index, and a contiguous run in a sorted index is one partition.
So your filename convention is a shard key. Nobody told the person who chose it.
Amazon publishes the consequence as a rate: at least 3,500 writes a second and 5,500 reads a second per partitioned prefix. Marlow is nowhere near that. Lesson 016 priced the shop at four cover fetches a second, and after lesson 022's CDN took sixty two percent of them the bucket sees about one and a half.
Stagefront is a ticketing service, invented for this course, where a stadium show goes on sale at exactly 10:00 and oversold seats are a lawsuit. Lesson 016 asked, in a check question nobody has answered since, where a PDF ticket for every seat sold should live. Put it in a bucket and the write rate is lesson 017's sold-out stadium: forty thousand seats in eighty seconds, so five hundred PUTs a second. Against 3,500, that is fine with a factor of seven to spare.
Then somebody keys them by date. tickets/2026-03-14/ is a lovely key right up to the Saturday morning when eight shows go on sale at ten o'clock, at which point four thousand writes a second land in one partition, above the floor, and the store starts answering with 503 SlowDown. It will split the partition on its own, and that is genuinely automatic, and it takes tens of minutes. An on-sale is eighty seconds long. You do not get rescued inside eighty seconds by a system that reacts in half an hour.
This is lesson 014's boulders and gravel in a new costume, and 014's warning comes with it: the test is pieces per partition, not distinct values. A date is a boulder that all eight shows land on. Split by show and you get eight boulders, which by 014's count is still an assignment rather than a shard key, and here that is enough: tickets/<show-id>/<seat>.pdf puts five hundred a second in each of eight prefixes, every one of them under the floor. The show id was already in the key. All you did was move it to the left.
A listing is not an index
Lesson 016 mentioned in passing that a listing is paginated at a thousand keys a call, so Marlow's 1.2 million covers take 1,200 round trips. Worth knowing what that actually buys you, because it is less than people assume.
Some of it got better six years ago. Since December 2020 an S3 bucket gives you strong read-after-write consistency on everything: write an object and the next GET sees it, the next listing sees it, the next metadata read sees it. For the fourteen years before that, a listing lagged, and a generation of code carries sleeps and retries that exist only because of it. If you learned this before 2020, relearn it.
Now the part that is still true and that nobody mentions.
A thousand keys a call over 1.2 million keys is 1,200 calls. Each one of those calls is strongly consistent, and each one happens at a different moment, forty milliseconds or so apart, so the whole walk takes about fifty seconds, and the bucket is being written to throughout. A key that arrives sorting behind the cursor you have already passed will not appear in your results, and a key you already listed may be gone by the time you finish.
A paginated listing is twelve hundred correct answers to twelve hundred slightly different questions. It is not a snapshot of anything, and no setting makes it one.
Which adds a term to something lesson 016 already got right by accident. Marlow's sweeper runs monthly, lists the bucket, and deletes any object that has no row in the covers table; lesson 016 gave it a day of grace so it could not delete the bytes of an upload whose row was still in flight, then asked the reader what would go wrong with a grace period of one minute instead. Here is one more thing that goes wrong: the walk itself takes fifty seconds. A day covers that by about eighteen hundred to one. The founder did not design it for that and would not have thought of it, and it is the whole reason the sweeper has never eaten a cover.
The sweeper is also free, which surprises people. A LIST call is priced like a write rather than like a read, at about five dollars a million calls, so 1,200 of them cost less than a cent.
What you cannot do with a listing is ask it a question. Not "which covers are bigger than half a megabyte", not "which ones did this publisher send in March", not even "how many objects are in here" without walking every one of them. There is no secondary index, there is no COUNT, and the key is the only thing you can filter on and only by prefix. Lesson 016's covers table exists for exactly this reason, and today is where that row stops looking like bookkeeping and starts looking like the only index you have.
Amazon's own answer to this is the most honest feature in the product. S3 Inventory writes you a manifest, daily or weekly, as CSV or Parquet, into another bucket: every key, size, storage class, version, encryption state. A scheduled batch export, because the live API cannot answer the question. It is up to a day stale, which is fine for a sweeper and useless for anything a customer is waiting on.
And the hook, one more time, because it belongs here: a listing shows you current objects. Not the parts of an unfinished multipart upload, and on a versioned bucket not the old versions either. Both are billed.
No partial write, so no rename
The bytes half of the store has one rule that explains most of its API, and lesson 016 stated it without taking it anywhere: there is no partial update. Change one pixel of a 180 kilobyte cover and you write 180 kilobytes. Lesson 043's closing line was do not write what did not change, and here there is simply no handle to grab. The unit is the object.
A rename is the same rule in disguise. There is no rename call, because it would have to move a key in the index while the bytes stayed put. What you do instead is a server-side COPY to the new key and a DELETE of the old. The COPY does not pull the bytes through your machine, which is the good part. It is two operations rather than one, which is the bad part: for a while both keys exist, and a process that dies in the middle leaves an orphan.
Price it on Marlow. Re-keying 1.2 million covers is 1.2 million COPY requests at about five dollars a million, so six dollars and an afternoon of watching a script. Not expensive. Not free, and not atomic, and at an object over five gigabytes the COPY itself has to be done in parts.
Notice the asymmetry in those two prices. Lesson 016 published reads at about forty cents a million. Writes, copies and listings are about five dollars a million, which is twelve and a half times as much. In a store where nothing can be modified, every mutation is a whole write, and the price list is built around that.
A single PUT tops out at five gigabytes. Above that there is one road, and it has three moves: create an upload and get an id, send numbered parts, then call complete with the list of parts and their checksums. Each part is at least five megabytes except the last, at most five gigabytes, and there are at most ten thousand of them. The largest object the store will hold is five terabytes, which is a limit of its own and not a consequence of those two, since ten thousand parts of five gigabytes would be fifty.
Ten thousand parts is also an arithmetic trap worth keeping. The minimum part size for an object is the object divided by ten thousand, so Kilmore Sands' seven hundred gigabytes needs parts of at least seventy megabytes. A client configured with a fixed eight megabyte part size tops out at eighty gigabytes and fails on anything larger, always at the same point, which makes it look like a network problem rather than a division.
The complete call is the interesting one. Until it lands, a GET on that key returns 404; the moment it lands, the whole object is there. The object appears entirely or not at all.
That is the closest thing an object store gives you to a transaction, and nothing else in the API comes near it. It is also why the hook's twenty seven abandoned uploads were invisible: an upload that never completes never becomes an object, so there is nothing for a listing to list, and the parts you already sent sit in a side area that only ListMultipartUploads and ListParts can see.
Parts are also where throughput comes from, in both directions. One HTTPS connection to an object store will move perhaps eighty to a hundred megabytes a second if everything is going well, and that is a per-connection ceiling you cannot talk your way past. Aggregate rate comes from running many at once: parts on the way up, and ranged GETs on the way down, using the Range header to ask for bytes 0 to 70,000,000 and so on. The AWS CLI runs ten such transfers concurrently by default, which is why aws s3 cp is so much faster than curl on the same wire.
Run that against June's restore drill. Lesson 013 moved 4,500 gigabytes and measured about 200 megabytes a second; lesson 043 pinned the figure at 201 and established it is per machine, because Kilmore Sands came back in fifty eight minutes while fifteen other shards restored alongside it. Spread 201 over ten concurrent ranged GETs and each stream is doing about twenty megabytes a second, a quarter of what one connection can do. So the bucket was not the limit, and neither was the pipe. Whatever held the restore to 201 was on Galewatch's side of the wire: the disk it was writing to, a decompressor, a single-threaded restore process. Lesson 013 asked which. Lesson 043 narrowed it to the machine. Nobody has opened the machine.
One last absence, because Galewatch runs into it every day. There is no append. You cannot stream a write-ahead log into one growing object; every segment is its own key, and a fleet of nineteen shards turning over sixteen megabyte segments all day produces thousands of keys a day. The bytes are nothing; the requests are the bill.
Eleven nines is a model of disks
Lesson 003 used the published numbers and lesson 016 ran them against Marlow: eleven nines of durability, a 99.9% availability SLA, which for 1.2 million covers is one expected loss about every eighty three thousand years against up to 8.8 hours a year of not being able to reach them. Durability and availability are sold separately.
Eleven nines is not three copies.
Three copies gives you 3x the storage and survives two simultaneous losses. The alternative is erasure coding, which comes out of coding theory and is easier than its reputation. Split the object into k fragments. Compute m more fragments of parity from them. Store all k plus m on different failure domains, meaning different disks, machines and racks, so that one event cannot take several of them at once. Any k of them, data or parity, in any combination, reconstruct the original.
Work one scheme. Take k of 8 and m of 4: twelve fragments, each an eighth of the object in size, so you store 1.5 times the object, and any four of the twelve can be destroyed with no loss at all.
Hold that against replication. Half the storage of three copies, and twice the failure tolerance. That is not a clever optimisation, it is the entire reason this shelf costs two and a half cents a gigabyte while the block storage under Marlow's box A costs three or four times as much. Providers do not publish their actual schemes, so treat 8 plus 4 as representative rather than as a fact about any particular store.
What erasure coding costs you is repair traffic, and this is the number worth keeping. When a fragment dies you cannot copy it from anywhere, because no single disk holds a copy of it; you have to read k fragments and compute the missing one. So repairing one lost fragment reads eight times its size. An eighteen terabyte disk failing means reading 144 terabytes off its peers to rebuild what it held. Keep the same four parity fragments and widen to k of 12 and your overhead drops to 1.33x while every repair reads twelve times the loss. Repair bandwidth, not disk price, is what sets the width.
| Scheme | Stored | Losses survived | A repair reads |
|---|---|---|---|
| Three copies | 3x | 2 | the size of the loss |
| k 8, m 4 | 1.5x | 4 | 8 times the loss |
| k 12, m 4 | 1.33x | 4 | 12 times the loss |
Small objects also fight it. Cut a 180 kilobyte cover into eight fragments and each one is 22 kilobytes, at which point the index record and the per-fragment bookkeeping start to rival the data. Real stores pack many small objects into one container before coding it, or just replicate the tiny ones.
Now the honest part about the eleven nines, which lesson 003 taught you to ask for. It is a model, and the model assumes independent failures. A firmware batch that bricks a thousand drives in a month is not independent. Nor is a rack, a data centre, a region, a deployment. Nor is the one on 28 February 2017, when a mistyped command in S3's us-east-1 region took capacity offline for hours and much of the internet with it. Every byte survived that afternoon. It was an availability event, which is the distinction lesson 016 drew.
And the failure the eleven nines says nothing at all about is you.
A DELETE is honoured immediately and durably. A lifecycle rule with a typo in its prefix deletes at eleven nines of reliability, which is the one time in your life that number works against you. The protections exist and you have to turn them on. Versioning makes a delete write a tombstone called a delete marker and keeps the old version, which you then pay for. Object Lock gives you write-once storage that, in its compliance mode, cannot be deleted by anybody, including the account root, for a retention period you set in advance.
Versioning's own trap is the hook again in a third costume. Turn it on, keep overwriting the same key, and every old version is billed forever, invisible to an ordinary listing, until somebody adds a second lifecycle rule to expire noncurrent versions.
Three lines on the bill
Lesson 016 itemised Marlow's covers and the shape of it surprised people: $5.40 a month to keep 216 gigabytes, $4.16 to answer 10.4 million requests, and $187 to push 1,872 gigabytes down the wire. Lesson 022 put a CDN in front and the egress fell to about $71 while origin requests fell to 3.95 million.
Three lines, priced in three different units. Storage is per gigabyte-month. Egress is per gigabyte out. Requests are per call, flat, with no reference whatsoever to how big the object was.
That third one is where the design decisions live, because a flat price per call meeting a variable number of bytes gives you a crossover. A read costs about four ten-millionths of a dollar. A gigabyte of egress costs about ten cents, which is a ten-billionth of a dollar per byte. Divide one by the other and the answer is four thousand bytes.
Under about four kilobytes an object, the request costs more than the bytes it returns.
Check it against Marlow, where the object is 180 kilobytes: four thousand over a hundred and eighty thousand is 2.2 percent, and lesson 016's own bill says $4.16 of requests against $187 of egress, which is 2.2 percent. The crossover reproduces a number published twenty eight lessons ago.
Push it the other way and the store stops looking like storage. Keeping a four kilobyte object for a month costs a ten-millionth of a dollar. One GET of it costs four ten-millionths. Read it once a month and you are paying four times as much for the reading as for the keeping; read it hourly and the storage line rounds to zero. For small objects this is a request service with some bytes attached.
Lesson 016 left a check question about putting Marlow's sessions table in a bucket, and this is the arithmetic it wanted. Lesson 007 measured forty thousand live sessions at 200 bytes, read by sixty authenticated requests a second in Christmas week. That is 155 million requests a month, or $62, for eight megabytes whose storage line is two hundredths of a cent. Lesson 007's primary key read takes 0.2 milliseconds, against an HTTP round trip. The bill is the small objection.
Storage classes are the most mis-sold piece. They are not discounts, they are different promises, and each one charges you for the bet you placed.
An infrequent-access class charges less per gigabyte and more per request, and adds two floors nobody reads: a minimum billable object size of 128 kilobytes and a minimum billable storage duration of thirty days. Marlow's covers clear the size floor at 180 kilobytes, so the trap there is subtler. Lesson 022's CDN already absorbs sixty two percent of the reads, so the requests that still reach the bucket are the long tail, which is precisely the set that would now cost more per call. Leave them where they are.
Galewatch's backups look like the perfect archive candidate: written once, read only in a drill. The deeper you go the longer the store takes to answer, from milliseconds in the shallowest archive class to minutes or hours in the cold ones, and every one of them adds a per-gigabyte retrieval fee. Lesson 013 derived the entire shard count from a contractual recovery hour. A backup you cannot start reading inside your recovery objective is not a backup, and the discount is paid for out of the one number the fleet was designed around. The retention makes it worse: the archive classes set their minimum billable duration at ninety days and the deepest one at a hundred and eighty, so a twenty eight day object is paid for more than three times over.
Which leaves lifecycle rules, the only retention you get. There is no TTL you can set on a PUT and no expiry header the store honours: nothing on the write path at all. You declare rules on the bucket and the store applies them on its own schedule, roughly daily, some time after the day you asked for rather than on it.
Galewatch had one of those rules and trusted it. It expired objects after twenty eight days and did exactly that, perfectly, for five months, while twenty seven unfinished uploads it was never going to look at grew into a fifth of the bucket.
Recap
Two systems wearing one name. A sorted, sharded index from bucket plus key to a small placement record, and a fleet of disks holding fragments that know nothing about keys. A GET is a lookup and then a fetch. Every strange thing in the API is one of those halves showing through.
Your filename convention is a shard key. A prefix is a contiguous range of a sorted index, which makes it a partition, with published floors of 3,500 writes and 5,500 reads a second. Lesson 013 and lesson 014's whole argument, decided by whoever named the files. Put the part with the most pieces on the left.
A listing is not an index, and a paginated listing is not a snapshot. Every call is strongly consistent and there are 1,200 of them spread over fifty seconds, so a key that sorts behind the cursor is a key you will not see. Marlow's day of grace covers that by eighteen hundred to one, and nobody designed it for that.
Multipart is the closest thing to a transaction the store offers. The complete call makes the whole object appear at once, and an upload that never completes never becomes an object. It is also where parallelism lives, which is the only way past one connection's eighty to a hundred megabytes a second.
Eleven nines is a model of disks. Erasure coding with k of 8 and m of 4 stores 1.5x and survives four losses where three copies store 3x and survive two, and a repair reads eight times what it lost. The model assumes independent failures, and your own DELETE is not in it at all.
Under four kilobytes, the request costs more than the bytes. Four ten-millionths of a dollar a call against a ten-billionth a byte, which puts the crossover at four thousand bytes and reproduces lesson 016's 2.2 percent exactly.
And the one to carry out of today: do not trust a listing to tell you what you own. Unfinished uploads, noncurrent versions and the objects a lifecycle rule has not got round to yet are all billed and none of them are in the answer.
Check your understanding
A colleague proposes storing one JSON object per user preference change, about 900 bytes each, at two thousand writes a second, keyed
prefs/<user-id>/<timestamp>.json. Give the two lines of the bill you would compute first, say which one decides it, and name the thing that breaks before the bill does.Galewatch wants its base backups in an archive class to halve the storage line. Say what you would need to know about the restore path before agreeing, which earlier lesson owns the number that decides it, and what you would propose instead.
Your service writes an object and then immediately writes a database row pointing at it, and a colleague says the write needs a retry loop because object storage is eventually consistent. Say whether they are right today, what they would have been right about six years ago, and which of the two orderings you would still keep and why.
A nightly job COPYs ten million objects to new keys to change a prefix, then deletes the originals. It is killed at sixty percent. Describe the state of the bucket, what a listing tells you about it, and how you would make the job safe to run again from the start.
You inherit a bucket where the listed objects total 4 terabytes and the bill says 11. Give three things that could account for the difference, the call or report you would use to check each, and which one you would look at first.
Stagefront wants to generate a PDF ticket per seat during the on-sale minute and have it downloadable immediately. Say what the key should be and why, what happens to the purchase if the PUT fails, and which earlier lesson already decided the ordering of the row and the object.
Next lesson
045 Search: The Inverted Index. Today the only index you got was a key, and the only question you could ask was a prefix; next lesson builds the structure that answers "which documents contain this word", which is the question Marlow's 1.2 million descriptions have been asking since lesson 010.