Galewatch is a telemetry company that exists only in this course. It takes a reading every two seconds from each wind turbine on the farms it monitors and sells the owners dashboards. On Thursday 8 October 2026, one of its own engineers fabricated thirteen and a half hours of weather.
Nobody broke in. The fault being chased was a real one and a nasty one: a few turbines reporting zero output with the blades plainly turning. It would not reproduce on the bench with a single emulated turbine, and reproducing a firmware fault means running the firmware, so an engineer brought up three hundred instances of the shipping image under an emulator and gave them the serial numbers out of the fault report, which were Ardnave Point's. Ardnave Point is the customer site lesson 014 won and lesson 031 split three ways, three hundred turbines off the west coast, the largest site Galewatch has.
The shipping image carries the production ingest key, and that follows from something lesson 053 published rather than from anybody's carelessness: Galewatch has one ingest key, shared by every turbine it reads, with no expiry and no way to find out how many copies exist. One key for every turbine means the key lives in whatever every turbine is built from. So the rig was not pointed at production by mistake. Running the shipping image is what reproducing a firmware fault means, and the shipping image goes where shipping firmware goes.
The engineer set it running at 18:40 and went home.
Three hundred simulated turbines at a reading every two seconds is a hundred and fifty readings a second, which is the figure lessons 037 and 041 both use for that farm. From 18:40 on the Thursday to 08:10 on the Friday is forty eight thousand six hundred seconds. Seven million two hundred and ninety thousand readings, about 1.5 gigabytes at the two hundred bytes lesson 002 sized one at, all of them carrying real turbine ids and all of them made up.
Lesson 019 put a unique constraint on (turbine_id, recorded_at) precisely so a reading could not land twice, and it refused exactly none of these. The ids were real and the timestamps were new. Every real turbine samples on its own clock, and lesson 034 measured how far apart those clocks sit: a median offset of four tenths of a second, and a worst case of forty one seconds. The rig ran its three hundred instances off one host clock, so its readings fell in the gaps between the genuine ones rather than on top of them, which is the seam lesson 019 named in its own text: the uniqueness is only as trustworthy as the clock inside the turbine, and clocks are lesson 034.
At 08:10 the asset manager at Ardnave Point, the one who sent lesson 051 its blunt email, rang to ask why the farm had produced half as much power overnight as the wind would suggest. They were right. Half the readings were zeros from a rig in an office.
Nothing in a row says which half.
A trust boundary is where you stop believing what you are told
A trust boundary is a line in your system across which you stop believing what you are told and start checking. Marlow Books, the online bookshop run by four people that lesson 001 found running on a single server nobody thought about, began with exactly one: the internet on the outside, everything the shop owned on the inside. That is the model most of us were taught, and for a single box it is not wrong. It is also the only arrangement in which it is ever right.
Two terms worth separating, because the rest of this lesson lives in the gap between them. A principal is whoever or whatever a request claims to be. Its authority is what that principal is allowed to do. Lesson 029 built the first half at the edge: sessions, tokens, API keys, who is calling. The second half is the one nobody writes down, and it is where the damage is.
The perimeter model says the boundary is your network edge, and it fails at Galewatch before you finish drawing it. A turbine holds a credential and sits on a tower in a field that belongs to a customer. Lesson 029 put it plainly: a key in flash on a machine bolted to a tower cannot be rotated without an engineer and a van. There is no inside that contains that key.
Marlow is no better, and it has no field. Lesson 053 found the shop's bucket write credential in an environment variable on both application boxes and in the founder's shell history, which puts a trust boundary through a laptop that nobody drew on any diagram.
So you do not get to choose where your boundaries are. You find out where they are by asking where your credentials already are, and lesson 053's answer for the ingest key, that there is no way to count the copies, is not a tidy fact about cryptography. It is an inventory problem, and the Thursday is what an uncounted copy looks like when it belongs to you.
Zero trust is a sentence about authority
Zero trust means no request is believed because of where it came from. Every request carries its own proof of who is making it, and the thing being used does the checking rather than something at the edge on its behalf.
Sold as a network story, that is a good idea with a decade of product behind it. It is also the cheap half. Once every request proves who it is, you have not finished; you have arrived at the question. Each of those proven identities may do some set of things, and somebody has to decide what that set is. The network work gets bought because you can buy it. The authority work is a spreadsheet nobody owns.
Watch what the published authority of Galewatch's ingest key actually is. Lesson 029 proposed the fix for a key you cannot rotate, which is to shrink what it can do: that key may write readings for those turbine ids and nothing else, so a leak costs you fake readings from one farm rather than the fleet. Lesson 053 then recorded that Galewatch never did it. So the authority is "write a reading for any turbine id", fleet wide, forever.
Here is the part lesson 029 could not have priced, because it was answering a different question. 029 priced a leak, somebody else holding your key. The Thursday was not a leak. The key was used by its rightful holder, on purpose, from inside the company, strictly inside its authority. Every check in the path returned the correct answer. Scoping the key per customer would have helped, and the mechanism is worth being exact about, because it is not the scope doing the work. A per-customer key cannot be baked into one shipping image; it has to be handed to each turbine when somebody commissions it. So the rig would have come up holding nothing, and a person would have had to go and choose a credential for it, which is the moment at which somebody picks the test one. What scoping would not have done is make the night recoverable once a legitimate holder had written it.
Lesson 053 left a sentence to build on, that scoping bounds a blast radius and does nothing for rotatability. The other half is the same shape. Scope answers who and where. It says nothing at all about undo.
And notice which permission a design review would have argued about. Not "may write a reading", which is the most boring line in the file and the one that fabricated the product. Reviews argue about reads, because a read is what a breach looks like in the news.
A blast radius is measured in what you cannot get back
Blast radius usually means how much a compromised thing can reach. Count the machines, the rows, the buckets, the accounts. By that measure the Thursday was a small event: the rig reached one farm's readings, which is exactly what the design intended, and touched no other customer, no money and no credential.
The measure is wrong, and this course has already built the better one twice without pointing it at a permission.
Lesson 039 established it at a transaction: a refund is not an undo but a second payment in the other direction, and a confirmation email cannot be unsent at all. Lesson 054 established it at a deploy: the handle stops at the first irreversible step, because a restart is reversible and a table rewrite is not. Put those on a credential and you get the measure worth using.
The blast radius of a credential is the set of things it can do that nothing else you hold can undo.
That last clause is the one that earns its place. Recoverability is not a property of the action; it is a property of the action against everything else in your hands. Deleting a cover is recoverable if you hold the original and a bucket that takes writes. The same delete is permanent if you do not.
Sort authority by that, and it falls into three tiers rather than the usual two. There is reading, which cannot be undone as disclosure and changes nothing about what your system holds. There is writing, which is undoable exactly as far as you still have the previous value and nothing downstream has consumed it. And there is the tier people forget, which is taking away your own ability to recover: a delete, a rotation, a revocation, a re-encryption, a limit lowered.
Marlow's bucket makes the point in three rows, with the Thursday underneath for contrast. Lesson 001 did the multiplication, 1.2 million covers at 180 kilobytes each, about 216 gigabytes.
| What it can do | Reach | How you get back |
|---|---|---|
| Read a cover | every cover | nothing changed |
| Replace a cover | every cover | the covers table says which, never what |
| Delete a cover | every cover | the page has a hole and no source |
| Write a reading | the whole fleet | you cannot tell it from a real one |
Read that as a sentence, since the spoken version skips tables. The reach column says every cover three times and the whole fleet once, so it sorts nothing; the third column sorts all four.
Lesson 016 moved those bytes to an object store and kept a Postgres row pointing at each one, the row that outlives the object, and lesson 044 is where that row stops looking like bookkeeping and starts looking like the only index the shop has over its bucket. Which means it can tell you which objects ought to exist and nothing whatever about what was in them. Lesson 053 said the credential that can write to that bucket can replace every cover in the shop, and asked for the monthly orphan sweeper to hold a different one, because those are two amounts of trust wearing one environment variable. The reason they are two amounts of trust is in the third column, not the second.
Now run Galewatch's night down the same column, and it is the worst row on the table.
Can you delete seven million two hundred and ninety thousand fabricated rows? You need a predicate, and there isn't one. power_kw = 0 catches every genuinely parked turbine in a quiet October night. A time window catches the real readings alongside the fake ones. The rig's timestamps happen to fall between the real ones, so somebody could reconstruct a filter from that artefact, by hand, this once, having first noticed it; that is a forensic exercise and not a recovery procedure, and it works only because this particular rig sampled on an offset nobody chose.
Lesson 051 built the mechanism that should save this and cannot. When a backfill writer accepts a record for a day already rolled up, it inserts (farm, day) into a small table and the rollup job drains that table first. That is the right machinery, pointed the wrong way: it recomputes a day from the rows, and the rows are the problem. Re-running Ardnave Point's farm-day gives you the same wrong answer faster.
What would have caught it is cheaper than any permission, and lesson 034 already specified it. The identity of an event should come from a counter, not from a clock. Make a reading name itself (turbine_id, seq), with the turbine incrementing on every sample, and the rig has nowhere to put its readings. Three hundred fresh instances of a shipping image boot three hundred counters at zero, so lesson 019's constraint refuses the first row and every row after it. That is an alarm at 18:40 instead of a phone call at 08:10. An identifier with room between its values is an identifier a forger can use, and a clock is nothing but room. Lesson 034 also published the price, new firmware on every turbine in the fleet, which for the older ones means an engineer and a van.
Every graph in an ingestion pipeline is pointed at absence
Thirteen and a half hours is a long time for this to run, and the reason is worth more than the story.
Lesson 026 gave Galewatch its headline number: readings landed per farm per minute. Ardnave Point's three hundred turbines at thirty readings a minute each is nine thousand a minute, and from 18:40 it read eighteen thousand. The rate on the graph the company actually watches doubled, on its largest site, and the graph was working perfectly.
No alert fired, because nothing at Galewatch alerts on that number going up. Every incident that ever taught the company to watch this graph was readings going missing: lesson 031's dotted power curve, lesson 041's five and a half hour hole. The thresholds point down because they were written the morning after the last outage.
A rate limit is the other guard a reader will reach for, and the course has already turned it down. Lesson 021 is blunt about it: rate limit that ingest path and you have protected nothing, because a 429 to a turbine is either resent a minute later or drops the product on the floor. That is right, and it is about capacity, and it rules out the one arithmetic that would have caught this. Thirty readings a minute from an id that can only produce thirty is a threshold a child could set, and at 18:40 three hundred ids went to sixty at once.
So take the threshold and throw away the refusal. An alarm does not have to answer the request, which means none of 021's reasons apply to it: the reading still lands, the flash still empties, and somebody's phone goes off. The same number is a limit or an alarm depending only on what you do with the answer, and lesson 021's case against the limit says nothing whatever against the alarm.
The instrument lesson 053 added for this exact family of question misses it as well. 053's fix for an unfinishable rotation was to stamp each accepted request with a key id and count by it, a second label on a graph that already exists. Add that label here and it reads one key, as it always has, because every turbine holds the same one. An instrument that tells you which credential wrote something tells you nothing in a system where everything shares one.
This is lesson 054's rule arriving somewhere new. Marlow's canary was green, correct and useless because every graph the founder watched was pointed at the new code while the damage was in the control group. The security costume of the same mistake is that your detection is pointed at the failure you have already had.
The control belongs on the switch, not on the action
Three lessons handed this one the same question and I owe all three an answer.
Who may flip a feature flag? Lesson 054 left that open after establishing that a flag is the only handle that works without a deploy, which is also why it skips every control. For Marlow the answer is published. Lesson 027 built the kill switch as two fields in the small uncached request every page already makes, read from a file on each box, changeable in thirty seconds over ssh with no deploy. So the authority on the most powerful handle in the shop is shell access, which at a four person company is everybody.
Who may change a scaling group? Lesson 055 left that one, and also answered it in passing with the best line in its section: a number with no derivation beside it is a number that gets raised by whoever is on call the next time a graph is red. Price what that permission does. 055 derived Galewatch's real ceiling at five writer boxes, from nineteen shards at a hundred connections each and ten connections per writer per shard, so ten writers fit and the eleventh does not. The group's maximum said six. On 055's Saturday the loop spent the afternoon adding writers into a ceiling nobody had written down, and the backlog that would have cleared at 13:28 cleared at 13:35. One reversible change to one tier, seven minutes, and every line of that configuration defensible.
So neither handle is itself irreversible. Flip the flag back, lower the maximum again, and the configuration is where it started. What you cannot flip back is what happened while it was over there, and lesson 054 already named the unit: the last point you could have got back to, which is never the switch.
A design review puts its tightest control on the dangerous action and leaves the switch that calls it open to the whole team.
Stagefront, the ticketing service in this course where a stadium show goes on sale at exactly 10:00 and two hundred thousand people press the same button in the same minute, has the cleanest version. Opening a sale is one configuration value. The code path behind it is reviewed to death, because oversold seats are a lawsuit. But lesson 039 put Stagefront's pivot at the charge, with the seats claimed and committed before the money moves, so the first charge that value lets through is past the pivot and the only thing after it is issuing a ticket. Flip that value at 09:40 and there is no flipping it back at 09:41. The forty thousand seats lesson 017 put in its stadium for the sake of the arithmetic have started going to whoever happened to be looking twenty minutes early, and lesson 039 says the money moved before the ticket did.
The practical move is small and I have never seen it on a checklist. For each switch you own, write down the first irreversible thing it can start, and put the control there rather than on the switch's own blast radius, which is zero.
The credential that fixes a thing cannot be minted by the thing
One question is left, and lessons 027, 029 and 042 have been circling it.
Lesson 027 set the test: fail open on a suggestion, fail closed on a decision, and it offered a third option that is usually the right one, which is to fail open and write down that you did. Lesson 029 took that to the check at the edge and closed it off properly: a check is never a suggestion. Lesson 042 did the breaker's version.
Point the same question at an administrative action and the answer looks trivial. Can a stranger rotate your signing key while your authorization service is down? Obviously not. Fail closed, and nobody argues.
Then book the price, which nobody does. The reason you are reaching for an administrative action during an outage is that there is an outage, and if the thing that authorises administration is downstream of what is broken, you have built a system that locks you out precisely when you need in.
Lesson 003 brought the large version into this course and lesson 053 has already used it once: October 2021, the game platform Roblox down for seventy three hours with the failure in a Consul cluster, and Nomad and Vault both sitting on top of Consul in the company's own report, which lesson 048 read, so nobody could schedule a container or retrieve a production secret. What 053 took from it was the duration. A credential's lifetime, it said, is how long you may be unable to reach the thing that mints it. The second term is the one the Roblox shape is about, and it is not a duration. It is a dependency. A ninety day credential buys you nothing if the minter sits behind the failure, because the credential you need is the one you need now.
Hence break-glass: a credential that is deliberately outside the blast radius of the system it recovers. An account whose key is in a safe, a root credential on a different provider, a console nobody's service mesh fronts.
Be honest about what that is. A break-glass credential is long lived, broadly scoped and effectively never rotated, which is everything lessons 029 and 053 argued against. You cannot make it safe. The resolution is to stop trying, and it turns out to be lesson 027's third option wearing different clothes: fail open on the recovery path, and write down that you did. Restrict a break-glass credential and you have a safe you cannot open on the night the building is on fire. Instrument it and every use costs somebody a conversation the next morning.
Which makes the receipt the entire mechanism, and a receipt your own incident can delete is not one. Lesson 059 owns that.
Recap
A trust boundary is where you stop believing what you are told and start checking, and you do not choose where yours are. You discover them by asking where your credentials already are, which at Galewatch is a tower in a field and at Marlow is the founder's shell history.
Zero trust is a sentence about authority. The network half is the half you can buy; the other half is a decision, per credential, about what that identity may do, and the Thursday happened inside a correct answer to every check in the path.
A blast radius is measured in what you cannot get back. Reach is the number everybody counts and it gave the same answer for reading, replacing and deleting every cover Marlow owns; recoverability separated all three. The full version has a second clause that does the work: what this credential can do that nothing else you hold can undo. Which means a blast radius is not a property of a permission at all. It is a property of a permission and whatever else you are holding, which is why the same delete is two different risks at two companies.
Scope answers who and where, and says nothing about undo. Galewatch has no scope to speak of, and the rig stayed inside the authority its key really has anyway, which is the part worth keeping: every check in the path said yes correctly, and the night is still gone.
An identifier with room between its values is an identifier a forger can use, and a clock is nothing but room. Lesson 034's counter would have turned a phone call at 08:10 into a constraint violation at 18:40.
Every graph in an ingestion pipeline is pointed at absence, because it was written the morning after the last outage. Galewatch's headline number doubled on its largest site and nothing fired. The threshold that would have caught it was ruled out years earlier for a reason that was right about something else: a number is a limit or an alarm depending only on what you do with the answer, and lesson 021's case against refusing the reading was never a case against ringing a bell.
The control belongs on the switch, not on the action. Write down the first irreversible thing each switch you own can start, because the switch is usually the reversible part and that is exactly what makes it feel safe.
And the credential that fixes a thing cannot be minted by the thing. Break-glass is lesson 027's fail-open-and-write-it-down pointed at authorization, which is why it cannot be made safe and has to be made loud.
Check your understanding
Galewatch's on-call proposes the cheapest guard anybody can think of: alert whenever a single turbine's readings-per-minute goes above thirty. Say what else that alert would fire on given what lessons 034, 037 and 041 have already put into this system, what it would still miss, and whether you would ship it before or after lesson 034's counter.
Marlow's monthly sweeper deletes orphaned objects from the covers bucket and holds the same credential as the deploy that writes them. Say what the blast radius of that credential is under today's definition, what you would want in place before the next run, and whether your answer changes if the bucket keeps old versions.
It is 09:55 on a Stagefront on-sale day and the service that answers "may this operator open the sale" is unreachable. Describe the break-glass path you would want to exist, who may use it, and what has to be true of the record it leaves behind for you to defend the design in a review.
A colleague proposes that every production change go through a review with two approvers. Say which of the handles above it would genuinely cover, which one it would slow down in a way you would regret by the second incident, and what you would do instead for the ones it misses.
Lesson 053 left Galewatch's rotation stopped one step short, with the old ingest key still accepted and the last step five months out in a customer's maintenance plan. Say what the blast radius of the old key is during those five months, which of today's three tiers the missing step sits in, and what you would ship in the meantime.
You inherit a reporting service whose credential can delete from the object store, and reporting has never deleted anything. Say how you would find out whether that permission has ever been used, what you need before you remove it, and what you do if the honest answer is that nobody can tell.
Take any service you work on and name the one credential whose misuse you could not undo with everything else you hold. Then say what would have to change for the answer to be "none of them", and whether that change is worth its price.
Next lesson
058 Offline-First Systems and Sync Conflicts. Today left a credential sitting in flash on somebody else's hardware in a field; lesson 041 left the question of what that flash chip actually does in the eighth hour, and 058 opens it, because a device that holds both your data and your credential while it is disconnected puts a trust boundary somewhere you cannot reach.