Secrets, Keys and Rotation

Galewatch is a telemetry service invented for this course. It takes a reading every two seconds from each wind turbine on the farms it monitors, and sells those farms' owners the dashboards. On Thursday 22 October an engineer in their second week wanted to understand the ingest format, so they searched the log store for a turbine serial they had seen on a screen.

The line came back with everything in it. The serial, the reading, both timestamps, the response code, and the key header, in full, all forty characters of it.

They asked in a channel whether that was meant to be there.

It had been there since August. On the first Saturday of that month Ardnave Point, Galewatch's largest site at three hundred turbines off the west coast, lost its link for seven and a half hours, and when the link came back a writer process died holding three million readings. Lesson 041 is that Saturday. An engineer at the farm found the hole on the Tuesday, and while they were chasing it somebody set the four ingest boxes to log every request instead of only the failures, which is what you do when a process has died silently and left you nothing to read.

Nobody turned it off. Eleven weeks.

Ardnave Point on its own sends a hundred and fifty readings a second, so that one farm wrote the ingest key into a log line 12,960,000 times a day. The store keeps thirty days, so on the Thursday morning about three hundred and ninety million live copies sat in an index everybody in the company can query, plus whatever is in that index's nightly backups and in the download folder of anyone who has ever exported a search.

Every number in that paragraph is beside the point. One copy, in one place you do not control, is the entire loss. You cannot be partly in possession of a forty character string.

So on the Friday they rotated it, which was right, and found out what the obvious thing costs.

Galewatch has a single ingest key, shared by every turbine it reads. Lesson 029 named what that is: an API key, a password you posted to somebody, with no expiry and no way to count the copies. It also gave the answer for a key you cannot rotate, which is to shrink what the key can do, and Galewatch never did it. Scoping would have made this fire smaller and the key no easier to replace.

Newer controllers take a configuration update over the link they send readings on. Older ones do not: the key lives in flash, and it gets there by somebody standing at the base of the tower with a handset.

Tarrow Ridge is the farm this course has already measured: sixty turbines at the end of a long access road, from lesson 017's Wednesday in October. Its controllers are the old generation. It is not the only farm of that generation, and the list of which farm has which is an artefact nobody at Galewatch has maintained.

Galewatch does not own those turbines. The owners do, their technicians do the climbing, and Tarrow Ridge is serviced in March and in September. September's round finished a month before the Thursday.

By Friday evening every controller that could take a push had the new key. The old one is still accepted, and it is worth being exact about why. Lesson 017 established that a turbine deletes its local copy of a reading only once an ingest box has answered 200, so refusing the old key does not blind Tarrow Ridge. It turns Tarrow Ridge into a backlog, and lesson 041 is what a backlog does on the way back in. Nobody has published how much flash those turbines hold. So the last step sits in another company's maintenance plan, five months out.

So Galewatch spent a day's work and now has two keys that open its ingest endpoint instead of one, and one of them is in a search index.

Three words that get used as one

A secret is any value whose only protection is that other people do not have it. A key is a secret an algorithm uses, to encrypt something or to sign it. A credential is a secret you present to prove you are allowed to do a thing.

Those last two separate in a way that organises everything after this. A credential is checked by somebody holding a list, so rotating one is a deletion from that list, and the list is usually a row in your own database. A key is used by code and checked by nobody, and whatever it produced is still out there, encrypted or signed with it, so rotating one is a rewrite of everything it touched.

Same word, two jobs.

Encryption at rest means the bytes on the disk are ciphertext, and it protects you against somebody who gets the disk: a stolen drive, an exposed backup, a server returned to the leasing company with its contents intact. Encryption in transit means the bytes on the wire are ciphertext, and it protects you against somebody watching that wire.

Neither protects you against a process that is allowed to read the thing. And a process that presents a credential is always allowed to read it, because it has to put the bytes in a header.

That is not a gap in the mechanisms. It is what they are for, and it is why secrets escape from places that have nothing to do with disks or wires.

Galewatch's Thursday is the plain version. Lesson 026 is about instruments, what an instrument does is write down what happened, and what happened contains the header. Notice which decision did the damage. Logging the header on a failure is old and defensible, because a 401 you cannot explain is worth an afternoon. Logging it on success was a debugging flag nobody owned, which rhymes with lesson 048's configuration service, whose availability had never mattered to anybody until a graph noticed eight days later.

The other two venues are already built here. Lesson 040's outbox row holds a payload you meant to send, and a payload assembled in a hurry holds whatever the request held. Lesson 037's log is worse, because a topic has more consumers than you remember. Then the crash: a request dump on an error report, a stack frame holding its arguments, a screenshot pasted into a support chat. Lesson 029 called a token a bearer instrument and said every place it can be copied from is part of your security, and a key is that instrument with no expiry on it.

Galewatch's thirty day retention will delete those log lines on its own, and it buys nothing, because the key does not expire when the line holding it does. Lesson 037 put retention well: how long you are allowed to be wrong for. Here it is how long the secret stays readable, and it is the only cleanup anybody has.

Deleting a secret from the places it leaked into is work you can never finish and can never verify. Making the leaked value worthless is one deletion from one list. Rotation is not the hygiene habit it gets filed as. It is the only remedy that terminates.

Rotation is expand and contract with a deadline

Lesson 052 took stored data through expand and contract: widen the world so both shapes are legal, move everybody across, then narrow it again. Five deploys, and its honest finding was that almost nobody reaches the fifth. It also found that the wait between deploys is set by where the old shape lives, and gave four venues for that.

A credential goes through the same five steps, and today generalises both findings by changing one word. For stored data the thing you wait for is a reader. For a credential it is a client.

Issue the new one. Accept both. Move the clients. Confirm nobody is using the old one. Delete it.

Now look at what each step changes, because this is where the habit falls apart. Step one adds a valid credential. Step two adds a second accepted one to your list. Step three moves work around and removes nothing. Step four is a measurement. Step five is the only step that takes anything away from anybody.

So the first four steps make your exposure strictly worse, and the whole benefit arrives in the last five seconds. Galewatch's Friday is not an unlucky rotation. It is what a rotation is, stopped one step short.

I have watched teams run steps one to three every quarter for years and call it key rotation on a compliance form, and in a sense it is. The old keys are all still accepted. The list only grows.

The step that needs somebody to go and look

Step four is where it dies, and the reason is structural. Every store you can query tells you what is installed. Whether the old credential is still in use is a different fact, and it lives on the wire.

Galewatch's configuration store will cheerfully confirm that three hundred of Ardnave Point's turbines took the push. It cannot tell you whether Tarrow Ridge's sixty are still presenting the old key, because a turbine's flash is not something Galewatch can read. That fact exists only in the requests arriving at the ingest boxes.

The fix is nearly free, and it is what they did get on the Friday. Stamp the accepted request with which key authenticated it, a key id, and count by it. Lesson 052's registry writes a schema id into every record on a stream so a reader knows which shape it is holding; this is the same idea with a credential in it. Lesson 026 established that Galewatch's one real metric is readings landed per farm per minute, so the instrument is a second label on a graph that already exists, and the rotation becomes finishable the moment that label reads zero for the old key.

Lesson 029 closed lesson 028's attribution seam with the line that a key is how a graph learns a name. A key id is how a graph learns which key, and without it step four has no evidence and step five has no permission. Most engineers I have asked can rotate a credential. Almost none can say how they would know the old one had stopped being used.

The March round happens and the graph goes to zero. Do you delete?

You delete, because waiting does not improve the evidence. A turbine silent for a fortnight with its link down comes back presenting whatever is in its flash, and a graph reading zero says nothing about it. So step five is a decision rather than a certainty, and the defensible version is a window longer than the longest link outage this course has published for that fleet, which is lesson 041's seven and a half hours.

Four rotations, four different jobs

Read that as a sentence, since the spoken version skips tables. Those four are the same five steps and four different jobs, and what separates them is who has to act before the deletion is allowed. At the top nothing acts, because every holder of the old credential loses it on a timer. At the bottom somebody drives to a hillside and they work for your customer.

The last step of a rotation finishes by itself only when the thing carrying the old credential has an expiry. Otherwise somebody has to go and look, and that is why almost no rotation ever finishes.

Lesson 004 said short certificate lifetimes are what fixed this industry wide, on the grounds that at ninety days nobody renews by hand and the automation you are forced to build is the thing that saves you. True, and there is a second half. A short lifetime also puts step five on a clock. You are not trusting anybody to confirm the old credential is gone. It left on its own.

The one case where the signing key is cheap

Lesson 029 built the token properly and handed the signing key to today, and whether that rotation is pleasant or miserable turns on a choice made for other reasons.

A symmetric key is one value used for both halves of an operation, so whoever can check a signature can also produce one. An asymmetric key is a pair: a private half that signs and a public half that only verifies.

Lesson 007 named the usual recipe, HMAC with SHA-256, and HMAC is symmetric: the secret that checks a signature is the secret that makes one. That is the right place to start and the wrong place to stop, and lesson 029 is why. Its strongest finding was that the best reason to verify a token inside your own process is availability rather than speed, because a signature checked locally cannot be unavailable. Follow that advice with a symmetric key and every service that verifies tokens can also mint them, including the one a contractor wrote in 2023 that nobody has read since. Any verifier's compromise becomes your identity system's.

With a pair, verifiers hold the public half, there is nothing worth stealing, and rotation is genuinely nice: publish the new public key beside the old, start signing with the new private key, wait out the longest token, delete. Lesson 029 priced those lifetimes already. A token at Marlow Books, the four person online bookshop this course keeps breaking, lives fifteen minutes, and at Stagefront, the ticketing service where two hundred thousand people press the same button at ten in the morning, five.

The cost of a symmetric signing key is not paid on the day you choose it. It is paid on the day you have to rotate it in a hurry, and on that day it is the difference between an afternoon and a project.

Rotating a key you encrypted with

Rotate, revoke and re-encrypt get used interchangeably and name three different amounts of work. To rotate is to start using a new key for new work. To revoke is to stop accepting an old one. To re-encrypt is to go back over everything the old key protected and do it again.

For a credential, rotate and revoke are the whole story and there is nothing to re-encrypt. For an encryption key it inverts. Starting to use a new key is free and changes nothing about yesterday's bytes, which are still only readable with the old one, so revoking it destroys your data.

Price it where this course already has the numbers. Lesson 044 took Marlow's bucket of book covers apart: 1.2 million objects at about 180 kilobytes each, 216 gigabytes in all. A warning in passing: an object store calls an object's own name its key, which is not the kind of key this lesson means.

That lesson also published the rule which decides this whole section: there is no partial update in an object store, and the unit is the object. Change one pixel of a 180 kilobyte cover and you write 180 kilobytes. Lesson 044 priced the consequence for renaming, where moving an object means a server-side COPY to the new name and a DELETE of the old, and the bill for the shop was 1.2 million COPY requests at about five dollars a million: six dollars and an afternoon of watching a script. Re-encrypting every cover is those same two mechanisms for a different reason, so I am borrowing that six dollars rather than inventing a figure.

Here is the mechanism meant to make it cheaper, and then why it does not work in a bucket.

Envelope encryption gives every object its own data key, encrypts the object with it, encrypts that data key with a long lived master key, and stores the wrapped result beside the object. The master key lives in a key management service, which holds keys, performs operations with them and never hands them out. That is the one arrangement here where the process using a key cannot read it, which is why a master key never turns up in a log line, and why a data key can. Rotating the master key then means re-wrapping 1.2 million data keys. A data key for AES-256 is thirty two bytes, so 1.2 million of them are 38.4 megabytes against 216 gigabytes of covers. Five thousand six hundred and twenty five to one. The bytes stop being the problem.

Except that the no-partial-update rule reaches the metadata as well as the contents. A wrapped data key is a small field, and in a bucket there is no call that changes a small field. Changing it means writing the object again, all 180 kilobytes of it, which is the same COPY and the same six dollars you were trying not to spend. The object's own metadata is where it is most natural to keep that wrapped key, and where several storage products will keep it for you.

So envelope encryption collects its five thousand to one only if the wrapped data key lives somewhere with an UPDATE statement in it. Marlow has such a place. Lesson 044 called the covers table in Postgres the only index the shop has over its bucket, and a wrapped key column on that table turns a master key rotation into 1.2 million row updates. Lesson 043's rule is that every update is an insert, and lesson 052 ran it on this exact number of rows: a 240 megabyte table became about 480 until the cleaner caught up, with the write ahead log carrying most of that again, done in batches with a pause. The covers table carries narrower rows than that one, so less again. A few hundred megabytes of engine work, against 216 gigabytes and every object rewritten.

Now the opinion, because that arithmetic is a trap for a bookshop.

Marlow should not encrypt its covers. A cover is the front of a published book, lesson 022 measured its edges serving sixty two percent of the fetches to strangers on purpose, and encryption at rest answers one question, which is what a stolen disk tells somebody. A stolen disk full of book covers tells them what is in the shop window. So there is nothing here to re-encrypt and no master key worth owning. Take whatever free provider managed encryption the bucket offers, which rotates on somebody else's schedule without a line of code, and spend the attention on the credential that can write to that bucket, because that one can replace every cover in the shop.

Which is lesson 029's rule finally doing some work. Marlow's bucket credential should write under covers/ and nothing else, and the monthly sweeper that deletes orphans should hold a different one, because those are two amounts of trust wearing one environment variable.

Where the new key comes from, and the morning it does not

All of that assumes a process can get a credential when it needs one, and lesson 048's conclusion lands harder here than it did there.

Its rule was fail static: when the control plane cannot be reached, keep using the last answer you had. Then the part it said nobody writes down. Fail static buys you the fleet you have and nothing you are about to start, because a process beginning during the outage never had the fact, and a configuration outage is collected one restart at a time.

Its cheap half of the fix was to write the last good value to disk, so a cold process has something to be static about. For a routing table that is obviously right. For a secret it is the thing this lesson has spent two sections trying to prevent, except now it is your own design doing it, on every box, forever.

The large version came into this course in lesson 003. In October 2021 the game platform Roblox was down for seventy three hours with the failure in a Consul cluster, and 048 noted from the company's report that Nomad and Vault both sat on top of Consul, so nobody could schedule a container or retrieve a production secret. The running processes were not what stopped them. The inability to start anything was.

That lesson settled the general question well: what a process does on a cold start is decided by what a wrong value does, not by how old the value is. A six week old local TTL is harmless, so use it; a six week old slot map writes to the wrong machines in silence, so refuse to start and page somebody. Put a credential through that test and it lands in a third category, because a stale credential is not wrong. It works. What is wrong with it is its age, and nothing inside the process can tell.

Which is where short lifetimes earn their reputation twice, and where the bill arrives. Lesson 023 collected several of this course's expiries into one dial at different radii: a replica's lag, a cache's time to live, an edge's expiry, a token's lifetime. A credential's lifetime is another radius on it, and reading it the way 023 reads the others gives the sentence nobody writes on a design document.

A credential's lifetime is how long you may be unable to reach the thing that mints it.

Fifteen minute credentials with background refresh mean an issuer outage under fifteen minutes is invisible, and one over it takes the fleet, not a restart at a time but all at once. Annual credentials mean the issuer can be down a week unnoticed, and that the last four rotations never finished. Both are real positions. What is not a position is choosing the first and running the issuer on one machine.

Lessons 049 and 050 both left the same question, which is where a certificate comes from. Lesson 049 has it in miniature: a Stagefront box came back during an on sale and spent nine seconds collecting routes, addresses and a certificate from a busy control plane, with the application itself ready in about two, so a certificate sits on the critical path of a birth. Lesson 050 asked it about a promoted region, where the uncomfortable half is that a failover drill in which no process starts has not tested the credential path at all. A drill at two in the morning measures the half you were not worried about, and nobody worries about the standby whose processes have been up for six weeks holding credentials they fetched when somebody still remembered how.

Then the failure that survives all of this, which is the smallest one. Lesson 004's certificate expired at 05:30 on a Sunday in March and took the shop off the internet for six hours and twenty one minutes, watched the whole time by a health check that curled http://localhost:8080/health from inside the box and had never in its life left the machine. After that morning the founder moved to a ninety day certificate with automatic renewal, which is the right fix and leaves one gap.

A rotation involves three facts rather than one. The credential in the store, the copy the process has loaded, and the one going out on the wire. Lesson 004's eleven minutes covered renewing the certificate and reloading nginx, and automation that writes a file has changed only the first of those three. A renewal that succeeds against a process which never reloads is an outage with a green deployment behind it, and the only place all three are visible at once is a real handshake from outside the box.

Recap

A credential is checked by somebody with a list, and a key is used by code and checked by nobody. Rotating the first is a deletion from a list. Rotating the second leaves everything it encrypted or signed still needing it.

Encryption at rest answers what a stolen disk tells somebody, encryption in transit answers what a wire tells somebody, and neither says anything about a process that is allowed to read the secret. So secrets escape through logs, message payloads and crash dumps. Galewatch's key went into a log line 12,960,000 times a day from one farm, and the count is irrelevant: one copy is the whole loss. Deleting a leaked secret from everywhere it reached never finishes and can never be verified. Rotating it is one deletion, and that is the only reason rotation matters.

Rotation is lesson 052's expand and contract with a deadline, and only the last of its five steps takes anything away from anybody. Issue, accept both, move the clients, confirm the old one is unused, delete. Running the first four on a schedule without the fifth grows your list of accepted credentials and nothing else, which is most of what gets called key rotation.

Every store you can query tells you what is installed, and whether the old credential is in use is a fact that lives on the wire. A key id on the accepted request, counted on a graph you already have, is the whole of step four, and without it step five has no permission.

The last step finishes by itself only when the thing holding the old credential has an expiry. That is the real argument for short lifetimes, underneath the one about forcing automation: the contract step goes on a clock instead of on somebody's maintenance round. A token signing key rotates itself in fifteen minutes, and a key in flash on a tower waits for a technician who works for your customer. The bill sits on the other side of that sentence, because a credential's lifetime is how long you may be unable to reach the thing that mints it. Lesson 048's fail static buys you the fleet you have and nothing you are about to start, and a shorter credential shortens the fleet you have to minutes.

Envelope encryption moves a master key rotation from the bytes to the wrapped keys, and only if the wrapped key lives somewhere with an UPDATE. In a bucket it does not, so the natural place to keep it is the place that cancels the saving.

Check your understanding

1. The March round at Tarrow Ridge happens and fifty eight of the sixty turbines get the new key. Two are skipped because a hatch is frozen shut and the technicians will not be back until September. Lesson 029 asked how you would find out which turbines use a key, and today's key id answers that, so assume you have it. Say what you do about the old key, what you tell the farm's owner, and what would change your mind in July.

2. Galewatch cannot delete the key from the backups of its log index. State what the rotation fixes and what it does not, then change the logging so the next secret never lands there, and name what you give up.

3. Marlow's bucket write credential sits in an environment variable on both application boxes and in the founder's shell history. A rotation gets a new value into the environment variable. Say what it does about the shell history, what would have to change for step five to be honest here, and whether your answer differs for the credential that can only read.

4. Stagefront rotates its token signing key in a week with an on sale on the Tuesday. The key is asymmetric and its tokens live five minutes. Write the order of the steps against that calendar, say what breaks if somebody runs step five first, and how long the damage lasts.

5. A Stagefront box starts during a twenty minute outage of the service that issues credentials. Using lesson 048's test, which asks what a wrong value does rather than how old it is, say what that box should do, and what makes a missing credential different from the missing routing table 048 was arguing about.

6. Lesson 050's failover drill promotes the standby region, serves real traffic for twenty three minutes and passes, and no process there was restarted during it. Name what the drill did not test, the cheapest change that would test it, and what you expect that change to find the first time it runs.

Next lesson

054 Deployment Strategies: Blue-Green, Canary and Feature Flags. Today kept asking for a way to move clients across in a controlled order and watch what happens while they move, and lesson 054 is where that machinery finally gets built.