Audit Logs and Immutable Records

Galewatch is a telemetry company that exists only in this course. It takes a reading every two seconds from each wind turbine on the farms it monitors and sells the owners dashboards. On Wednesday 28 October 2026 a loss adjuster asked it for one sentence, and the company spent the afternoon finding out that it could not write that sentence.

The weekend before is lesson 058's. Tarrow Ridge, the sixty turbine farm lesson 002 measured, lost its link at 06:20 on Saturday 24 October and got it back at ten to ten on the Sunday. Each turbine held 16,384 readings in flash, stopped, and came home with the beginning of the storm and none of the end. A blade had been damaged, so the farm's owner asked for the weekend on the Monday, because their insurer wanted the two second record, which is the thing Galewatch sells.

The report went out on the Tuesday and it was a good one. Honest about the hole, with lesson 051's three columns on every turbine's day, and a note saying the Saturday runs to 15:26 and holds 64 percent of the day.

The adjuster read it and asked for something else, and it was not about the hole.

They wanted a sentence confirming that the readings in the report were the readings the turbines produced, and that nobody at Galewatch had changed them since.

Here is what the company has. The broker lesson 037 put between the ingest boxes and the database, holding seven days. Nineteen Postgres shards. One base backup of each shard, taken every Sunday night into an object storage bucket that lesson 044 found wearing a lifecycle rule which deletes anything older than twenty eight days. And lesson 051's rollup rows, one per turbine per day.

Four copies of that Saturday. Not one of them can answer the question.

What makes it worse is three weeks old. On Thursday 8 October a test rig running the shipping image wrote seven million two hundred and ninety thousand readings into production overnight, all carrying real Ardnave Point turbine ids and all invented, and lesson 057 went looking for a predicate that would separate them from the genuine ones and reported that there isn't one. So Galewatch does not merely lack a proof. It holds, as a matter of its own record, several million readings no turbine ever made, in the same tables an insurer is reading.

Somebody worked out on the Wednesday afternoon that the broker still had the window, and that the window was closing. The Saturday morning's live readings arrived on the Saturday and age out on the Saturday coming. The replay arrived at ten to ten on the Sunday and ages out on the Sunday, four days away. Lesson 037 also split those two apart, because a reading more than a minute behind its arrival time goes down the backfill lane, so one weekend's evidence sits in two topics with two different clocks on it and nobody had ever noticed.

Forty eight hours at a reading every two seconds is 86,400 a turbine, of which 33,116 never came home, so 53,284 did. Times sixty turbines is 3,197,040 readings, and at lesson 002's two hundred bytes each an engineer copied about 640 megabytes into a bucket before going home.

Nothing in that copy is false. Nothing in it is evidence either.

A copy is not a witness

An audit log is a record of who did what and when, kept so that somebody other than the person who did it can check. The record, the keeping, and the somebody else: three parts, and it is the third that gets dropped, because the first two look like logging and only the third says anything about who can edit the result.

The word for the property you want is tamper-evident, not tamper-proof. You are not going to stop somebody holding your credentials from changing your data. You are going to arrange that the change leaves a mark.

Out loud, since the spoken version skips tables. The only copy that knows when anything arrived has the shortest life of the four. The shards hold the current answer and no record of yesterday's. A base backup is a photograph of the answer rather than a record of the question. And the rollup is arithmetic over the rows, so whatever is wrong with the rows is wrong with it, faster, in the form a customer reads.

What they have in common is the thing that matters. All four were written by the same company, with the same credentials, out of the same stream of assertions. Replicating something four times answers one question very well: is the data still there. It says nothing about the other question, which is whether the data is still what it was. A copy tells you the bytes survived. A witness tells you the bytes did not change. No copy anywhere in this course was bought for the second question, lesson 050's second region included, and nobody was wrong to buy any of them.

So the adjuster is not really asking about 3.2 million rows. They are asking who could have changed those rows, and lesson 053 published Galewatch's answer already: one ingest key, shared by every turbine, with no expiry and no way to find out how many copies exist. Integrity is a property of your system, not of your data. You cannot look at a row and tell.

Which leaves retention. Lesson 037 said retention is how long you are allowed to be wrong for. Turn it round and it is also how long you are able to answer, and that half nobody prices. A retention policy is a deadline on a question nobody has asked yet.

Lesson 037 even published the shape of the deadline. A partition on disk is a run of segment files, and retention closes a segment and later deletes the whole thing, so you always have a little more history than you asked for and never less. For capacity planning that is a gift. For a deadline it is the worst property available, because you can neither promise the extra nor measure it, so the only honest planning figure is the seven days.

Nobody at Galewatch ever decided to destroy the arrival record of that weekend. Somebody typed seven days into a configuration file when the broker went in, because a week is the usual starting point, which is lesson 037's own phrase for it. The retention nobody chose is the delete nobody authorised, and it runs on a schedule, with no operator, leaving no line anywhere saying it happened.

Every immutable store sits on something mutable

Fifty eight lessons have treated immutability as a property of storage, and each was right in its own frame. Lesson 016 closed on an immutable file with a hashed name being the most cacheable object in computing, lesson 022 gave it a year and the immutable directive, lesson 037 built a log that cannot be rewritten, and lesson 039 found that an append needs no compensation.

Now take the claim at Galewatch, that a reading once written is never changed, and ask what it is resting on.

Nothing refuses an UPDATE. No constraint in that schema would turn one down, and lesson 043 says what happens when you run it, in a sentence it credits to lesson 010: every update is an insert, so the old version is still on the page, present but dead. Which looks like a free history and is not, because lesson 043 published the cleaner in the same breath. A dead version survives only until vacuum reaches it, and lesson 011 put autovacuum's trigger at fifty rows plus a fifth of the table, which for Marlow's 1.2 million titles is about a month of mornings. The previous value is there for a length of time nobody has ever set, kept by a mechanism whose entire purpose is to remove it. Vacuum and compaction are the same bill with different handwriting, which is lesson 043's line, so an LSM tree gets there too: compaction reads several runs and writes one, dropping superseded versions as it goes.

Go down a layer. The backup's twenty eight days is a lifecycle rule, and lesson 044 was specific about those. You declare them on the bucket and the store applies them on its own schedule, roughly daily, some time after the day you asked for rather than on it, with no expiry you can set on a write and nothing on the write path at all. The same lesson published what else that bucket will do: a delete is honoured immediately and durably, a lifecycle rule with a typo in its prefix deletes at eleven nines of reliability, and versioning keeps the old copy only if somebody turned it on and accepted the bill.

Down once more, to the credential, which is lesson 057's third tier: the authority that takes away your own ability to recover. A delete, a revocation, a re-encryption. The shortest path to destroying data is not an UPDATE at all, because lesson 053 published that revoking an encryption key destroys what it protected without touching a single row.

Three layers down, and under every one of them is something somebody at the company can change.

You cannot make a record unchangeable. You can only make a change detectable. The design question is therefore never whether the rows can be edited, because they can. It is who would be able to tell.

One real exception exists and lesson 044 already covered it, so take it as placed rather than taught again: Object Lock in its compliance mode gives you write-once storage that cannot be deleted by anybody, including the account root, for a retention period you set in advance. That is genuine immutability and it is the only kind for sale.

Then read its two conditions back, because they are the price. You set the period in advance, which means deciding how long this matters before knowing what it is for. And a record you cannot change is a record you cannot fix: a reading written wrongly, a name a customer has asked you to erase, a field somebody filled in with a fiction. Lesson 052 found the general version from the other end, that when you add a column the value you backfill is a claim about history. Write-once storage is where a wrong claim about history becomes permanent on purpose, and lesson 084 owns retention as an obligation rather than a setting, so it will have to live in that corner properly.

A chain tells you about the middle

The cheap mechanism is a hash chain, and it is cheap enough that the interesting part is where it stops.

A hash is a function that turns any number of bytes into a short fixed-length value, such that changing one input byte changes the output beyond recognition and nobody can work backwards from the output to the input. Give every row in your log a sequence number and two more fields: the hash of the row before it, and its own hash computed over its contents including that field.

seq  | actor      | action               | prev     | hash
-----+------------+----------------------+----------+------
4471 | ingest-03  | config push, farm 7  | a91c7f.. | 55d8e1..
4472 | svc-rollup | backfill 2026-10-24  | 55d8e1.. | 0e37ba..
4473 | eng-4      | key id 4 accepted    | 0e37ba.. | bd904c..

Each row's own hash is the next row's prev, so the rows are welded into a sequence rather than piled in a table. Only prev has to be stored. A row's own hash is computed from the row, so whoever checks recomputes it, which is why the chain costs one hash a row and not two.

That is the third field this course has asked for on a write, and the three are worth naming together, because each buys one property and nothing else. Lesson 019 said a client should mint an idempotency key, which makes a write nameable. Lesson 058 said to make that key a counter rather than a random string, which makes a missing write noticeable, because a gap in a sequence is a question and a missing random string is nothing at all. Today adds the hash of the previous row, which makes a changed write noticeable. Nameable, missing, changed, and the third costs thirty two bytes.

Now price it in the obvious place and watch the answer come out wrong. Tarrow Ridge is sixty turbines, and lesson 051 put a turbine's expected day at 43,200 readings, so the farm produces 2,592,000 readings a day. At two hundred bytes each that is 518.4 megabytes, and thirty two bytes on every one of them adds 82.9 megabytes a day. Sixteen percent. Not free, and not a reason to stop either.

It is still the wrong thing to build.

A hash is computed by whatever writes the row, and at Galewatch that is an ingest box. Lesson 057 drew the trust boundary in the only place it can be drawn, which is that the ingest key sits in flash on a tower in a field, and the rig that fabricated thirteen and a half hours of weather held a legitimate copy of that key and used it correctly. Chain the night of 8 October and you get an immaculate chain over 7,290,000 readings no turbine made. Every link verifies. Every hash is right.

A chain protects a record from you. It does not protect it from the person the record is about. It proves nobody edited history after it was written down, and says nothing about whether what was written down was true. The fix for that night is still lesson 034's counter, taken from the turbine rather than from a clock, and lesson 057 had already said so.

So go the other way and put the thirty two bytes where the question gets asked. Give Galewatch one log with a row per administrative action: a config push to an ingest box, a backfill request, a key accepted, a routing table edit, a lifecycle rule changed, a deploy.

Lesson 038 left the rule for what goes in one: an event's payload should be what you would want in a log line a year from now. There it was a style preference. Here it decides the columns, which means the actor, and the value before as well as the value after, because a row saying somebody changed the lifecycle rule is worth nothing next March beside one saying they changed it from twenty eight days to fourteen.

Be generous and call it a hundred rows a day, which is more than that company has ever managed in a day. Thirty six thousand five hundred rows a year, at two hundred bytes, is 7.3 megabytes.

One part in seventy of a single farm's single Tuesday.

The record of every decision anybody ever made is smaller than one day of your product, and that arithmetic decides where tamper-evidence goes. It is also why the instinct to make the data immutable is so often a way of avoiding the cheap thing. The data is enormous, chaining it costs sixteen percent, and no auditor was ever going to ask about a reading. They ask who changed the rollup job.

One structural rule goes with it, and lesson 040 settled it for messages. A record written after the action is a record that can be missing, and you cannot deduplicate a message that does not exist. Write the fact and the row about the fact in one transaction, the way 040's outbox does, and the pair either both happened or neither did. An audit log is an outbox whose consumer is a question somebody asks next year.

Then the limit, which is where most writing on this stops.

A chain catches an edit in the middle. Change row 4,472 and its hash changes, so 4,473's prev no longer matches and every row after it is visibly wrong.

Delete row 4,471 and everything after it, though, and what remains is a perfectly valid chain. Truncation leaves no mark whatever, because a chain has no opinion about where it ought to end. And if you hold the whole table you need not even truncate: rewrite 4,472, recompute 4,473 and every row after it, and the chain verifies end to end, since you also own the job that does the verifying.

A chain proves the middle and says nothing about the end, and the end is exactly where somebody who has just done something is standing.

Who would be able to tell

Which makes the chain the cheap half rather than the mechanism. The mechanism is one number, somewhere you cannot reach.

Publish the head: the hash of the latest row, as a line in a file, once a period. Anybody holding yesterday's head can check that today's chain still contains yesterday's history, because the only way to produce a chain whose row N carries that hash is to hold the same rows up to N, and producing a different set with the same hash means breaking the hash function rather than editing a table.

An anchor is a copy of one number that you cannot edit. The paid versions are a second account owned by somebody who does not run the system, which is cheap and sits inside the same blast radius the moment one credential governs both, so lesson 057 tells you to go and check rather than assume; a different provider, which costs a dependency and buys less correlation, lesson 050's trade in miniature; or lesson 044's Object Lock, compliance mode, on a daily file holding one line, which is the one place where write-once storage costs nothing, because you will never need to correct a hash.

Then the free one, which is the best idea here. Every report Galewatch sends a farm owner or an insurer is a copy of a claim leaving the building. Put the chain head on it. From the moment that adjuster holds the number, the Saturday cannot be rewritten without the adjuster being able to notice, and Galewatch has paid nothing, administers nothing, holds no new credential. Your counterparty wants to be your witness. They are keeping the receipt anyway, for their own reasons, which is precisely what makes them useful.

Which is the answer to the Wednesday. The sentence the adjuster wanted is one Galewatch could have written in a minute, if the previous week's report had carried a line nobody would have read.

Cadence is the only dial. Your tamper window is the gap between anchors. Anchor nightly and somebody holding the database can rewrite up to a day of it and pass verification; anchor hourly and it is an hour. No choice of hash function moves that number, which is why a design review arguing about which one to use is usually arguing about the wrong thing.

And a chain nobody recomputes is decoration. Lesson 013 made Galewatch run a restore drill, which is how the company learned that its real restore time was six hours and twenty minutes against a contract that said one. Same move here: a job that walks the chain from the last anchor and compares, on a schedule, with a person woken up when it disagrees. An unverified chain is a backup nobody has restored. It fires for the first time on a day when nothing is wrong, and that is how you find out whether it works.

Marlow Books is an online bookshop run by four people that exists only in this course, with 1.2 million titles and a founder who does the deploys on Friday afternoons. It has none of this and needs about an afternoon of it.

Lesson 029 told the story and left the debt sitting in its own text. On the second Saturday in December one of the temporary Christmas staff, sent the admin link by a buyer in November, opened it on their own phone, typed a sentence about the courier and pressed Closed at 10:40, reading the word the way anybody in a warehouse reads it. The banner was good enough that the shop looked like it was having a considered day, and bought forty five minutes of silence.

Nothing anywhere records who pressed it.

Here is the part worth the afternoon. That switch has two mechanisms, and only one of them can ever be recorded.

The admin form is an HTTP request through a handler that knows the session's customer id, and lesson 026 says the shop already writes a line per request with a path, a status, a byte count and a duration. Add the signed in customer id to that line and the December Saturday becomes answerable for the cost of one field. Lesson 026 asked for one other field on that same line for the same kind of reason, an id that lets any two lines be joined later, so this is the second entry on a list the founder already has.

The other mechanism is lesson 027's, chosen deliberately: two fields in a file on each box, changed in thirty seconds over ssh with no deploy, so they survive the mode where the database is the problem. No request, no handler, no line. Lesson 057 priced the authority on it as shell access, which at a four person company is everybody.

An audit log covers the mechanism somebody built, and an incident uses the mechanism somebody kept.

Resist the obvious repair, which is to route the emergency path through the application so that it can write a row. Lesson 027 picked the file precisely so the switch would work while Postgres was the broken thing, so an audit row that goes to Postgres is a row that is missing on the one afternoon you will want it. That is lesson 057's break-glass argument arriving at the record instead of at the credential: the receipt for an emergency action cannot depend on the thing the emergency is about. What survives is a line appended to a file on the box that noticed the change, shipped off the box afterwards.

That buys two thirds of what you wanted. The box can see the value go from open to closed at 10:40 and cannot see whose hands were on the keyboard. A record of an effect is not a record of an actor, and a good deal of what gets called an audit log is a record of effects with a hopeful name. The actor only ever arrives by making the action a request.

Lesson 057 asked you to write down, for each switch you own, the first irreversible thing it can start. The companion takes ten minutes: write down which of your switches can be reached by a path that writes nothing.

Recap

A copy tells you the bytes survived. A witness tells you the bytes did not change. Galewatch holds one weekend four times over and not one of those copies is a witness, because all four were written by the same company with the same credentials out of the same stream of assertions. Integrity is a property of your system rather than of your data, which is why nobody can answer the question by looking at a row.

A retention policy is a deadline on a question nobody has asked yet. Lesson 037's seven days is how long you are allowed to be wrong for and also how long you are able to answer. The retention nobody chose is the delete nobody authorised: on a schedule, with no operator, leaving no line saying it happened.

Every immutable store sits on something mutable. A table with no constraint against an UPDATE, a cleaner whose job is removing the previous version, a lifecycle rule applied on somebody else's schedule, a bucket that honours a delete immediately and for good, a key whose revocation destroys the data without touching a row. You cannot make a record unchangeable, only make a change detectable, and the one thing you can buy is lesson 044's write-once storage, whose two conditions are that you choose the period before knowing what it is for and that a record you cannot change is a record you cannot fix.

Nameable, missing, changed. Lesson 019's minted key, lesson 058's counter and today's hash of the previous row each buy exactly one of those. Thirty two bytes for the third, which on Tarrow Ridge's 2,592,000 readings a day is sixteen percent and on a log of a hundred administrative actions a day is 7.3 megabytes a year, one part in seventy of that same farm's single Tuesday. The record of every decision anybody made is smaller than one day of your product, so chain the decisions and leave the data alone.

A chain protects a record from you and not from the person the record is about. Chain the night of 8 October and every link verifies over several million readings no turbine made, which is why lesson 034's counter is still the fix for that night and a chain never was.

A chain proves the middle and says nothing about the end. Truncate it and what is left is valid; hold the whole table and you can rewrite and recompute. So the mechanism is the anchor rather than the chain, your tamper window is the gap between anchors, and the cheapest anchor in any business is the counterparty keeping your report anyway. An unverified chain is a backup nobody has restored.

An audit log covers the mechanism somebody built, and an incident uses the mechanism somebody kept. Marlow's kill switch is a form and an ssh session, and only the form can ever be recorded. A record of an effect is not a record of an actor, which is most of what the phrase gets used for, and the actor arrives only when the action is a request.

Check your understanding

1. Galewatch decides to put a chain head on every report it sends a customer. Lesson 058 already asked what belongs on that weekend report; say instead which documents count as a report for this purpose, what you do when one has to be reissued because the first was wrong, and what the customer has to be told for the number to be worth printing.

2. Someone proposes chaining the readings table itself so that 8 October cannot happen again. Give the strongest version of their case, say what the chain would and would not have caught that night, and say what you would spend the sixteen percent on instead.

3. You are asked to keep the arrival record for a year rather than seven days, and lesson 037 put a fleet week at 75 gigabytes. Price the obvious version, then say what you would keep instead of everything, which question your cheaper version can no longer answer, and how you would have answered the adjuster with four days of retention left.

4. Marlow's founder agrees to add the customer id to every access log line and asks how long to keep those lines, which lesson 026 put at about a gigabyte a day. Give him a number, say what decides it, and say what you would do about the fact that today's answer is whatever log rotation shipped with the box.

5. Stagefront, the ticketing service in this course where two hundred thousand people press the same button at ten in the morning, has a seat claim whose audit is a solicitor's letter. Say what has to be in the record of a claim for it to be useful in that letter, which of today's mechanisms you would skip there and why, and who holds the anchor.

6. A customer exercises a right to have their details erased, and the rows naming them sit inside a chained log under a two year write-once retention somebody set last March. Say what you would actually do, what you would have designed differently in March, and what the chain can still prove afterwards.

7. Take a system you work on and name an action somebody could take today that you could not later prove had happened. Say whether the gap is in the record, in its retention or in who can edit it, and what the cheapest honest fix is.

Next lesson

060 Phase 2 Capstone: Hardening the Phase 1 Service. Thirty lessons of mechanisms land on one small shop at once, and today supplies the part a capstone usually leaves out, which is how anybody would later prove the hardening was ever switched on.