On Monday 14 September an email arrived at Galewatch, the telemetry service in this course that collects a reading from each of its customers' wind turbines every two seconds. It came from the asset manager at Ardnave Point, a farm of three hundred turbines, and it was blunt.
Your dashboard told my control room 97.4 percent availability for August. All month. It never once dropped below 97. The report you sent on 1 September says 96.7. My warranty says 97. Which of your numbers would you like me to take to my board?
Both numbers came out of the same database. Neither was produced by broken code. And the readings underneath them had been complete and correct since Tuesday 4 August, six weeks before the email.
Two things compute that availability figure. A streaming job reads the log lesson 037 built and keeps a running count per turbine, which is what feeds the tile in the customer's control room. A nightly batch job runs at 02:00, scans the previous day off the sharded readings tables lesson 014 routed, and writes one rollup row per turbine per day. The monthly report sums those rows. Lesson 037's fifth check question asked you to choose between exactly these two designs. Galewatch built both, which is the honest answer and also where this story comes from.
The first Saturday in August is lesson 041's Saturday. Ardnave Point's link was down from 05:40 to 13:10, its turbines held seven and a half hours of readings in flash, and when the link came back a writer process died with three million of them in a queue inside itself. The hole in farm time ran 05:58 to 11:31. An engineer found it on the Tuesday, the log still had the records because retention is seven days, and lesson 019's unique constraint on (turbine_id, recorded_at) refused the duplicates on the way back in.
So the raw readings were whole again on the Tuesday.
Nothing recomputed the Saturday.
Two jobs and one dial
A batch job has a bounded input. It starts, reads a set of data that is sitting still, writes an answer, and exits. Yesterday's orders. Last month's readings. The job knows when it is finished because the input ends.
A stream job has an unbounded input. It starts and does not exit. Records arrive, it updates whatever state it keeps, and it emits answers as it goes. There is no end to read to, so no moment when the answer is final.
Those sound like different crafts, and they are sold as different products by different vendors. They are the same computation at two settings of one dial, and the dial is how long you are willing to wait for data that has not arrived yet.
Galewatch's nightly job computes a day that ended at midnight, at 02:00, so it has two hours of slack it never chose, because somebody picked a quiet hour. The streaming job closes a ten minute bucket a couple of seconds after the bucket ends. Same subtraction, same division, two hours against two seconds. A batch job is a stream job that agreed to wait, and nearly everything that is hard about pipelines lives in that one number.
To talk about it you need two clocks, and this course already has them on every row. Lesson 034 gave every Galewatch reading a received_at stamped by the ingest box alongside the recorded_at the turbine wrote. The turbine's stamp is event time: when the thing happened. The ingest box's stamp is processing time: when your system found out. On a good day they differ by milliseconds. On 1 August they differed by up to seven and a half hours.
Every aggregation you will ever write picks one of those two clocks, and the choice is usually made by whoever wrote the first GROUP BY. Group by processing time and your buckets are always complete the moment they close, because a record cannot arrive before it arrives. They are also meaningless: the Saturday's morning wind ends up in a Saturday afternoon bucket, and a customer asking what happened at nine in the morning gets a flat line. Group by event time and the buckets mean what the customer thinks they mean, and you have signed up for the hard problem, because a record for nine in the morning can turn up at any point in the future.
Three hundred turbines at one reading every two seconds is 150 a second for the farm, so 12,960,000 readings in a day and 43,200 per turbine. On 1 August four turbines were down for a gearbox inspection, so the true availability is 296 of 300, which is 98.67 percent. The hole from 05:58 to 11:31 is 19,980 seconds, which is 9,990 readings a turbine and the three million across the farm that lesson 041 published.
So a turbine that turned all Saturday has 33,210 readings in the table out of the 43,200 the calendar expects, and 33,210 over 43,200 is 76.88 percent. Multiply by the 296 of 300 that were actually turning and the farm comes out at 75.85.
| Denominator, per turbine | Farm answer | Why |
|---|---|---|
| Expected from the calendar, 43,200 | 75.85% | numerator short by the hole |
| Readings the job actually had, 33,210 | 98.67% | hole cancels, top and bottom |
The batch job divided by what the calendar says a day holds, with a numerator missing nearly a quarter of the day, and published 75.85 percent. The streaming job had less again than the batch job did, 29,700 a turbine, because its buckets from 05:40 to 13:10 closed empty and nothing ever reopened them. It divided by what it had received, the hole cancelled out of both halves of the fraction, and it published 98.67, which is correct.
The careless number was right and the careful number was wrong. Hold onto that, because the reason is not luck in the direction it looks.
The hole cancelled because the entire farm's link went down, so available readings and stopped readings disappeared in exactly the same proportion. A ratio over received data is correct precisely when the data you are missing is missing at random, and that is the assumption nobody writes down.
Change the failure and the arithmetic inverts. One turbine's radio dies on a Tuesday and sends nothing all day. The streaming job divides 299 available turbines by the 299 it heard from and reports 100 percent, on a day a turbine was dark. The batch job divides by 300 and reports 99.67, which is the conservative answer and the one a contract can be written against.
The bias runs one way, and it is the wrong way. A turbine that has faulted is likelier than a healthy one to be a turbine whose radio or controller has stopped too, because the fault and the silence often share a cause. The readings you are missing lean towards the unavailable ones, so a ratio over what arrived is not merely noisy. It is optimistic, systematically, in the direction nobody audits it in. Silence makes your availability number go up.
The watermark nobody knew they had set
A watermark is a promise. When a pipeline says its watermark has reached 10:00, it is asserting that it believes it will see no more records with an event time before 10:00, and that any bucket ending at or before 10:00 may now be closed, emitted and forgotten. The word collides with lesson 040's high water mark, which is a position in a sequence of rows; this one is a position in time, and it is a guess rather than a fact.
Pick the watermark too tight and you emit numbers that late records would have changed. Pick it too loose and the tile in the control room is an hour behind, which at Galewatch means the product does not work, because freshness is the product. Lesson 008 said it plainly: sixty seconds of cache on a turbine panel is sixty seconds of lying about a machine somebody may be standing next to.
What should Galewatch's watermark be? There are two things it has to cover, and the course has measured both.
The first is transport. A reading sits in a turbine's flash while the link is down, and lesson 041 measured seven and a half hours of it at this very farm.
The second is the clock on the turbine, and this is the half teams forget. Lesson 034 ran a query across the fleet and found a median offset of four tenths of a second, 2.1 at the ninety fifth percentile, 9.6 at the ninety ninth, and a worst case of 41 seconds on a turbine that had been reporting cheerfully for two years. A turbine whose clock is 41 seconds slow writes recorded_at 41 seconds in the past. Its reading arrives promptly, carrying an event time from three quarters of a minute ago, late through no fault of the network.
So a two second watermark on a ten minute bucket is not a clean pass or fail. A turbine slow by d seconds loses, out of every single bucket, whatever it generated in the d minus two seconds after that bucket's end, because those readings carry an event time inside the bucket and arrive after it was emitted. At the median four tenths of a second, nothing. At the ninety fifth percentile's 2.1, a one in twenty chance of a single reading. At 41 seconds, a 39 second slice of every bucket, which at one reading every two seconds is about twenty readings each time against the 300 a bucket holds: 6.5 percent of everything that turbine ever says, gone on an ordinary Tuesday with nothing broken anywhere.
Here is the part I did not expect when I started pulling on this. Lesson 037 routes a reading to the backfill topic when its recorded_at is more than a minute behind its received_at, and the team that wrote that rule in July was solving a latency problem: they wanted four hours of replayed flash out of the way of the reading that just arrived. That rule is a watermark. Sixty seconds, chosen for a different reason, by people who would not have used the word. And lesson 034's band says up to 41 of those 60 seconds can be eaten by clock error alone, leaving nineteen seconds of real transport slack on the worst turbine in the fleet.
It misclassifies in both directions, and the size of the error is exactly 034's distribution. A reading that spent thirty seconds in flash, on a turbine whose clock is 41 seconds slow, has a gap of 71 seconds and goes to the backfill lane as though it had spent hours there. A reading that spent 45 seconds in flash on a turbine with a good clock stays in the live lane and arrives after its bucket closed.
And nothing in 034 says which way those clocks run. DM-11's was slow, by the four point two seconds between its own panel and the technician's laptop, but 034's cause was a daemon that had not reached a time server since the day the turbine was commissioned, and an unattended crystal drifts whichever way it likes. So assume some of that fleet is fast. A turbine 41 seconds ahead stamps a recorded_at in the future, so its reading never looks late to anybody's rule and lands in a bucket that has not opened yet. A watermark protects you from late records and nothing at all protects you from early ones. It will be counted, in the next bucket, up to ten minutes after the minute it belongs to, and no graph anywhere has a colour for that.
There is a way to stop guessing, and Galewatch has had the data for it since June. Lesson 034's offset estimator takes the gap between the two columns and keeps the minimum per turbine over a day, on the reasoning that the minimum is the clock offset plus the smallest transit any reading achieved. Take the same subtraction on the same two columns and read the other end of the distribution, and you have your watermark, measured instead of guessed. The minimum tells you how wrong the turbine's clock is. A high quantile tells you how long you have to wait.
Run it at Ardnave Point over the week of 1 August and the answer is brutal. The week holds 90,720,000 readings, and 4,050,000 of them came out of the Saturday's flash, all stamped received_at of about 13:10, with event times spread evenly back to 05:40. So the gaps in that batch run from zero to 27,000 seconds at 150 readings per second of event time. Every other reading the farm sent that week arrived within milliseconds of being taken, which is the assumption the next number rests on and the reason it is so lopsided: the entire top of the distribution is one Saturday afternoon. The top one percent of the week's gaps, 907,200 readings, is everything above 20,952 seconds.
Five hours and forty nine minutes. That is the ninety ninth percentile of one real week's lateness at one farm, and if you sized an honest event-time watermark from it the control room would be looking at a number from before breakfast.
Which is why the lane split exists, and it is doing something subtler than it looks. By routing anything more than a minute late into a separate topic, lesson 037 truncated the live topic's lateness distribution by construction. The live stream gets a watermark it can defend, because the records that would have broken it have been moved somewhere else. The price is that the fast path is now knowingly incomplete, which is a very different thing from accidentally incomplete, and it is fine for a tile and fatal for a contract.
Lesson 046 reached this same trade first, in another room: a write path can forget, a change stream cannot, and the change stream is never quite now. Generalise it past caching, which is what today is for, and you get the trade this whole lesson turns on. You may have the number now, or you may have all of it. Time is the dial between them, and there is no setting that gives you both.
Re-running a day
Both of Galewatch's jobs were blind to the Saturday, for different reasons, and the difference matters.
The streaming job's rollup for the morning buckets was closed and emitted long before 13:10. Subscribing it to the backfill topic as well would not have saved it: the records would arrive, the watermark would already be hours past them, and a bucket that has been emitted and dropped from memory cannot be amended by a job that no longer remembers it exists.
The batch job had no such excuse. It ran at 02:00 on Sunday 2 August, read the readings tables as they stood, and wrote a Saturday rollup row of 75.85 percent. On the Tuesday the three million readings came home. The row did not change, because nothing read the readings again.
Four weeks later the monthly report summed thirty one rows, one of which had been written two days before the data it described was repaired. A 22.8 point error on one day of thirty one is 0.74 points on the month, and 0.74 is the whole distance between the 97.4 the control room watched all August and the 96.7 that reached the customer. One Saturday, one job nobody re-ran, and a warranty threshold sitting between the two answers.
A derived number is only as fresh as the last thing that recomputed it. Your raw layer can be perfect, repaired, audited and whole, and every screen in the product can still be showing an answer from the morning the data was broken. The pipeline's correctness is not a property of the data. It is a property of whether anything has read the data since.
So you have to be able to re-run a day, which brings lesson 019 back at a bigger scale. Reprocessing means running the same computation again over a window you have already computed, because the inputs changed underneath you. It is idempotency with a day's blast radius instead of one request's, and 019's three piles sort it exactly as they sorted a payment.
A rollup written as an upsert keyed on (turbine_id, day), computing the whole day from scratch and overwriting whatever was there, is in 019's first pile. Free. Safe to run four times. It is 019's own fix for the publisher payout job, which writes each CSV to object storage under a key of (publisher, month) so that writing it twice overwrites harmlessly, applied to a number instead of a file.
A rollup written as available = available + :n is in the second pile at best, and in practice nobody builds the fence. 019's rule is the ten second test: a conditional update is idempotent when its own effect falsifies its condition, and adding to a counter does nothing whatever to the condition that let you add to it. Lesson 036 is the other half of why this matters. At-least-once delivery means your stream job will see some records twice, so an incrementing aggregation is not just unsafe to re-run on purpose, it is quietly wrong all the time, by an amount nobody can measure.
Choose an aggregation whose second application changes nothing, and your delivery guarantee stops being your problem.
Two more things, both cheap, and one of them would have caught this in August.
Store the counts, not the rate. Galewatch's rollup row held a percentage. A percentage has thrown its denominator away, so you cannot re-weight it, you cannot audit it, and you cannot tell a 75.85 that means a quarter of the turbines were stopped from a 75.85 that means a quarter of the readings never arrived. A row carrying expected 43,200, received 33,210, available 33,210 has the whole story written on its face, and the gap between the first two columns is an alert a junior engineer could write. One such row is a turbine id, a date and three integers, call it sixty bytes, against a turbine's 43,200 readings at lesson 002's two hundred bytes each, which is 8.64 megabytes. One part in a hundred and forty four thousand, for the ability to prove your own report.
Then mark the days you have dirtied. When the backfill writer accepts a record whose event time falls in a day that has already been rolled up, it inserts (farm, day) into a small table, and the rollup job drains that table before it does anything else. The Tuesday recovery would have put one row in it. Re-running one farm-day means reading 12,960,000 rows at 200 bytes, which is 2.6 gigabytes. Ardnave Point is the one farm that does not get lesson 014's promise that a farm's questions land on a single box, because at two hundred turbines it stopped fitting one shard and lesson 031 widened its key to (farm, turbine) and split it over three machines. So the re-run is three parallel scans of 864 megabytes instead of one of 2.6, and three of the nineteen machines instead of all of them. The cheapest query in this entire story.
Both at once, and the bet nobody tests
What Galewatch has is the shape everybody builds, usually without deciding to. A slow, complete path that recomputes from the stored history. A fast, incomplete path that covers the time since the slow one last ran. Something in front that serves whichever one the screen needs. Nathan Marz named that the lambda architecture, and for a few years it was the default answer to this problem.
The criticism arrived from Jay Kreps in 2014, and it was about staffing rather than architecture. You are maintaining two implementations of the same business rule, in two codebases, with two sets of bugs, and you are betting they agree. Nobody tests that bet. The alternative that got named kappa drops the batch layer: keep one stream job, and when you need to reprocess, replay the log from an earlier offset through a second copy of the same code, then cut over.
One implementation instead of two is worth a lot. Read the precondition, though, because it is the whole thing: your log has to hold enough history to rebuild the answer. Galewatch's log holds seven days. A monthly report cannot be rebuilt from seven days, and a customer contract that quotes a figure for last March cannot be rebuilt from it at all. Lesson 037 said retention is how long you are allowed to be wrong for, and kappa asks you to extend that to the lifetime of every answer you have ever published.
So Galewatch's batch layer reads Postgres instead of the log, which is neither of the two shapes with a Greek letter and is the right answer for a company holding a year of readings on lesson 031's nineteen machines. The useful question is never which architecture. It is whether the thing that recomputes your history can reach your history.
What I would actually ship, the Monday after that email, is two things, and neither of them is a rewrite.
Compute the same number both ways on purpose, and graph the difference. Not the two numbers, which will be watched by nobody after the first fortnight: the difference, with a threshold on it. Lesson 024's GitHub failover in October 2018 turned 43 seconds of partition into 24 hours and 11 minutes of degraded service because recovery reconciles decisions rather than packets, and that day of reconciliation was on nobody's calendar. A pipeline running two paths gets to schedule its reconciliation nightly, for free, and most teams never look at the answer. Galewatch's would have fired on Sunday 2 August, with a 22.8 point gap on one farm.
And put the provenance on the glass. Lesson 012 argued for showing the staleness next to the reading rather than hoping, lesson 023 put that argument into the vocabulary, and lesson 034 drew the clock's error band around the numbers on the panel. The same move works here, and it costs less than anything else in this story: say which path produced the number, and what it is complete through. Most teams who run both paths cannot tell you which one the number on the screen came from, and no tool is missing: nobody ever had to say it out loud, so nobody built the field for it.
There is one test for what a pipeline is allowed to feed, and lesson 050 wrote it a lesson early for a different reason: a decision may move away from its home only when it is revocable or its data has an owner where you moved it. Read that as a rule about derived numbers and it sorts Galewatch's screens in about a minute. An availability tile is revocable, so serve it fast and label it. A contractual monthly report is not, so it waits for the complete path and it carries its completeness with it.
Stagefront, the ticketing service in this course where a stadium show goes on sale at ten in the morning and two hundred thousand people press the same button, sits at the far end of that test. Lesson 024 kept its seats on one machine on purpose so there would be no second copy with an opinion. The seat map a buyer is looking at is already ten seconds old, by the TTL lesson 008 set, and lesson 009 defended that in words worth borrowing today: a value that changes faster than anybody can read it, and that nobody is allowed to decide from. The second half of that clause is the rule for pipelines. Its revenue reporting, by contrast, is perfectly ordinary pipeline work. Same company, same records, and the line between them is whether somebody takes an irreversible action on the answer.
Marlow Books, the online bookshop run by four people that has supplied most of this course's bad mornings, should build none of this, and saying why is the useful half. Its gift page is an aggregation over seven days of order lines that takes about 900 milliseconds, wrapped in a sixty second cache since the shop took five requests a second. That is a batch job with a schedule of one minute and a watermark of zero, and it is correct, and nobody needs to call it a pipeline. The publisher payout job is a batch job over a year of order lines that 019 already made safe to run twice. Two pipelines, both already built, neither of them a project.
Recap
A batch job is a stream job that agreed to wait. The difference between them is one number, how long you hold a window open for data that has not arrived, and every genuinely hard thing about pipelines lives in that number rather than in the choice of tool.
Event time is when it happened and processing time is when you found out. Group by processing time and your buckets are always complete and never mean anything. Group by event time and they mean what the customer thinks, and you have signed up for records that arrive after you answered.
A watermark is a promise about your slowest producer, and you can measure yours. Lesson 034's offset estimator is the minimum of the gap between the two stamps; your watermark is a high quantile of the same gap on the same two columns. Galewatch's ninety ninth percentile for one real week was five hours and forty nine minutes.
A watermark protects you from late records and nothing protects you from early ones. A clock that is fast puts a reading in a bucket that has not opened, and it will be counted, in the wrong place, silently.
A ratio over received data is right only when the missing data is missing at random, and the one thing you know about missing telemetry is that it is not. A faulted turbine is also a turbine whose radio may have failed, so silence makes your availability number go up.
A derived number is only as fresh as the last thing that recomputed it. Galewatch's raw readings were whole on 4 August and its report was wrong on 14 September, because repairing data is not the same event as recomputing what was derived from it.
Reprocessing is lesson 019's idempotency with a day's blast radius. Make the rollup an overwrite of a recomputed whole, keyed on the window, and the second run is free. Make it an increment and at-least-once delivery has been lying to you all along.
Store the counts, not the rate. A percentage has thrown its denominator away, so it cannot be re-weighted, audited, or distinguished from the other reason it could have come out that low.
Alert on the difference between the two paths, not on either number. If you run both, you are maintaining one business rule twice and betting they agree. A nightly reconciliation is the only form of that bet anybody could check, and most teams never look at it.
Check your understanding
Marlow's founder wants the gift page's seven day bestseller list to update within a second of a sale instead of once a minute, and proposes a stream job over the shop's orders instead. Price what that buys using 008's figures, say what it costs him on an ordinary Tuesday, and say what you would ship instead.
You inherit a nightly rollup job that has run green for two years, and discover its source topic has a backfill lane nobody told the job about. Say what you would check first, in what order, and what single number you would have on a graph by the end of the day.
Somebody at Galewatch proposes setting the event-time watermark to eight hours so that no reading is ever late. Make the strongest case for it, then say what it does to the product, which earlier lesson owns the thing that breaks, and what you would do instead.
Stagefront wants a live "seats sold" counter above the stage map and an end of day settlement figure for the promoter. Say which of the two a stream job may compute, which it may not, and what you would require of whichever one touches money before it ships.
Your reconciliation graph between the batch and stream numbers has sat at 0.3 percent for six months and nobody can explain it. Say what you would do about the 0.3, then say what you would do differently if the graph read exactly zero.
A colleague wants to delete Galewatch's batch layer and reprocess from the log instead. Give the one question that decides it, say what the answer is for this company, and say what you would build in the gap that answer leaves.
Next lesson
052 Schema Evolution and Backward Compatibility. Today's pipeline has two jobs reading the same records and computing the same rule, which is survivable right up to the morning somebody adds a column and only one of them gets deployed.