Stagefront is a ticketing service that exists only in this course, where a stadium show goes on sale at exactly 10:00 and two hundred thousand people press the same button inside a minute. Its failover plan ran to eleven pages and had never once been run. At 02:00 on Sunday 13 September 2026, somebody finally ran it.
The drill was booked for an hour and took twenty three minutes, most of which was somebody reading the pages out loud. The seat database's standby, which has sat in a second region for a year and a half, came up as primary in ninety seconds. DNS carried the service name over on a sixty second record, which lesson 004 would have told them to check and which somebody had. By 02:06 every graph was green: requests served, errors flat, nothing paging. The engineer on call wrote PASS in the ticket and went back to bed.
What nobody opened until Tuesday was the latency panel. A purchase at two in the morning used to finish in about two hundred and thirty milliseconds, a hundred and fifty of which lesson 004 measured at the payment provider. From 02:06 it took one and a quarter seconds. Same code, same query plans, same indexes, nothing deployed, nothing broken. Nine purchases a minute at 1.2 seconds draws the same line on a throughput graph as nine purchases a minute at 230 milliseconds, and at 02:06 on a Sunday nobody was looking at the other graph.
Lesson 003 published the one Stagefront promise no yearly average is allowed to absorb: at least 99.95 percent of purchase requests succeed in under two seconds, between 09:55 and 10:15, measured per on-sale, pass or fail. Lesson 006 published the other half of that arithmetic, that the checkout p99 inside the window is already over a second, which the two second bound deliberately allows for. Add the second this drill added and that p99 crosses two seconds, so at least one purchase in a hundred misses a bound that allows five in ten thousand to miss it. The window fails by twenty times over.
The failover worked. The drill passed. Run the identical procedure at 10:02 and Stagefront loses the thing it exists for, and the ticket from September says PASS.
The ocean is billed per statement
A region is a cloud provider's cluster of data centres in one geography, far enough from the next one that the two are meant to fail separately. Lesson 005 already defined what sits inside it: an availability zone is one data centre, or a tight cluster of them, inside a region. People talk about those two words as one size of decision. Lesson 002's ladder says otherwise. One data centre round trip is 0.5 milliseconds. Mumbai to Virginia is 200. They are four hundred times apart, and only one of them changes how you are allowed to write code.
Everything this course has built for Stagefront sits in one region. I'll put it in Mumbai, so the distances here are the ones lesson 002 already measured, and the standby in Virginia, which is the pair lessons 011, 012, 023 and 033 have all priced.
Here is the purchase path, as the course built it, with the database promoted in Virginia and the twenty application boxes lesson 006 put behind a balancer still sitting in Mumbai.
| Step | Where it runs | Ocean crossings |
|---|---|---|
| Read the seat map | Redis, Mumbai (047) | 0 |
| Claim the hold: select, insert, commit | Virginia | 3 |
| Charge the card | the provider, 150 ms (004) | 0 |
| Confirm: update, commit | Virginia | 2 |
| Refresh the cached map | Redis, Mumbai (046) | 0 |
Five crossings at 200 milliseconds is one second, on top of the 230 the purchase cost on Saturday. That is the whole mystery.
Nobody wrote a slow query. The plans are byte for byte the ones that ran yesterday. What changed is that a transaction is a conversation, and the ocean does not bill you per request. It bills you per statement.
That makes the round trip count a property of your code rather than of your network. Lesson 015 built that hold deliberately: claim it in one fast transaction, do the slow unreliable thing with nothing locked, confirm it in a second. Three statements and two commits is a lean version of that shape. An ORM that lazily loads the show, then the venue, then the price band inside the same transaction will happily issue eleven statements, and eleven statements is 2.2 seconds. The same work expressed as one function call on the database's side is 200 milliseconds. No tuning changes that ratio.
Counting is the entire optimisation, and almost nobody counts.
So the diagnosis for September is not really about databases. A region is the set of components close enough to appear together in one request. The drill moved one of them. Fail over half a request path and what you have bought is a tax.
Lesson 046 left this bill unpaid and pointed here by name. Put a write-through cache in one region and the code that writes it in another, and every write waits a fifth of a second on the one machine in that path the writer needs no answer from. Inside one data centre, latency is never why you refuse write-through. Across an ocean it is the only reason you need.
The floor underneath all of this is physics. Lesson 002 gives you a millisecond of round trip per hundred kilometres in fibre, and calls it the speed of light tax. Mumbai to Virginia is about thirteen thousand kilometres in a straight line, so the floor for that pair is near 130 milliseconds, and the 200 the course uses is the measured number, about one and a half times the floor. That is better than lesson 002's own Hyderabad to Mumbai pair managed at three to four times, and the reason is worth keeping: the overhead that inflates a short hop is mostly fixed, the detour out of each metro and the cut every router takes, so the longer the path the more of it is plain glass and the closer the measurement sits to the floor. For a pair nobody has measured, use lesson 002's warning anyway: expect two to four times the straight line figure, because fibre does not run in straight lines and every box on the path takes its cut. Carry whichever of the two you are entitled to, and never present the floor as a measurement.
No hardware moves it. Only moving the data does.
Active-passive, and the two numbers a plan actually has
Active-passive means one region takes every write and the other holds a copy and takes none. Active-active means both take writes. Every hard thing today is in the second sentence.
Before that, the vocabulary a failover plan is written in. RPO, the recovery point objective, is how much data you accept losing, measured in time. RTO, the recovery time objective, is how long you accept being unable to serve. Two numbers, and teams argue about the second one and inherit the first from a default.
This course has had an RTO since lesson 013 and never used the word. Galewatch, the telemetry service that collects a reading from each of its wind turbines every two seconds, ran a restore drill on a Thursday against a recovery target of one hour. It took six hours and twenty minutes, and that gap is the reason lessons 013 and 014 fixed a shard at 720 gigabytes. Notice how they came to know their real number: somebody restored a backup onto an empty machine and timed it.
Start with asynchronous replication, which is what almost every second region actually runs. Lesson 012 took a replica's lag apart into four terms, and across an ocean the first one swamps the rest: bytes have to cross a wire, and that wire is 200 milliseconds wide. No setting removes it. So an asynchronous cross-region standby has an RPO that cannot be zero, and at best is 200 milliseconds.
Price that at the minute Stagefront exists for. Lesson 004 published 500 payment calls a second at peak, so 200 milliseconds holds a hundred purchases. A hundred seats sold in Mumbai, paid for, confirmed to the buyer by email, that Virginia has never heard of. Promote Virginia and those hundred seats are free again. Sell them and you have oversold a hundred seats, which is the lawsuit lesson 007 put the seat holds in a database rather than a cache to avoid. Decline to sell them and a hundred people are holding a charge and no seat, which is the same lawyer with a different letter.
There is no third option.
And a hundred is the optimistic figure. It assumes the lag sits on its floor, which here it roughly does: 500 writes a second at a few hundred bytes each is under 200 kilobytes a second, so the link is not the constraint and lesson 012's single threaded replay has nothing to choke on. RPO is measured in seconds and paid in rows. Nobody converts it, and the conversion is the only form anybody can decide from.
Zero RPO costs a synchronous commit, and lesson 011 already gave the verdict in one sentence: Stagefront would not notice half a millisecond inside one data centre and could not run an on-sale on two hundred milliseconds across an ocean. Lesson 023 priced it again and found 200 milliseconds welded onto every write, which it set against lesson 015's three hundred millisecond card call and called two thirds of the charge. At Stagefront the comparison is worse than that, because lesson 004 measured this provider at a hundred and fifty: the ocean would cost more than the charge does. Run lesson 002's busy room over it and you get the shape of the breakage: 500 commits a second at 200 milliseconds each means a hundred commits in flight at every instant, each holding a connection and whatever locks it took on the way in. The seats are distinct rows, so buyers do not queue behind one another.
What queues is the pool.
The honest third option is a quorum across three regions, and lesson 033 has already done this arithmetic and handed the rest here. Leader in Mumbai, followers in Singapore and Virginia; a majority is the leader plus the nearer follower; Singapore is about four thousand kilometres, so every commit waits forty milliseconds. Then its sharp edge: the furthest member is free until the day the nearest one dies, at which point commits step from forty milliseconds to two hundred and nothing has gone down. A quorum's latency is set by its second fastest member, and that member can change with nobody being paged.
Lesson 033's twenty five commits a second comes with a qualifier that has to travel with it: taken in series. Real implementations batch and pipeline, so 500 a second is reachable while the forty milliseconds on each buyer's commit is not negotiable. That leaves Stagefront a failure nobody has modelled. Forty milliseconds on a seat claim at an on-sale is affordable. Two hundred is the window. So the morning somebody takes Singapore out for a planned rebuild, every commit silently moves to Virginia's price, and lesson 033 says plainly that nobody gets paged for it. A quorum's silent latency jump is an on-sale failure waiting for a maintenance window.
| Design | RPO | Every write pays | What it costs you |
|---|---|---|---|
| Async standby, second region | 200 ms, a hundred seats | nothing | the hundred seats, on the day |
| Synchronous standby | zero | 200 ms | one dead standby stops all commits (011) |
| Quorum, three regions | zero | 40 ms, or 200 when Singapore goes | three bills, and the silent jump |
None of those three rows gives you an ordering. They give you durability and a vote. Lesson 034 published the mechanism that does, Spanner's commit wait, with its verdict attached: you can buy a real "before", and the price is the width of your uncertainty, paid in latency on every write. What it leaves for a second region is one sentence. The stamp is cheap, single digit milliseconds, and the write underneath it still has to reach a majority, so an ordered cross-region commit pays lesson 033's forty milliseconds and lesson 034's few on top.
Commit wait buys a correct "before", not a cheap one. Lessons 072 and 073 are where it is worth the money.
Active-active is a lot of active-passive, cut finer
Everybody draws active-active the same way. Two regions, both serving, both writing, traffic split by whoever is nearer. It is a lovely diagram and it survives until the first interesting row.
10:00:03. Two buyers, one in Mumbai and one in Virginia, both click seat 14F. The regions are 200 milliseconds apart, which means for 200 milliseconds each region is entitled to believe its own copy, and both holds succeed, and both buyers get a confirmation. Lesson 024 named the only two answers available at that moment: answer and give up behaving like one copy, or refuse and stop being a service. There is no merge function for a seat.
So sort your data by what a conflict costs, which is a question about the data and not about the deployment.
Some data has no two writers at all. Galewatch's readings are the clean case, and lesson 034 refused vector clocks on exactly this ground: every reading is its own key, written once, never updated. Two regions can both ingest turbine readings forever with no coordination of any kind, and the only thing to get right is that a replayed backlog does not create a second copy of a reading, which is lesson 019's problem and solved.
Some conflicts are a merge. A basket, a document, a set of tags. Last write wins will lose data quietly; a real merge needs a value whose type can be merged, and lessons 058 and 087 own that argument properly.
And some conflicts are a lawsuit. A seat, a balance, a serial number. There is no merge, because the conflict is the product being wrong.
So the pattern teams actually ship looks nothing like the diagram. Give every show's seats one home region, named in a small table both regions read. Both regions run the whole stack, both serve every page, and a write to seat 14F goes to that show's home region whatever continent it arrived on. Stagefront is then active-active as a company and active-passive per show. Nothing has two owners, the fleet is live in two places, and active-active is a lot of active-passive, cut finer. The unit of failover stops being the company and becomes the show, which is also the unit the business thinks in.
The price is paid by the buyer in Virginia who wants a Mumbai show: 200 milliseconds a crossing, on a path their neighbours get for five. That is the correct price. The alternative on their behalf is overselling.
Lesson 022 published the rule this is a special case of: an edge moves bytes, not decisions. Here is the version with a region in it. A decision may move away from home when it is revocable, or when the data it decides about has an owner where you moved it. Marlow Books, the four person online bookshop this course keeps breaking, can decide "in your basket" anywhere, because that decision can be withdrawn at checkout. Galewatch can decide anywhere, because each turbine's key has one writer. A seat hold is neither revocable nor single owner, so it goes home.
The seat map is bytes and may live at the edge. The seat is a decision and may not.
Reads are where this gets quietly worse than the textbook says. An active-active read tier is lesson 012's replication lag with an ocean in it, and lesson 012 gave the rule for what that costs: the failure rate is the share of the lag distribution sitting above your gap, the gap being the time between a customer's write and their next read. Marlow's lag was 6 milliseconds at p50 against a 45 millisecond gap, so about three review posts in a hundred came back missing their review; against the 15 milliseconds of a customer on fibre in the same city, closer to one in five.
Run that rule across an ocean and it stops returning a percentage. Put the two paths side by side. The buyer's confirmation needs one trip home, because the primary answers the moment it has committed. The replica's copy needs the crossing, and then it needs lesson 012's replay before any query there can see it, which is single threaded and waits on disk a page at a time. Lesson 012 was explicit that bytes sitting safely on a standby's disk are still invisible until the commit record is applied. One path is a network trip. The other is a network trip followed by a queue of other people's transactions. So for the buyer sitting beside the far replica, the gap is shorter than the lag by construction, and the two distributions no longer overlap at all.
A customer reading their own write from the far replica gets a stale answer every single time, not three times in a hundred. Lesson 012's cruellest finding arrives doubled: the customers closest to you lose, and the customer closest to the copy loses hardest.
The fix is the one lesson 012 called cheapest, which is to not read. Pin a buyer to their show's home region for a window after their own write, or hand back the row the write already returned. Both work, and the first costs them the ocean at the moment they care most about the answer. Say that out loud in the design review rather than discovering it in a support ticket.
The failover nobody has run
A region going dark and a link going dark produce the same observation. Lesson 024 said the detector cannot tell dead from unreachable, and across an ocean the suspect is the link itself, which is also the only thing that could have told you otherwise. Lesson 032 named what happens when both halves act on their own reading: two primaries. For a seat database, two primaries is the lawsuit arriving by automation.
The course's best story about this is already published, in lesson 024. On 21 October 2018 GitHub lost a link for forty three seconds, automated failover promoted a cluster on the other coast, and the two sides had both taken writes; GitHub chose integrity and ran degraded for twenty four hours and eleven minutes, about two thousand to one against the partition. A machine made that call in forty three seconds from a policy somebody had written months earlier.
What this lesson adds is what that policy was. It was a cross-region active-passive failover, which is the design the whole first half of today describes, and the radius of its automation was a continent. It fired on a blip.
So here is an opinion with a bill attached. Automate failover for anything stateless, caches and read tiers included. Do not automate the promotion of the one database whose rows are a lawsuit. Automate the fence instead, which lessons 032 and 035 both built: the thing a machine should be allowed to do at 02:00 unattended is make the old primary's writes impossible, by taking its token, revoking its credential, dropping its listener. Then a human decides whether to promote, knowing the old primary cannot have taken a write since the fence went up.
Which exposes the quiet lie in every failover plan I have read, mine included. RTO is a promise about resuming service, not about resuming correctness. September's promotion took ninety seconds, and ninety seconds is what the eleven pages promise. GitHub's twenty four hours were not spent restoring a link, which came back on its own in forty three seconds; lesson 024 already told you what they were spent on, which was reconciling decisions. No plan I have seen puts a number next to that, because the number depends on how many rows the two sides disagree about, which depends on the RPO you inherited from a default.
Now the arithmetic for whether to own the second region at all. Lesson 003 settled the general shape: the shared cause is the ceiling, which is why Marlow's second server turned 43.8 hours of downtime into 32 rather than into thirteen minutes. The regional version of that ceiling is a short list: the deploy that goes to both, the schema migration that goes to both, the DNS record in front of both, and the control plane.
Lesson 049 named the control plane and lesson 048 priced it. Fail static buys you the fleet you have and nothing you are about to start. A failover is entirely a thing you are about to start: new boxes, new routes, a promotion, a certificate, a configuration push. So if the control plane lives in the region you just lost, the procedure needs the thing that is gone. Test the runbook with the control plane unreachable, or you own two regions and a plan that depends on one of them.
Then the money. Lesson 003 priced Stagefront's purchase chain at 19.3 hours a year, of which 8.8 belong to the payment provider nobody there operates. Assume a region-wide outage of four hours once every two years, and substitute your own figure: that is two hours a year. A second region removes those two hours if the failover works, which is about a tenth of the bill, and moves the 8.8 not at all. Lesson 003 also published the cheapest nine on the table, and it is not geographical: drive mean time to recovery from three and a half hours to twenty minutes and 42 hours becomes 4.
For the window the arithmetic is sharper and almost nobody does it. Stagefront is scored between 09:55 and 10:15, so forty on-sales a year is 800 minutes of scored time out of 525,600, wrapped around the 400 minutes of real intensity lesson 003 counted. A four hour outage overlaps a given window if it begins any time in the 260 minutes before that window ends, so forty windows cover 10,400 minutes of the year, and an outage landing uniformly has about a two percent chance of touching one.
Two percent.
If Stagefront bought its second region to protect the on-sale, it bought a lottery ticket. If it bought it so that an ordinary Tuesday afternoon does not cost the company a day of trading, it bought the right thing for the right reason, and the design document should say which, because the two purchases want different runbooks and different rehearsals.
Marlow should not buy one at all, and the reason is lesson 003's ceiling rather than the price. The shop's outages are a certificate, a deploy at 4:50 on a Friday, a payout job that kills Postgres on the first Sunday of the month, and one database. A second region removes none of those. It adds a second place for the founder to deploy to on a Friday afternoon.
Back to that Sunday morning in September. Lesson 013's restore drill is the model, and it is not complicated: Galewatch knows its real recovery time because a person restored a backup onto an empty machine and timed it, and the number came back six times worse than the target.
The September drill measured a promotion, which was never the risky part. The thing Stagefront is scored on cannot be measured at 02:00, so either you fail over during a real on-sale, which nobody will authorise and nobody should, or you build a load rehearsal that reproduces the window and fail over underneath it.
The eleven pages now have a first step they did not have: bring the application tier up in Virginia before promoting anything, and move DNS last. Promote the database first and you get September's second of ocean, on every purchase, with every graph green. One hop the ordered version cannot fix is lesson 003's ceiling wearing a new hat: the payment provider sits in somebody else's data centre, lesson 004 measured 150 milliseconds to it from Mumbai, and nobody at Stagefront knows what it measures from Virginia. Finding out takes an afternoon.
Recap
The ocean is billed per statement. Distance is charged per round trip, and the round trip count belongs to your code, not to your network. Five crossings inside one purchase is a second, and the same work in one call on the database's side is 200 milliseconds.
A region is the set of components close enough to appear together in one request. Fail over half a request path and every graph stays green while every customer pays the ocean, so the order of a failover is part of its design: application tier up first, promotion second, DNS last.
RPO is measured in seconds and paid in rows. Two hundred milliseconds of asynchronous replication at 500 purchases a second is a hundred seats, and both ways out of a hundred lost seats are a letter from a lawyer. Convert the number before anybody is asked to accept it.
Active-active is a lot of active-passive, cut finer. Sort data by what a conflict costs: no two writers, a merge, or a lawsuit. Then give every entity in the third pile one home region and route writes by entity rather than by geography. A decision may move away from home only when it is revocable or its data has an owner where you moved it, which is lesson 022's bytes-not-decisions with a region in it.
Across an ocean the distributions stop overlapping. Lesson 012's rule, lag against the gap, returns a certainty instead of a percentage: a customer beside the far replica reads a stale copy of their own write every time. The customers closest to you lose, and the customer closest to the copy loses hardest.
RTO is a promise about resuming service, not about resuming correctness. The promotion takes ninety seconds and the reconciliation took GitHub a day in October 2018. Lesson 033's second fastest member is the same kind of trap one layer down: forty milliseconds a commit becomes two hundred the morning the nearest follower goes for a rebuild, and nothing is down and nobody is paged.
A drill at two in the morning measures the half you were not worried about. Lesson 013 got a real number by restoring a backup and timing it. A failover rehearsed at the hour when your system has no load has told you about your promotion script and nothing about your business.
Check your understanding
Your service runs in one region with an asynchronous standby in another, and a product manager asks for "zero data loss failover" by next quarter. Write down the three designs priced today, say what each charges every write, and say which one you would take back to them along with the number you would make them accept.
A team proposes active-active across two regions for a payments service, because each region handles its own country and the countries never overlap. Say what you would check before agreeing, name the row that breaks it, and say what you would do with an account that moves country.
Your checkout makes eleven database statements inside one transaction and runs comfortably at 40 milliseconds. Somebody suggests moving the database to a region 90 milliseconds away to be nearer a new market's customers. Give the arithmetic, then give the two changes that would make the move survivable and say which one you would do first.
Infrastructure schedules a rebuild of your three region quorum's nearest follower for 09:30, on a day with a 10:00 on-sale. Lesson 033 already told you what that does to commit latency. Today's part is the next one: carry the new number through a purchase and decide whether the window promise still holds, say who you go and talk to and what you ask them for, and say what you would do instead if the rebuild cannot be moved.
A colleague wants the cross-region promotion fully automated so a 03:00 region failure needs nobody awake. Argue both sides from today's two examples, then say exactly which part of the sequence you would let a machine do unattended.
Your company has owned a second region for two years and has never failed over to it. You have two weeks of one engineer's time. Say what you would spend it on, what you would deliberately not do, and what number you would publish at the end.
Next lesson
051 Data Pipelines: Batch, Stream and Both at Once. Today a write had one home and the problem was distance; next it has two destinations and the problem is that the fast path and the correct path disagree, which is the same reconciliation GitHub spent a day on, scheduled on purpose.