Stagefront is an event ticketing service that exists only in this course, where a stadium show goes on sale at 10:00 and two hundred thousand people press the same button in the same minute. Lesson 047 took apart the September on-sale where the seat map took half a second to appear. Two things shipped afterwards.
The first was the fix 047 argued for: a small dictionary inside each of the twenty application boxes lesson 006 counted, holding the seat map for two seconds. Twenty caches at a two second TTL is ten fetches a second from the shared cache instead of 3,333, so eighty megabits where there had been 26.7 gigabits. Two lines of code.
The second was a graph. September's embarrassment was that one node had been offered nearly three times its network card while every dashboard in the building was in operations, so somebody added the missing one: bytes out, per cache node.
On Thursday 15 October, at four minutes past ten, the graph is what noticed.
A forty thousand seat show had gone on sale at 10:00 and sold out in lesson 017's eighty seconds. Nothing failed. The seat map came back in single digit milliseconds, no error rate moved, and the boxes sat at the 75% CPU lesson 006 measured for an on-sale minute. But the node holding the seat map had pushed about eight gigabits during that minute against its ten gigabit card, and the point of the local dictionary was that the number should have been eighty megabits.
Eight gigabits is a thousand requests a second for a one megabyte map. At 3,333 arrivals spread over twenty boxes, a thousand is six boxes' worth.
Six boxes were not using their local cache at all.
They were the six that had restarted. Lesson 006 named the confusion that does this, which is that a liveness check kills a process while a readiness check only stops traffic, and 006 told the reader to go and check which one their process manager thought it was doing. Nobody at Stagefront went and checked. A box at 75% CPU sometimes answers its check slowly, so across eight days and four on-sales, six boxes out of twenty had been restarted for no better reason than that.
Each of the six came up, asked the configuration service for its local TTL, got nothing back, and fell through to the constant compiled into the binary. The constant was zero. A cache you cannot configure is a cache you switch off, which is the careful choice, and it had been written in September by somebody thinking about one process.
The configuration service had stopped answering at 02:14 on Wednesday 7 October, when its disk filled with its own logs. Eight days. Nothing alerted, because its availability had never mattered to anybody.
Nothing had gone wrong in those eight days either. Fourteen boxes were still serving the two seconds they were told about before the 7th, and would have served it for a year. The outage was entirely in the future, and six boxes had already collected their share of it.
Two questions that look like one problem
Service discovery answers one question: given the name of a service, which addresses are serving it right now. Configuration answers another: given the name of a setting, what value should every process be using. Underneath they are one chore, getting a single fact into every process and keeping it there, and this course has walked up to that chore three times from three directions.
Lesson 031 got there first and put it best. A formula needs no agreement and gives you no control; a table gives you control and needs agreement. The ring is the formula. A slot map is the table: you can edit it, drain a machine with it and pin a hot key in it, and in exchange you own a few thousand rows that have to be identical in every process that routes, including the one that started thirty seconds ago and the one that has been up for six weeks.
Lesson 029 got there second, with a revocation list. Kick two hundred bot accounts a minute against a five minute token and the list holds about a thousand entries at sixty four bytes an id, so sixty four kilobytes, which fits anywhere. Size was never the problem. Freshness is the product. Without the list, a scalper you kick at 10:03 keeps buying until their token expires at 10:08, and since lesson 003 gives an on-sale about ten minutes of real intensity, that is half the event. With the list, the exposure is however long it takes to reach every checker, which is the thing today has to put a number on.
Lesson 047 got there third, with one integer. Two seconds.
A few thousand rows, sixty four kilobytes, one integer. The size tells you nothing about how to deliver any of them.
What tells you something is what happens during the window when two processes disagree, and there are only three answers.
It converges and nobody is hurt. A box holding four seconds of local TTL while another holds two gives its customers a staler seat map, and both are serving. Lag is the whole cost and the poll interval is the bill.
It diverges in silence. This is 031's incident: two processes with different machine lists compute two different rings, both confidently, and neither errors. No error rate, no log line, no graph. Twelve minutes of writes went to the wrong machines and it took three days to notice.
It fails towards yes. A box without the new revocation list lets a banned account keep buying, and the sentence to keep is 023's, as 029 compressed it: a stale permission renders on no screen anywhere.
| What a disagreement does | The worked case | What it needs |
|---|---|---|
| converges | 047's local TTL | a poll; the interval is the lag |
| diverges silently | 031's slot map | a version, and a loud refusal |
| fails towards yes | 029's revoked ids | a push, and an audit |
Sort configuration by what a disagreement does, never by what the value holds. The row you are in picks the delivery mechanism, and it is the only thing that does.
That leaves one distinction, and most people have it backwards.
A registration expires. A decision does not. A registry entry carries a lease, and it is a gentler thing than lesson 032's lease on a job, where the promise to stop had to be kept by the machine losing it. Here nobody has to keep anything: the registry forgets an entry nobody renews. A configuration value carries no such clock, so it stays until somebody changes it. That inverts the risk between the two halves of today's title: a registry nobody maintains decays into garbage by itself, towards an empty list if entries expire and towards a list of dead boxes if they do not, while a configuration store nobody maintains is correct indefinitely.
So everybody worries about the registry, because it moves. The config store is the one that hands you a value from before the last three deploys.
A registry is a list that goes wrong by itself
Start with DNS, because it was the first registry and it is still the one in the way. A name resolves to a set of addresses with a TTL, and that is all you get: no health information, and no way to tell a client the answer it holds is now wrong. Which makes DNS a cache, and lesson 047 already named what a cache like that costs, since a level you cannot invalidate is a level whose TTL is your real correctness bound. Worse than 047's dictionary, because its TTL is advisory and the clients lie about it: a JVM caches a positive lookup for its own thirty seconds, or forever when a security manager is installed, a connection pool holds a socket it resolved once at startup, and you configure thirty seconds and find out what you bought during an incident.
So people build a registry: a list of the addresses currently serving a name, maintained by something that is watching. What makes it more than a list is health checking, and lesson 006 owns health checks while lesson 042 owns the breaker, so the question left for today is narrow. Who is allowed to say a box is gone.
The box says so about itself. Self-registration with a heartbeat: the process announces its address on startup, repeats every few seconds, and the registry drops the entry when the heartbeat stops. Cheap, and it inherits lesson 032's renewal cadence trap in its purest form. Renew from a background thread and the lease stays healthy while the work it vouches for is wedged on a socket. 032 wrote that about a lease on a job, where at least the holder was doing the job. A heartbeat has less behind it: lesson 006's objection to a curl of localhost was that a check proving a process is running has proved that a process is running, and a heartbeat does not establish even that much about the part that answers requests.
Something else says so. A checker, central or an agent on each box, probes the real endpoint and deregisters what fails. You get a real check, and you import 006's coupling failure with it, where a check that queries a shared dependency is a machine for converting partial failures into total ones. That import is worse than 006's version, and the multiplier is the number of consumers. When 006's balancer removed the whole fleet, one component held a wrong answer, and its only job was routing requests. A registry's answer reaches every client there is, so if your metrics pipeline resolves its own destination through the registry, lesson 026's instruments go with it. That is lesson 003's shared cause being the ceiling, arriving at the one component that cannot afford to be on the wrong side of it.
Nobody is allowed to say zero. Lesson 006 named the balancer's panic threshold and said plainly that it stops the wipeout and not the spiral. A registry wants the same floor and has a better reason for it. "Nobody is serving this name" is almost never a true statement about a service that was serving a second ago. Far likelier is that the thing which knows has broken. Envoy's default is to stop honouring health status once fewer than half the hosts in a pool look healthy and to route to every host in the list, dead or alive, on the reasoning that a coin toss beats a certainty.
Put that floor in the client as well, and a client that refuses to replace a working list with an empty one is a client whose registry is allowed to disappear. Which generalises much further than an empty list.
Fail static, and the process that started thirty seconds ago
The rule every mature discovery system lands on is fail static: when the control plane cannot be reached, keep using the last answer you had. Do not fetch, do not fail, do not fall back to a default.
What makes it worth the words is the price it changes. Lesson 003 established that downtime adds, that every component in the path of a request contributes its own outage hours, then refined it in a way that pays here: the chain is the request path, not the architecture diagram, so you count only what the request actually touches. A process that reads the registry on every call has put the registry in that chain, and on 003's five component purchase chain at 99.78 percent and 19.3 hours a year, a config service at three nines would add 8.8 hours by itself. A process that reads its own cached copy has taken the registry out of the chain. It is no longer a dependency of every request. It is a dependency of every change.
Then comes the part nobody writes down, and it is why the hook took eight days.
Fail static only protects a process that already has an answer to be static about. So a control plane outage cuts your fleet in two, and the halves have nothing in common. A process that was already running holds the value and is entirely unaffected, for as long as it lives. A process that starts during the outage has nothing. It is not degraded. It never had the fact.
Read lesson 031's handoff sentence again: identical in every process that routes, including the one that started thirty seconds ago and the one that has been up for six weeks. The six week process is fine. The thirty second process is the entire failure surface.
Fail static buys you the fleet you have and nothing you are about to start. So what converts a quiet control plane outage into a real one is a restart, and restarts arrive for reasons with no connection to the control plane at all. A deploy, which is lesson 054. Autoscaling up, which is lesson 055. A box reaped by the platform. Or Stagefront's version, which is the cheapest and the most common: a liveness check firing at a box that was merely busy. A configuration outage is collected one restart at a time, and at Stagefront it was collected fastest during the only minutes of the year that pay for the system, because that is when boxes get restarted.
Lesson 003 brought the large version of this into the course already, for its shape rather than its mechanism: in October 2021 the game platform Roblox was down for 73 hours, three full days, with the failure sitting in a Consul cluster. 003's point was that when the machinery a system uses to find its own parts breaks, most of the tooling you would reach for to diagnose it is broken too, and Roblox's own report says exactly that, naming the circular dependencies its observability stack had on Consul as something it had still to remove.
What 003 did not need is the rest of that report. Consul was doing both halves of this lesson's title, and the report notes that Nomad and Vault both sit on top of it, so with Consul unhealthy nobody could schedule a new container or retrieve a production secret. Hold that against the two halves of the fleet above. The running processes were not what stopped Roblox. The inability to start anything was.
Two causes, in the company's own framing: a relatively new streaming feature enabled on Consul, which under unusually high read and write load produced excessive contention, and a pathological performance problem their load triggered in BoltDB, the embedded store Consul keeps its write-ahead log in. One shared cluster carrying multiple workloads made both expensive. The first two theories were slow hardware and traffic growth, so they replaced the Consul nodes and then the whole cluster, and neither helped. They brought the product back with services that normally run hundreds of instances down to single digits and the non-essential parts off, because a fleet all starting at once is lesson 020's retry storm pointed at the thing that has just broken.
Notice what fail static would and would not have done there. Cached service lists keep running things talking to each other. They cannot let a scheduler start a container that has never run, and Roblox needed a different fleet from the one they had.
The cheap half of the fix is in nobody's design document. Write the last good value to disk. Then a process that starts during an outage has something to be static about, and the whole thirty second failure surface closes for the cost of a file write per change. Run Stagefront's eight days through it: every one of those six boxes had the two seconds in memory before it was restarted, so every one of them would have had it on disk, and the graph on the 15th would have shown eighty megabits.
The expensive half is that you have built another copy nobody can invalidate. Lesson 030's rule was to count the copies and write down how long two may disagree, and a file on disk has no bound in that column: the honest entry is however long ago this box last spoke to the control plane, which could be six weeks. So stamp it with a time, then decide out loud what a process does with an answer that old. I do not think there is one correct choice, but there is a correct way to pick, which is to run the three rows of the table above. A six week old local TTL is harmless, so use it. A six week old slot map routes writes to the wrong machines in silence, so refuse to start and page somebody. What a process does on a cold start is decided by what a wrong value does, not by how old the value is.
Push for speed, poll for truth
Two ways to move a value, and they fail at opposite ends.
A poll is every process asking every few seconds. Galewatch, which collects a reading every two seconds from each wind turbine on the farms it monitors, has the working example, and lesson 033 recorded what it bought. The four ingest boxes reload lesson 014's routing table from the database every five seconds, wholesale, and moving the shard divisor out of a config file and into that table is what turned lesson 031's twelve minute disagreement into a five second one. A config file goes out one box at a time. A table everybody polls arrives at everybody on the same cadence.
The load is trivial, and worth computing once so you stop worrying about it. Fleet size over interval: twenty boxes at thirty seconds, one client to a box, is two thirds of a request a second. You can poll far more often than instinct suggests.
What a poll costs is the thing 033 found by accident, and it is worth naming properly. A cache refreshed wholesale inherits a deletion nobody asked for. When route-2 was promoted at 14:41 holding the table as it stood at 13:55, all four ingest boxes faithfully replaced a correct table with an older one, and by 14:41:10 none of them had ever heard of Drumlea Moss. A wholesale reload replaces good state with bad without hesitating. A merging reload never deletes anything, which is its own bug.
Neither is the answer, and the answer is one integer. Ship a version with the value and have the reader refuse anything lower than what it already holds. That is 031's make the wrong owner loud, moved from the data path to the control path, and it costs a comparison.
A push is a watch, a long poll or a stream: the control plane tells you when something changed. It buys latency, which for 029's revocation list is the entire product, because there the poll interval is the revocation delay. Thirty seconds of a banned scalper buying is a poor trade against a list you could have had in a hundred milliseconds.
What a push cannot do is tell you it failed. A dropped connection, a box that was restarting during the fan-out, a watch that stopped delivering months ago. Nothing anywhere says "this box has not heard from us".
So every configuration system that works is a push for speed sitting on a poll for truth. etcd and ZooKeeper, the two lesson 033 named when it introduced Raft and ZAB, both pair a watch with a periodic full resync, and that is not belt and braces: the resync is the correctness mechanism and the watch is the optimisation. Lesson 037 handed today the same shape in a different costume, since a compacted topic is the latest value per key, rebuildable from offset zero, so a consumer that reconnects reads the current table out of the log rather than asking anybody for it.
One cost in the push column is worth being frightened of. A watch is a long lived connection per client, so your control plane's load scales with fleet size rather than with change rate, and it peaks the moment every client reconnects at once, which is the moment after it comes back. A poll is bounded by the arithmetic above. A watch is bounded by nothing, and Roblox's streaming feature was a change to exactly this mechanism, enabled under unusually high load.
Marlow's two files and the row with no limit
Marlow Books is a four person online bookshop that also exists only in this course, running about forty requests a second on an ordinary day, and it is carrying the one row in this course that has no number in its last column.
Lesson 027 built the shop a checkout kill switch, as two more fields in the small live uncached request lesson 022 had already invented for the customer's name and basket count. The values come from a file on each of the two boxes rather than from the database, because the switch has to survive the mode where the database is the problem, beat the nine edge caches holding the old page, and be flippable by whoever is awake, which at a four person shop is somebody in the packing room with a tape gun. Thirty seconds over ssh, no deploy. Every one of those reasons is right.
Lesson 030 then counted the copies of every fact the shop depends on and found the row with no limit in the last column. Two files, on two boxes, with nothing on earth reconciling them. 030's rule is that a row with no limit on disagreement is either a fact nobody writes or a bug nobody has had yet, and this one has been open since September.
The textbook answer is to put the switch in a configuration service, and for Marlow that is wrong, for 027's own reason: a config service is one more thing that can be down, and this switch's whole job is to work on the day things are down. Lesson 033 priced the honest version at three machines to get what you thought two would buy, for a shop whose Postgres still runs on box A next to the application.
What closes the row without breaking 027's constraint is far smaller than a service: two changes and a refusal.
One writer instead of two edits. The founder stops opening files and runs one command that writes both boxes and reports which ones it wrote. The disagreement window becomes the length of that command, and when it fails halfway it says so out loud instead of leaving a coin toss in production. That is a shell script with an exit code, and it moves 030's last column from no limit at all to a few seconds or a loud error.
Then make the state visible, which is 031's rule again at this radius. Put the switch's value and its file's modification time into the health endpoint lesson 006 already requires of both boxes, so the question "do my two boxes agree" has an answer somebody can look at. Lesson 026 owns the instrument and this costs two fields.
And one refusal, which the first section of this lesson exists to justify. Do not give that switch a lease. A registration should expire, because a process that stops saying it is alive probably is not. A decision should not, because a kill switch with a TTL is a switch that turns the shop back on by itself at three in the morning while everybody who flipped it is asleep.
None of this settles the argument lesson 030 started with lesson 027, which was whether a switch may be inconsistent in a way a stock count may not. I think 030 has the better of it, since a grey checkout button is a refused order and therefore a decision. The useful part is that the fix above makes the question stop mattering: you do not need a ruling on whether a switch may disagree for three months once it cannot disagree for more than three seconds.
Recap
Service discovery and configuration are one chore seen from two ends, getting a single fact into every process and keeping it there. The course arrived at it three times before today: 031's slot map of a few thousand rows, 029's sixty four kilobytes of revoked token ids, and 047's one integer of local TTL.
Sort configuration by what a disagreement does, never by what the value holds. A disagreement that converges wants a poll, and the interval is the bill. One that diverges silently wants a version number and a loud refusal, because 031's two rings both compute confidently and neither errors. One that fails towards yes wants a push and an audit, because 023's stale permission renders on no screen anywhere.
A registration expires and a decision does not. A registry entry has a lease and decays into garbage on its own, towards an empty list or a list of dead boxes depending which way its policy leans, while a configuration value stays until somebody changes it. The registry is the half that moves; the config store is the half that hands you a value from before the last three deploys. And because it decays, a registry should not be able to say zero: nobody serving a name is almost never true of a service that was serving a second ago, so 006's panic threshold belongs in the client too, as the rule that a working list is never replaced by an empty one.
Fail static buys you the fleet you have and nothing you are about to start. It takes the control plane out of 003's request chain and makes it a dependency of change, which is most of the value available here. What it costs is a fleet split in two, where the six week process is fine and the thirty second process is the whole failure surface, so a configuration outage is collected one restart at a time: by deploys, by autoscaling, and at Stagefront by a liveness check firing on a box that was only busy. Roblox's 73 hours is the same shape at scale, with running services still running and nothing new able to start. Writing the last good value to disk closes that surface, and stamping it with a time is what makes it safe, because what a process does on a cold start is decided by what a wrong value does, not by how old the value is.
Every configuration system that works is a push for speed sitting on a poll for truth. A poll is bounded and cheap at fleet size over interval, and it will overwrite good state with old state wholesale, which is how four ingest boxes forgot a wind farm existed. A push is fast and cannot tell you it failed.
Check your understanding
Your service reads its feature flags from a config service on every request, and somebody proposes caching them for sixty seconds instead. Price both designs with lesson 003's chain, then say what new failure the cached version introduces and which of today's rules closes it.
A registry runs health checks that call each box's
/health, which queries the shared database. The database pauses for forty seconds. Walk through what the registry does, name the earlier lesson this repeats, and say why the consequence is worse here than it was there.You have one integer to distribute to two hundred boxes, the sampling rate for your tracing. Pick poll or push, give the interval or the mechanism, and say what you would do about a box that has been unreachable for a week and is still sampling at the old rate.
A colleague moves the shard routing table from a config file into a table every process polls every ten seconds, citing lesson 033. A week later two processes write the same key to two different shards for about nine seconds during a deploy. Explain how both polls succeeded and both answers were wrong, and give the one field that prevents it.
Your control plane has been down for three hours and nothing is broken. Your manager asks whether this is a Sev 1 or a Sev 3. Answer with the question you would ask first, and say what you would do in the next ten minutes under each answer.
Stagefront wants to delete the compiled-in default for the local TTL, so that a box with no configuration refuses to start rather than running with no cache. Say what that fixes, what new outage it makes possible, and the third option you would ship instead.
Next lesson
049 API Gateways and Service Meshes. Every answer today lives inside your own processes, so every language and framework in the building needs its own copy of the registry client, the poll loop, the version check and the fail static rule. Lesson 049 is what happens when somebody moves all of it out of the application and into the network underneath, and what that move costs.