Marlow Books is a four person online bookshop that exists only in this course, and in November its founder finally did the responsible thing and bought an uptime monitor. Nine dollars a month. It fetched the home page once a minute from a machine in Singapore and sent an email if the page didn't come back.
At the end of December it reported 99.99% availability. Four failed checks out of 44,640.
December contained the Friday before Christmas from lesson 001: the podcast mention at 10:40, just over four hundred requests a second by 10:46, eleven second book pages by 10:47, and the health check restart at 10:52 that killed every checkout in flight. From 10:46 until a little after noon, about ninety minutes, most people who tried to buy a book gave up or got an error.
Ninety minutes out of 44,640 is 99.80%.
Both numbers are right. The monitor measured whether one request for one page from one city came back with anything at all inside thirty seconds, and by that definition the shop was up, because an eleven second page passes when your timeout is thirty. The founder measured whether people bought books.
That is the whole subject. Availability is a fraction you decide how to compute before you can measure it, and most arguments about availability turn out to be arguments about a definition nobody wrote down.
The definition comes first
The fraction is good events over valid events. Everything interesting is in what you put in each half.
The reliability engineering literature calls that fraction a service level indicator, an SLI, and the term earns its place because writing one down forces the four decisions Marlow's monitor made silently.
What counts as an event. The monitor counted one synthetic request a minute against the home page, which is the cheapest and most cacheable page on the site. Real traffic that Friday was book pages, search and checkouts, hundreds of thousands of requests in that ninety minutes alone, and none of it was in the number.
What counts as good. Any response inside a thirty second timeout, in the monitor's world, which quietly declares eleven second pages acceptable. Put a latency bound in the definition instead and the number starts telling the truth. Lesson 002 gave Marlow's API a p50 of 15 ms and a p99 of 600 ms, so a rule like "returned a 200 in under one second" costs the shop almost nothing on an ordinary Tuesday. On the podcast Friday the pages that came back at all took eleven seconds. Every one of them was a 200, and every one of them failed that rule. A request too slow to be useful is unavailable in practice, and that is how latency ends up inside an availability definition rather than beside it.
Where you measure. Server side numbers are the easiest to collect and they exclude, by construction, every failure that happened before the request reached your server: the DNS lookup that went nowhere, the TLS handshake that hung, the load balancer that returned 503 before your application ever heard about it. Measure as close to the user as you can afford, and know which failures your chosen vantage point cannot see.
Over what window. This one decides the answer more than the system does. Four hours of downtime is 99.95% of a year and 99.45% of a month, from the same four hours.
So write the sentence out before you argue about the target. "The proportion of checkout requests that return a 200 in under one second, measured at the load balancer, over a rolling 28 days." Boring sentence. It is also the only version of the number that two people can usefully disagree about.
Nines, in hours you can argue about
Percentages are hard to feel. Hours are not.
| Availability | Per year | Per month | Per week |
|---|---|---|---|
| 99% | 3.65 days | 7.3 hours | 1.7 hours |
| 99.5% | 43.8 hours | 3.7 hours | 50 minutes |
| 99.9% | 8.8 hours | 44 minutes | 10 minutes |
| 99.95% | 4.4 hours | 22 minutes | 5 minutes |
| 99.99% | 53 minutes | 4.4 minutes | 1 minute |
| 99.999% | 5.3 minutes | 26 seconds | 6 seconds |
A year is 8,760 hours, a month is 730, a week is 168. Multiply by the fraction you are missing and you have the whole table, which means you never have to trust mine.
The per week column is the one that ends meetings. Somebody says "we should be at five nines" and what they have agreed to is six seconds of failure a week, forever, including the week of the certificate renewal and the week the intern joins. Six seconds is less than one deploy. Say the sentence out loud in the meeting and watch the target move.
Each nine costs roughly ten times the one before it, and the shape of the work changes at every step. Getting from 99% to 99.9% is monitoring, an alert that reaches a human, and a runbook. Getting to 99.99% means redundancy in every layer of the path and automatic failover that you test on purpose, because failover you have never tested is a coin flip you have not priced. Getting to 99.999% means multiple regions, and somewhere around there you cross a line worth noticing: a mobile connection that drops one request in two hundred is delivering 99.5% to that user no matter what you do. You can spend a year buying a nine your users cannot receive.
Pick the target from what the downtime costs. Marlow's founder looked at 43.8 hours a year, decided a bookshop survives an unlucky afternoon a month, and moved on. That is a real decision, made once, written down. A company selling stadium tickets at 10:00 on a Saturday morning cannot make it.
Downtime adds
Lesson 001 multiplied three components at 99.9% and got 99.7%, and left it there. Here is the version you can do standing at a whiteboard.
Stagefront is this course's ticketing service, the one where a stadium show goes on sale at exactly 10:00 and two hundred thousand people press the same button in the same minute, and where an oversold seat is a lawsuit rather than an apology. A purchase crosses five things, and all five have to work.
| Component | Availability | Its own downtime |
|---|---|---|
| CDN | 99.99% | 53 minutes |
| Load balancer | 99.99% | 53 minutes |
| Application tier | 99.95% | 4.4 hours |
| Ticket database | 99.95% | 4.4 hours |
| Payment provider | 99.9% | 8.8 hours |
Multiply the five availabilities and you get 99.78%, which is 19.3 hours a year. Nobody multiplies five decimals in their head, and nobody has to, because when every number is close to 1 the product is very nearly 1 minus the sum of the gaps.
53 min + 53 min + 4.4 h + 4.4 h + 8.8 h = 19.3 hours
Same answer, no calculator. Downtime adds. Every component you put in the path of a request contributes its own outage hours to the total, and adding a component can only ever make the number worse.
Now look at where those hours come from. The payment provider, the one box on that diagram nobody at Stagefront wrote or operates, contributes 8.8 of the 19.3 hours. Forty five percent of the entire outage budget belongs to a company you can only email. Your dependency's published number is your ceiling for everything on that path, and there are two ways around it: stop needing it during the critical moment, which usually means accepting the order and charging later, or carry a second provider and the permanent cost of a second integration. Lesson 018 is the first of those and lesson 042 is the second.
One refinement that saves a lot of pointless worry. The chain is the request path, not the architecture diagram and definitely not the org chart. The admin reporting service can be down all week without touching a single purchase, so it contributes nothing to this number. Pick the one request that matters, trace it, and count only what it actually touches.
Two of something, if they really are two
The cure for a chain is a copy, and the arithmetic is intoxicating.
Marlow's one box is up 99.5% of the time. Put a second identical box behind a load balancer so that either one can serve the site, and the shop is down only when both are down at once:
0.005 x 0.005 = 0.000025 -> 99.9975% -> 13 minutes a year
Forty three hours to thirteen minutes for the price of one server. This is the most seductive calculation in the field and it is almost always wrong, because it assumes the two boxes fail independently, and Marlow's two boxes will not.
Walk the ways that box dies, the ones lesson 001 listed and two it did not. A disk fails, or the host gets retired: independent, and the second box carries on. A bad release at 4:50 on a Friday: both, within seconds of each other, because the deploy script does not know how to feel doubt. An expired TLS certificate: both, at the same second, on a date chosen a year ago. The publisher payout job that gets Postgres killed on the first Sunday: both, because there is still one database. A power event in the rack: both, if you put them in the same rack, which you did, because it was easier.
Say 32 of Marlow's 43.8 hours come from causes that hit both boxes together. The other 11.8 hours are per box hardware and host trouble, 0.135% each, so both boxes hitting that at the same moment is about two parts in a million, which is a minute a year and rounds away to nothing.
32 hours (shared causes) + ~1 minute (both boxes independently)
= about 32 hours -> 99.63%
The second server bought a 27% reduction in downtime, not the factor of two hundred the first calculation promised. Call it the shared cause is the ceiling: redundancy removes only the failures that are not shared, so the value of a copy is decided entirely by the size of the correlated part, and the correlated part is the number nobody computes.
Cloudflare paid for this lesson in public on 2 July 2019. Redundancy is most of what they sell: machines in well over a hundred cities, traffic routed around anything that fails. They pushed a new managed rule to their web application firewall, to every machine, everywhere, at once. The regular expression in that rule backtracked catastrophically, CPU went to 100% across the network, and traffic through Cloudflare fell by roughly eighty percent for about half an hour until an engineer used the global kill switch to turn the firewall off.
Every machine was redundant. The deploy was not. That is how this arithmetic actually breaks in the wild: not two disks dying on the same Tuesday, but one change arriving everywhere in the same second.
So when someone offers you redundancy, ask what it is redundant against. Two application servers behind a load balancer protect you from one host dying. They do nothing about a bad release unless the release is staged, nothing about a bad config unless config rolls out slowly, and nothing at all about the single database they are both talking to. Lesson 054 stages the change; lesson 011 copies the database.
How often, and how long
Underneath every availability figure sit two numbers that are far more useful than the figure itself. MTBF, mean time between failures, is how often the thing breaks. MTTR, mean time to recovery, is how long it stays broken.
availability = MTBF / (MTBF + MTTR)
Marlow breaks about once a month, so MTBF is around 720 hours, and a typical outage lasts three and a half hours from start to finish. That gives 720 / 723.5 = 99.52%, about 42 hours a year, near enough to the 43.8 that lesson 001 assumed.
Now break those three and a half hours into what actually happened on a bad Tuesday:
customer emails; the founder reads it 95 min
finds a laptop and gets connected 25 min
works out what is genuinely wrong 60 min
fixes it, restarts, watches it recover 30 min
-------
3 h 30
Thirty of those two hundred and ten minutes were repair. The rest was not knowing, and then not being sure. Almost every incident timeline looks like this once somebody writes it down honestly, whether the system has one founder or four hundred engineers.
Which makes the cheapest lever on availability an unglamorous one. An alert that reaches a phone at 10:44 instead of a customer email at 12:21 removes 95 minutes by itself. A runbook that says which four things to check removes most of the next hour. Drive MTTR from three and a half hours to twenty minutes and, with exactly the same monthly failure rate, annual downtime falls from 42 hours to 4, which is 99.95%.
Same number of failures. A tenth of the damage.
The reverse of that lever is worth respecting, because it is where the catastrophic numbers live. In October 2021 the online game platform Roblox was down for 73 hours, three full days, and the failure sat in the service discovery layer, a Consul cluster, which is the machinery the rest of the system uses to find its own parts. When that is what breaks, most of the tooling you would normally reach for to diagnose the problem is also broken.
Seventy three hours is 0.83% of a year. That single incident put a ceiling of 99.17% on the whole year, however flawless the other 362 days were, and if you wanted to average four nines with a 73 hour outage already on the books you would need the next one to arrive in 2104.
MTBF is what gets quoted in the sales deck. MTTR is the one you can change this afternoon.
Four promises that get called one word
Four different guarantees get flattened into the word "up", and they come apart in expensive ways.
Available means you can reach it and get an answer right now. Reliable means it keeps doing the right thing over a period rather than at an instant. Durable means the data is still there. Correct means the answer is true.
Amazon designs S3 for eleven nines of durability, 99.999999999%, while the availability commitment in its service level agreement is 99.9%. Eight orders of magnitude between two promises about the same product, on purpose, because they are promises about different things. Losing your object should be close to unthinkable. Not being able to fetch it for a few minutes is a Tuesday.
Nine hundred wind turbines, each sending a reading every two seconds, feed a telemetry service called Galewatch, which also exists only in this course. When the link to a farm drops for four hours in bad weather, the turbines buffer their readings locally and replay them when it comes back. For those four hours the ingest path for that farm is unavailable and not one reading is lost: availability broke, durability held.
What broke instead was freshness. The dashboard kept showing the last reading it had, with no hint that it was four hours old, so an engineer spent a morning looking at a healthy green tile for a turbine that had been feathered since eleven. Available and wrong is the failure mode nobody designs for and everybody ships.
Marlow has met the same shape from the other direction. A caching bug served a stale stock count for a day: every request returned a 200 in 8 ms, the uptime monitor was delighted, and the shop sold 40 copies of a book it had 12 of. Perfect availability, no correctness at all, and a week of apology emails. Lessons 008 and 009 are about that trade in detail.
When somebody tells you a system is up, the useful reply is to ask which of the four they mean.
Up is not a yes or no
Real systems fail in pieces, and a single number that counts every request equally will lie to you about which piece.
Marlow's degraded mode is a good one: serve the catalogue and search from cache, disable checkout, say so honestly on the page. Measured by requests, the shop is about 95% available in that state. Measured by revenue it is at zero. One SLI over all traffic reports a decent day.
The fix is to weight the definition by the request that matters, which in practice means having two SLOs rather than one. A browse target and a checkout target, with different numbers, because they cost different amounts when they break and they break for different reasons.
Stagefront pushes that idea to its limit. Forty on-sales a year, ten minutes of real intensity each, is 400 minutes a year in which the business actually happens, out of 525,600.
An eight hour outage at 02:00 on a Tuesday costs Stagefront an apology and shows up in the annual number as 99.91%. A ten minute outage at 10:00 on an on-sale morning costs a stadium and shows up as 99.998%. The annual average rewards the outage that destroyed the business and punishes the one nobody noticed.
So Stagefront runs two targets. A dull one for the site, and a brutal one for the window: at least 99.95% of purchase requests succeed in under two seconds between 09:55 and 10:15, measured per on-sale, pass or fail. Miss it once and you have missed it, and no yearly average is allowed to absorb it.
Call it the minute that matters. Availability averaged over a year describes your infrastructure. Availability measured over the windows where the money moves describes your business, and those two numbers can disagree completely.
The budget that makes a target real
Three initials, and the difference between them is money.
An SLI is the measurement. An SLO is the objective you hold yourselves to internally. An SLA is that same shape of promise written into a customer contract with a remedy attached when you miss it.
Keep the SLA looser than the SLO, always, so that you find out you are in trouble before your customer's lawyer does. If both are 99.9% there is no gap between the day you start failing and the day you start paying.
Engineers routinely over-read what an SLA buys. The remedy is nearly always a service credit, a percentage of that month's bill. A provider you pay 40,000 rupees a month goes down for two hours, the SLA offers a 10% credit, and you receive 4,000 rupees for an outage that cost you a weekend and a customer. An SLA tells you how seriously the provider takes their own number. It is not insurance and it was never priced as insurance.
The SLO is the one that changes behaviour, and it does so through the quantity underneath it. Whatever you did not promise is your error budget, and it comes in minutes you are allowed to spend:
30 days = 43,200 minutes
0.1% of 43,200 minutes = 43.2 minutes
Forty three minutes a month, at 99.9%, to spend on risky migrations, bad releases, cutovers and plain bad luck. That turns a target into a decision procedure, which is the only reason to have one. Budget left in the second week means you ship the scary migration on Wednesday. Budget gone means you stop shipping features and go fix whatever keeps eating it. It cuts the other way too: a team that reaches the end of every month with 40 of its 43 minutes unspent is being too cautious and could be shipping faster.
Which is why an SLO of 100% is worse than useless. Everyone nods at it in the meeting, and then it decides nothing, ever, because there is no budget to be out of.
Planned downtime still counts
Marlow's founder deploys twice a week, and each deploy takes the site down for about ninety seconds, because there is one process and it has to restart.
104 deploys x 90 seconds = 9,360 seconds = 2.6 hours a year
That is 6% of the shop's 43.8 hours, considerably less than the founder's anxiety about it would suggest. Two things follow, and they pull in different directions.
Nobody trying to buy a book at 16:50 on a Friday knows or cares that the downtime was planned. From outside, a maintenance window is an outage with better manners, and the honest internal number includes it. Contracts routinely exclude announced maintenance from the SLA, which is fair commercial writing and quietly corrosive engineering, because the moment your own dashboard stops counting planned downtime you stop paying for making deploys cheap.
And cheap deploys are the real prize here. The founder ships on Friday afternoons partly because every deploy hurts, so changes get batched into those two windows, so each release carries half a week of work, so each one is likelier to be the bad one that costs three and a half hours. Removing the ninety seconds removes the reason to batch. Lesson 054 covers how; the availability saving is 2.6 hours and the actual win is shipping a fifth of the risk five times as often.
Recap
The definition comes first. Availability is good events over valid events, and the four choices inside that fraction (what counts as an event, what counts as good, where you measure, over what window) decide the answer more than your architecture does. Write the sentence down before you argue about the number.
Nines are hours. 99.9% is 44 minutes a month; 99.99% is one minute a week; 99.999% is six seconds a week and probably invisible to a user on a mobile connection. Each nine costs about ten times the last one, and the work changes shape at every step.
Downtime adds. In a chain where everything must work, add the outage hours rather than multiplying the percentages. The component you do not operate is usually the biggest term, and the chain is the request path, not the architecture diagram.
The shared cause is the ceiling. Two copies only remove the failures that are not shared. Bad deploys, expired certificates, one database and one rack hit both at once, which is why Marlow's second server turned 43.8 hours into 32 rather than 13 minutes, and why one regex rule took most of Cloudflare's traffic off the internet for half an hour in 2019.
How often times how long. Availability is MTBF over MTBF plus MTTR, and almost all of MTTR is detection and diagnosis rather than repair. Finding out sooner is the cheapest nine you will ever buy, and one 73 hour outage caps a year at 99.17% no matter what else you do.
The minute that matters. Up is not a yes or no: a shop with checkout disabled is 95% available by requests and 0% by revenue. Weight the definition by the requests that carry the value, and measure the on-sale window separately from the year.
Check your understanding
Write the SLI for Marlow's checkout as one sentence, then say what the 99.99% figure from the founder's uptime monitor would have looked like under your definition on the podcast Friday.
A Stagefront purchase crosses a CDN at 99.99%, a load balancer at 99.99%, an application tier at 99.95%, a database at 99.95% and a payment provider at 99.9%. Someone proposes adding a fraud check service at 99.99% to the purchase path. What does that cost in hours a year, and what would you ask for in exchange?
Marlow's box fails about once a month and takes three and a half hours to recover. You have budget for exactly one of: a second server, or paging alerts plus a runbook. Which do you buy, and what is your estimate of the downtime each one removes?
Galewatch's link to a farm drops for four hours and the turbines buffer readings until it returns. Which of availability, durability and correctness broke, and what would you put on the dashboard so the engineer at the other end is not misled?
Your team's SLO is 99.9% over 30 days and you have spent 38 of your 43.2 minutes by the 20th. A product manager wants a database migration shipped on the 22nd. What do you say, and what would you have said if you had spent 4 minutes?
Next lesson
004 The Request's Path: DNS, TCP, TLS and HTTP in Plain Terms. Today you counted components in a request path and multiplied their failure rates; next lesson opens that path and shows you what a browser is actually doing in the 180 milliseconds before your server sees a single byte.