Marlow Books is a four person online bookshop that exists only in this course, and on a Sunday in March its founder spent six hours at a cousin's wedding while the shop sold nothing at all.
The graphs looked wrong in a way that took a while to name. Requests per second at nginx had gone from the usual forty to about two. Not errors. Just absence. CPU flat, Postgres bored, disk quiet, and the health check green every ten seconds all morning, because the health check curls http://localhost:8080/health from inside the box and had never in its life been asked to leave the machine.
One graph was normal. Connections per second sat exactly where it always sat.
At 11:40 a customer sent a photograph of their phone. Full red screen. Your connection is not private.
The TLS certificate had expired at 05:30 that morning. It had been renewed by hand every year since the shop launched, six times, because the founder set the site up before that was easy to automate, and the reminder lived in a calendar on a phone that had been replaced in October. Renewing it and reloading nginx took eleven minutes once they knew. The shop had been unreachable for six hours and twenty one minutes, and its own monitoring had watched the whole thing without raising a finger.
Here is the part worth keeping. Not one of those visitors sent a request. Their browsers opened a connection, looked at what came back, and refused to ask the question. The shop was measuring a room that nobody was allowed to enter.
Lesson 003 put that in one clause: server side numbers exclude, by construction, every failure that happened before the request reached your server. This is that clause with a bill attached. Four things have to go right before your code hears a single byte, and only one of them is yours.
Four round trips before byte one
Nine hundred wind turbines, each reporting every two seconds, feed a telemetry service called Galewatch, which is also invented for this course. One of its wind farms is Tarrow Ridge: sixty turbines, and a site office in a portable building on a fixed wireless link with a 45 millisecond round trip.
Lesson 002 broke that office's nine second dashboard into five pieces and charged the first 180 milliseconds to "DNS, TCP, TLS setup, four round trips before byte one". That line was a promissory note. Here is what it buys.
site office Galewatch
|
|-- what address is the API? --->| resolver
|<-- 203.0.113.10 ---------------| 45 ms
|
|-- SYN ------------------------>|
|<-- SYN, ACK -------------------| 45 ms
|-- ACK + ClientHello ---------->|
|
|<-- ServerHello, certificate ---| 45 ms
|-- Finished, GET /farm/tarrow ->|
|
|<-- 200 OK, first byte ---------| 45 ms
------
180 ms
Four crossings, each one costing whatever the distance costs, and not one of them carries a single byte the product cares about. The name lookup finds an address. The TCP handshake agrees that both ends are there and can count. The TLS handshake proves the server is who the name says and agrees on a key. Only then does anyone mention a wind turbine.
A handshake is an exchange both sides must finish before anything useful passes. The word is doing real work: nobody speaks until everybody has agreed how to speak.
Now run the same code from an office five milliseconds from the data centre.
Tarrow Ridge 4 x 45 ms = 180 ms
city office 4 x 5 ms = 20 ms
Nine times the setup cost, same software, same server, same certificate. This is lesson 002's speed of light tax, and the thing to notice is the multiplier out front. You do not pay the distance once. You pay it four times before your application has been told that anyone is there.
Turning a name into an address
The browser has api.galewatch.example and needs something it can send a packet to. The system that converts one into the other is DNS, and for most engineers it is a black box that occasionally ruins a Tuesday.
Two pieces of vocabulary make the whole thing legible.
A resolver is the machine that does the looking up on your behalf, usually run by your network or by a public service, and it caches everything it learns. An authoritative nameserver is a machine that holds the real answer for a domain, because whoever owns that domain pointed at it.
When the resolver already knows the answer, your lookup is one round trip and you are done. When it does not, it walks down:
resolver -> a root nameserver "ask the .example servers" 15 ms
resolver -> a .example nameserver "ask ns1.galewatch.example" 15 ms
resolver -> ns1.galewatch.example "203.0.113.10" 40 ms
Seventy milliseconds of walking, plus the 45 to reach the resolver and come back, so a cold lookup from Tarrow Ridge costs 115 milliseconds instead of 45. Your 180 millisecond page just became 250, for a step whose entire answer is four bytes of address.
That almost never happens, and the reason is the number attached to every answer. A TTL, time to live, is how many seconds a resolver may keep an answer before asking again. Set it to 86,400 and every resolver in the world asks once a day. Set it to 60 and they ask every minute.
Marlow's founder met the trade from the wrong side. When the shop finally moved to a new host, they changed the address record at 21:00 on a Tuesday, watched the new box get almost nothing for an hour, and went to bed pleased that the old box was handling the load so calmly. Orders kept landing on the old box all through Wednesday, and a few more arrived on Thursday morning. The TTL was 86,400, the default the registrar's control panel offers, so a resolver that had asked at four on Tuesday afternoon was entitled to its stale answer until four on Wednesday afternoon.
The Thursday stragglers were a different animal, and worth knowing about: clients that resolve a name once when the process starts and then hold that address for as long as they run. No TTL reaches those. Only a restart does.
The fix is boring and you have to do it a day early: drop the TTL to 60 seconds, wait out the old TTL, then move, then put it back.
Then put it back is the half people skip, and it matters more than the move did. A 60 second TTL means every resolver on earth asks your nameservers again every minute, which drags them into the request path of very nearly every visit. Lesson 003's rule was that downtime adds along the request path, and DNS is the component nobody draws on the diagram. With a 60 second TTL, a ten minute DNS outage takes your whole site down within a minute even though every server you own is healthy. With a 24 hour TTL, most of your visitors never notice it happened.
Call it the TTL dial: it sets how fast you can change an answer against how long a cache can cover for you when you cannot give one. There is no correct setting, only a choice about which day you would rather have.
The internet paid this bill together on 21 October 2016. Dyn, a managed DNS provider, was hit by a very large distributed denial of service attack from the Mirai botnet, which was mostly compromised consumer cameras and video recorders. It came in waves through the day, and for hours a lot of users could not reach a long list of major sites. Those sites were fine. Their servers were up, their databases were up, their engineers were staring at healthy dashboards. Nobody could look up where they lived.
The cheap defence is one most teams never buy: delegate your domain to nameservers from two independent providers, so one of them going dark leaves the other answering. It costs a little money and an afternoon, and it removes a single point of failure sitting one layer above everything lesson 001 taught you to worry about.
DNS hands you an address and stops there. How that address gets chosen to be near you, and how a content delivery network abuses this step on purpose, is lesson 022.
The pipe, and the ramp nobody mentions
With an address in hand, the browser opens a TCP connection: SYN out, SYN and ACK back, ACK returned. Three packets, one round trip of waiting, and in practice the client's first real bytes go out with or immediately behind that third packet.
That is the famous part. The part that shows up in your latency numbers is what happens next.
A fresh TCP connection does not start at full speed. The sender has no idea what the network between here and there can absorb, so it starts small and doubles, which is called slow start. The opening window is about ten packets, roughly 14 kilobytes, and it doubles every round trip: 14, 29, 58, 117.
Lesson 002 said one round trip's worth of bytes on the Tarrow Ridge link is bandwidth times round trip time, 20 megabits a second times 45 milliseconds, about 110 kilobytes. So it takes four round trips, 180 milliseconds, before that connection is finally using the link it was given, and about a tenth of a second of transmission never happens at all.
That is why lesson 002's 2.4 megabyte bundle measured near two seconds when the division said one. Part of it is the ramp. The rest is a portable building's link being shared with everything else in the building.
Slow start is not a bug and you cannot turn it off, because it is the mechanism that stops every sender on earth from flooding every link on earth. What you can do is stop paying for it repeatedly, which is a few sections down.
Proving who is on the other end
TLS is the layer that turns HTTP into HTTPS, and it does two jobs that get muddled. It encrypts the conversation, which everyone remembers. It also proves the server is the one that owns the name you asked for, which is the job that failed on Marlow's Sunday.
The proof is a certificate: a file containing the server's public key, the names it is valid for, a date range, and a signature from a certificate authority, an organisation your browser already trusts. The client checks the signature chains to something in its trust store, that the name matches what it asked for, and that today's date is inside the range.
Version matters here in milliseconds. TLS 1.2 needs two round trips to finish. TLS 1.3, standardised in 2018, needs one, because the client guesses which key agreement the server will pick and sends its half in the very first message. The 180 millisecond sketch above assumed 1.3. On 1.2 the same page costs 225.
Two details from the handshake are worth carrying around.
The name you asked for travels in the clear. The client has to say which site it wants before encryption is set up, because a single address often serves hundreds of certificates, so the hostname rides in the first message in plain sight. Anyone on the path can see that you asked for Marlow Books, though not what you did there. Work to close that gap exists and is not yet universal.
And the date in the certificate is checked by every client, on every new connection, with no grace period and no negotiation. Lesson 003 called an expired certificate a correlated cause, the kind that takes every copy of your service at the same second on a date chosen a year earlier. Redundancy is worthless against it. Two servers, twelve servers, three regions: the same file expires on all of them simultaneously, and 003's shared cause is the ceiling arrives in its purest form. Short lifetimes are what actually fixed this industry wide. When a certificate is good for ninety days, nobody renews it by hand, and the automation you are forced to build is the thing that saves you. Lesson 053 covers rotating keys properly.
One more TLS 1.3 feature is worth knowing, mostly for where the bill on it arrives. A returning client can send its request in the very first packet, using a key kept from a previous session. Zero round trips of setup. Free.
The catch is that early data can be replayed. Anyone who captured those bytes can send them again later, and the server has no way to tell the copy from the original. For GET /book/1428 that is a shrug. For POST /checkout it is a second charge on somebody's card. So a transport feature lands squarely on a question about your handlers: which of your requests are safe to receive twice? Lesson 019 is that question, and it will keep coming back for the rest of the course.
Open it once
Everything above is the cost of a cold start, and the single largest win available to you is refusing to pay it twice.
HTTP has kept connections open by default since 1.1. Connection reuse, or keep-alive, means the browser sends a second request down the pipe it already has: no DNS, no SYN, no certificate, no ramp. One round trip instead of four.
Go back to lesson 002's Tarrow Ridge dashboard, the one making 61 sequential API calls. Each of those cost 45 milliseconds, which is 2,745 milliseconds of waiting, and it was already the worst thing on the page. Now suppose the client had opened a fresh connection per call.
61 calls x 180 ms = 10,980 ms
plus 300 ms of server time = about 11.3 seconds
The nine second page would have been seventeen. Keep-alive was quietly saving that dashboard eight seconds and got no credit for it, which is normal, because the optimisations that work are invisible and the ones that fail are famous.
Servers get this wrong more often than browsers do, because a browser has good defaults and your HTTP client library might not.
Stagefront is this course's ticketing service, where a stadium show goes on sale at exactly 10:00 and two hundred thousand people press the same button in the same minute. Its checkout calls the third party payment provider from lesson 003, the one contributing 45% of the whole availability budget. That provider's API sits about 30 milliseconds away and takes about 120 milliseconds of its own time per charge. At peak the tier pushes 500 payment calls a second, and all of Stagefront's outbound traffic leaves through a single network address.
Suppose every charge opens a fresh connection. Lesson 002's busy room, concurrency equals throughput times latency, prices it immediately:
warm connection 500 x 0.150 s = 75 calls in flight
fresh connection 500 x 0.210 s = 105 calls in flight
Forty percent more concurrency for nothing. Annoying, survivable. The thing that actually breaks is a limit lesson 001 listed in a throwaway line between file descriptors and inodes.
Every outbound connection needs a local port number, and on Linux the default range gives you about 28,000 of them. When a connection closes, the side that closed first holds the socket in a state called TIME_WAIT for 60 seconds, and for an outbound API call that side is usually yours. So one address can start roughly 28,000 divided by 60, about 470 new connections a second to one destination, and then it runs out.
Stagefront needs 500.
At 10:01 the application starts logging "cannot assign requested address", CPU sits at 30%, the payment provider's dashboard shows nothing wrong because the calls never arrive, and every engineer in the room is looking at the wrong system. The fix is a persistent connection pool, which is usually one line of configuration: 75 connections held open, instead of 30,000 opened and thrown away every minute.
The reason this one costs a whole evening rather than ten minutes is that the first hour always goes into the service that looks slow. The port table is not on anybody's dashboard until the night it is.
One connection, many requests, one lost packet
HTTP/1.1 has an awkward rule: one request at a time per connection. Ask for the next thing only when the last answer has fully arrived. There is a feature called pipelining that was meant to fix this, and it is disabled everywhere, for good reasons that no longer matter.
Browsers worked around it by opening about six connections per host, which is six lots of everything above.
HTTP/2 fixed it properly. Many independent streams share one connection, so a page fetches sixty things over one pipe, and the four round trips of setup get paid once instead of six times. On a good link it is a clear win and you should take it.
Here is what it does not fix, and this is the part textbooks skim.
TCP delivers bytes in order. If packet 40 goes missing, the kernel holds packets 41 onward, already sitting in memory, until the retransmission of 40 arrives. It has no idea those later packets belong to different streams. Everything waits. That is head of line blocking: the item at the front of the queue stops everything behind it, including work that has already finished.
Tarrow Ridge in bad weather is where this stops being theory. Say the fixed wireless link is dropping one packet in a hundred. The dashboard's sixty JSON responses are about 24 kilobytes in total, roughly seventeen packets.
chance all 17 arrive first time = 0.99^17 = 0.84
chance at least one is lost = 16%
That is lesson 002's tail arithmetic in a new costume. One load in six loses a packet, and when it does, all sixty responses wait at least one round trip for the retransmission, often more. Over six HTTP/1.1 connections the same loss stalls a sixth of the work. Which is the genuinely surprising conclusion: on a lossy link, HTTP/2 can measure worse than the thing it replaced, and the site office is exactly the kind of place where that happens.
HTTP/3 is the answer, and it is a bigger change than it looks. It runs over QUIC, which is built on UDP, so it does its own ordering per stream: a lost packet stalls only its own stream and the other fifty nine responses go straight to the application. QUIC also folds the transport and the crypto handshake into a single round trip, which takes a cold visit from four round trips to three.
HTTP/2 over TLS 1.3 DNS + TCP + TLS + request = 180 ms
HTTP/3 over QUIC DNS + QUIC + request = 135 ms
One caveat that nobody puts on the slide: you rarely get HTTP/3 on a first visit. The browser learns the server speaks it from an Alt-Svc header on an earlier response, or from a DNS record that advertises it. So the protocol that saves a round trip on a cold connection typically needs a warm one first, unless you publish the DNS record, which is precisely why that record type exists.
What the number in the answer promises
Status codes look like trivia and are actually a contract, mostly about one question: is it safe to send this again?
The families first. 2xx worked. 3xx means look somewhere else. 4xx means the caller sent something wrong. 5xx means the server broke.
The working rule that falls out: a 4xx will fail the same way if you resend the identical request, so retrying it just burns capacity, while a 5xx might succeed next time. One code breaks that rule and one code makes it precise, and both are worth memorising. 429 Too Many Requests is a 4xx that will succeed later, because nothing was wrong with what you sent, only with when you sent it. 503 Service Unavailable is a 5xx that means out of room right now rather than broken. Both often carry a Retry-After header telling you when to come back, and honouring it is the difference between backing off and joining a stampede. Lesson 021 builds the limiter that sends the 429; lesson 020 builds the client that listens.
Now the case that actually hurts, which has no status code at all.
Your request times out. Nothing came back. There is no number, and lesson 001 said why this is worse than an error: an error is information, and silence is not. The server may have charged the card and died before answering, or never received the request. From where you are standing those two look identical, and they always will.
HTTP names GET, HEAD, PUT and DELETE idempotent, meaning sending them twice has the same effect as sending them once, and pointedly does not include POST. That naming is a promise about your handlers that only you can keep, and lesson 019 is about keeping it.
One opinion, strongly held. Returning 200 OK with an error inside the body is among the most expensive API decisions I have watched teams make. Every load balancer, cache, retry policy, alert and dashboard between you and your caller reads the number and never opens the body. Lesson 003 has the bookshop version already: a caching bug served a stale stock count for a day, every response a 200 in 8 milliseconds, the uptime monitor delighted, and 40 copies sold of a book the shop had 12 of. If the answer is wrong, say so in the number.
Recap
Four round trips before byte one. A cold request pays a name lookup, a TCP handshake, a TLS handshake and then the request itself, and every one of them costs the full distance. 180 milliseconds from a wind farm, 20 from across town, before your code has heard that anybody wants anything.
The TTL dial. A TTL trades how fast you can change an answer against how long a cache can cover for you when you cannot give one. Low TTLs put your nameservers in the path of every visit; high ones mean a migration takes two days. Move it down a day early, then move it back.
Open it once. Connection reuse skips the lookup, both handshakes and the slow start ramp, turning four round trips into one. It is the largest free win here, and forgetting it server side runs you out of local port numbers at around 470 new connections a second per address, long before it runs you out of anything you were watching.
Head of line blocking. TCP delivers in order, so one lost packet stalls every HTTP/2 stream sharing that connection. On a link dropping one packet in a hundred, a seventeen packet page waits one time in six. QUIC exists because that stall has no good fix inside TCP.
The timeout has no status code. 4xx is the caller's problem, 5xx is yours, 429 and 503 mean later rather than never, and the failure that will actually cost you money is the one that returned nothing at all and left you unable to find out whether the work happened.
Check your understanding
Marlow Books is moving to a new host next Wednesday and its DNS records have a TTL of 86,400 seconds. Write the sequence of changes you would make, with timings, and say what you would set the TTL back to afterwards and why.
A mobile app makes 8 API calls when it opens, one after another, over a connection with a 70 millisecond round trip. Calculate the total setup and request time both with and without connection reuse, and say which single change you would make first.
Your service calls a payment provider 300 times a second from one machine, opening a new HTTPS connection each time. Estimate how many local ports that consumes, say whether it survives, and name the error message you expect to see first.
A colleague proposes enabling TLS 1.3 zero round trip resumption across your whole API, to save a round trip every time a returning client reconnects. Which endpoints would you allow it on, and what question would you ask about every other one?
An internal service returns
200 OKwith{"status": "error"}when its downstream dependency is unavailable. Name three specific pieces of infrastructure that will now behave incorrectly, and say what the response should have been.
Next lesson
005 Vertical vs Horizontal Scaling: Bigger Box or More Boxes. Today you followed one request from a name to a first byte; next lesson goes back to lesson 001's unfinished argument and decides, with numbers, when a bigger machine is the right answer and when it is the expensive way to postpone one.