CDNs and Edge Caching: Moving Bytes Closer

Marlow Books is a four person online bookshop that exists only in this course, and in the last week of February its founder finally did the one thing in this whole course that nobody argues about.

Lesson 016 had laid it out in November. The shop keeps 1.2 million cover images, 180 kilobytes each, 216 gigabytes in total, and once they moved to object storage in December the bill arrived itemised for the first time: $5.40 a month to keep them, $4.16 to answer the requests, and $187 to push the bytes down the wire. Lesson 016's closing line was that an immutable file with a hashed name is the most cacheable object in computing.

So the founder pointed a CDN at the bucket, changed the cover URLs on the book page to the CDN's hostname, and went back to the limiter work from lesson 021. Twenty minutes, start to finish.

A week later they opened the CDN's dashboard expecting a number starting with nine.

Hit rate: 62%.

Thirty eight cover fetches in every hundred were still going all the way back to the bucket, on files that are written once and never touched again. The founder did what anyone does with a bad hit rate and went looking for the TTL. Lesson 009 wrote that header down back when nginx on box A still served the covers itself, and December's move to object storage had carried it across onto every object: one year. There is nothing above a year.

The files could not be fresher. The expiry could not be longer. The hit rate was 62% and there was no dial in the building attached to it.

A CDN is lesson 008's third row, rented

Strip the marketing off the name and a content delivery network is a cache that somebody else runs, in a lot of cities at once. Lesson 008 already drew the shelf it sits on: in the process, in a shared store, at the edge, in the browser. Today is the third row, and the mechanism in it is one you learned fourteen lessons ago. Keys, values, TTLs, hits, misses, evictions. Nothing new.

Two things are new, and both of them are facts about the world rather than about caching.

The first is the origin, which is whatever holds the real copy. For Marlow's covers the origin is the object storage bucket; for a page it would be the boxes behind lesson 006's balancer. Everything the edge does not have, it asks the origin for. The customer never speaks to your origin again. The edge does.

The second is the fleet. An edge, also called a point of presence, is a rack of caching servers in a city, shared by every customer that CDN has. Your request finds a near one because the hostname resolves to an address announced from many places at once, which is anycast: the internet's own routing table decides which announcement is closest and sends you there. Lesson 006 ended on exactly this, when it said the chain of balancers has to stop somewhere and it stops at DNS or at anycast, neither of which belongs to you. A CDN is that ending, sold as a product.

Here is the part of the fleet that nobody puts on the marketing page. Marlow's CDN advertises well over a hundred locations. In the first week, nine of them ever saw a request for a Marlow cover, because nine is roughly how many cities Marlow's customers live in.

You do not get a hundred edges. You get as many as your customers have cities.

Why an unchanging file misses four times in ten

Lesson 016 priced an ordinary month at 001's forty requests a second: about 38 book pages a second, a tenth of them fetching an image, so four cover fetches a second across the whole shop. That tenth is a guess and 016 says so, and the mix under it is 006's Christmas one. Spread those four a second evenly over 1.2 million covers and any given cover is asked for once every 300,000 seconds, which is three and a half days.

Now cut the traffic nine ways. Each edge sees about 0.44 cover fetches a second, and evenly spread that is one request per cover per edge about once a month.

No cache on earth holds an object for a month to serve one request.

Traffic is never even, which is the only reason any of this works. Lesson 008 said it plainly: a few thousand titles carry most of a bookshop's day. The bestseller's cover is asked for every few minutes at all nine edges and is never gone. A few hundred titles down, the gap between two asks at one edge is a day. A few thousand down, a week. Out in the tail it is the month we just worked out.

So the hit rate is one crossing: how far down the catalogue that gap outlasts whatever the shelf holds. For Marlow it crosses at 62%, a measurement of the bookshop and not of the CDN.

There is a second mechanism underneath, and it catches everybody who has only run their own Redis.

Your TTL is a maximum, not a reservation. You are renting space on a shelf shared with every other customer of that CDN, and what actually decides how long your object stays is eviction, which is their decision and not yours. A year means "you may keep this for up to a year". It has never meant "keep this for a year". Lesson 008 watched a search engine crawler walk 40 distinct ISBNs a second and push the bestsellers out of a 256 megabyte Redis; at an edge you live permanently inside somebody else's version of that, with a working set you can neither see nor influence.

An edge cache's hit rate is your traffic's concentration divided by the number of edges. Both terms belong to the world. Neither is a setting.

The 62% was not worthless, and the arithmetic is better than it looks.

Of lesson 016's 1,872 gigabytes a month, 38% still leaves the bucket, which is 711 gigabytes, so the egress line falls from $187 to about $71 and 1,161 gigabytes a month never leave the origin at all. The request line falls the same way: 10.4 million origin requests become 3.95 million.

And then the correction that matters, which is lesson 008's word for word. A cache buys your median and leaves your tail. A cover that misses at the edge now costs the customer a trip to the edge, a lookup that finds nothing, a fetch from the edge to the bucket, and then 180 kilobytes coming back the long way. Before the CDN it was one fetch from the bucket. So the 62% got faster and the 38% got slower, and at a miss rate of 38% that slow path is not a tail event at all. It starts just above the median.

And one thing people believe about CDNs is not true. Every one of those 1,872 gigabytes still travels down a wire to a customer, and somebody still sends you a bill for it. A CDN does not reduce the bytes you ship. It changes who ships them and how far. What it genuinely removes is origin work: requests your boxes never see, bytes your storage never reads.

A cache in front of the caches

The fix for dilution is not a bigger TTL. It is another cache.

An origin shield is one designated edge that every other edge misses through. Marlow's nine edges stop talking to the bucket and talk to the shield instead; only the shield talks to the bucket.

Two things happen. The second is the reason to do it.

The shield sees everything the nine edges missed: 3.95 million requests a month against 1.2 million distinct covers that never change and take up 216 gigabytes, which is a small disk. Assuming the shield has room for all of it, which is a question to ask your CDN, after the first month it holds almost everything it is ever asked for and the bucket's egress falls from 711 gigabytes to whatever is genuinely new that month.

The second thing is fan-in. Without a shield, a cover that is cold everywhere produces nine simultaneous misses for the same bytes, one per edge. That is lesson 008's stampede, arriving once per edge, and nine is small only because Marlow is small. A real fleet makes it hundreds. The shield turns all of them into one request, because the other eight arrive to find the first one already in flight or already answered.

Marlow's founder turned it on that same afternoon, which took about four minutes.

If you take one configuration decision out of today, take this one. It is off by default, it costs a little extra latency on a miss, and it is the difference between your origin seeing your miss rate and seeing your miss rate multiplied by your geography.

Every key is a bet

In March the founder went further and put the book page HTML behind the same CDN, cached for ten minutes.

The default cache key at every CDN I have used is the method, the hostname, the path and the query string. That is all. Not the cookie. Not the customer.

Marlow's book page, since lesson 009's split was built in May, is a cached shell plus one primary key read for the live price and stock. Around that sits a header that says hello and counts the basket, which since lesson 007 comes from the session row and the customer it points at.

At 10:02 on a Tuesday, a signed-in customer called Anand loaded a book page. His response, with his name and his three basket items in the header, went into the edge nearest him under the key /book/9780140449136. That title was one of the month's better sellers at roughly a third of a request a second, so about two hundred people asked for it over the next ten minutes. Each of the nine edges had kept its own first asker, so those two hundred met nine different strangers, and the twenty odd in Anand's city got his name and his three books.

At 10:12 Anand's entry expired and whoever asked next at that edge donated their header to the following ten minutes. Most of the time that person was signed out and the page was harmless, which is why this had been happening on and off since Monday and nobody had said anything.

Lesson 010 taught that customers do not email you when a site is slow, they leave. They do email you when it calls them by a stranger's name. One did, at 10:31. The founder took the book pages off the CDN at 10:44 and left the covers where they were.

It could have been much worse. Most CDNs refuse outright to cache a response carrying a Set-Cookie header, because handing somebody's session to two hundred people is a different category of Tuesday. Anand's page carried no such header, because 007 chose an absolute session expiry over restamping the cookie on every request, so the CDN stored the page and the one default that guards against this never had anything to fire on. Nothing catches a name in a body.

Which is the whole of cache key design: the key must contain everything the response depends on. Anand's response depended on a cookie. The key did not mention cookies. Everything after that was arithmetic.

The obvious repair is to widen the key until it does mention cookies, and it is wrong, because a key that includes the session cookie gives every visitor a private copy of every page and a hit rate of nothing. Fix the document, not the key.

The book page is cacheable at all because lesson 009 split it: shell in the cache, stock and price read live from the row. The edge needs the same split one radius further out. The document the edge stores contains nothing belonging to any one person, and the name and the basket count arrive in a small second request once the page is on screen. The split you do for your own cache is the split that lets you use somebody else's. The book pages went back behind the CDN in the last week of March.

You can get this wrong in both directions, though, and only one of them is loud.

The quiet direction is the common one, and Marlow met it in April. The spring newsletter went out with ?utm_source=newsletter&utm_campaign=april appended to every link, and because the query string is in the key by default, every one of those links was an object the edge had never seen, while the warm copies under the clean URLs sat there untouched. Nine edges, every newsletter title, cold inside the ten minutes of the burst. Append a per-subscriber id instead and the hit rate really is zero. Nobody emailed about either. The fix is to tell the CDN which query parameters actually change the response and ignore the rest, as an allowlist rather than a blocklist, because May's campaign will invent a new parameter and an allowlist ignores it for free.

The second way to widen a key is a header you send rather than a setting you choose. Vary is a response header meaning "this response depends on that request header, so key on it too". Vary: Accept-Encoding is correct and costs you one copy per compression format. Vary: User-Agent keys your cache on the number of distinct browser version strings in the world, which is effectively unbounded. Vary: Cookie, on a site where every visitor carries a session cookie, gives you a cache with a hit rate of zero and a monthly invoice.

The five directives worth knowing

Cache-Control is the response header that tells every cache between your code and the customer what it may do. Lessons 008 and 009 kept reaching for this vocabulary and did not have it.

The pair that changes how you design is the first two. Cache-Control: max-age=0, s-maxage=60 says the copy in a stranger's browser, which lesson 008 reminded you cannot be recalled at all, lives for zero seconds, while the copy at the edge, which you can purge, lives for a minute. Staleness is the price of distance, and this is the header where you pay it a different amount per row of lesson 008's table. Most people set one number for both and then wonder why a bad deploy is still on somebody's laptop.

stale-while-revalidate is lesson 008's stampede fix, moved outward and improved. When the freshness window ends, the edge hands the stale copy straight back and fetches a new one behind it, so nobody waits on your origin and the thundering herd is one request. Lesson 008's jitter only ever fixed the synchronised variant of that problem. This fixes the general one, at the cost of one window's worth of staleness for whoever asks first.

Now the rule that decides whether any of this is worth your afternoon. If one key gets more than one request per TTL, you have a cache you are not using. Stagefront is a ticketing service that exists only in this course, where a stadium show goes on sale at exactly 10:00 and two hundred thousand people press the same button in the same minute. Lesson 017 measured the arrivals at 3,333 requests a second. Put s-maxage=1 on the on-sale landing page and the origin sees one request a second per edge instead of its share of 3,333. The page is up to a second stale, which during an eighty second sale is a genuine cost and not a free lunch, and lesson 023 owns the vocabulary for arguing about it.

Purge is a message. A new name is not

Lesson 009 framed invalidation as a message that has to arrive, against a TTL that needs nothing from anybody. At the edge it is the same message with a great many more addresses on it.

A purge is a request asking the CDN to drop an object from every node holding it. It works, usually in seconds, and you should still treat it as a convenience rather than something you rely on. Lesson 009 gave the first reason: it never reaches a browser, and a copy in a stranger's browser cannot be touched at all.

The second reason is new today. A purge is a stampede you fired on purpose. Drop a popular object from every edge at the same instant and every edge misses at the same instant, so your origin gets one request per edge for the identical bytes. Nine, for Marlow. Hundreds on a real fleet. The shield stands between you and that, which is its second job.

And the button that purges everything is the button that switches your CDN off. Your origin has been sized against a hit rate for weeks. Lesson 020 named half of that shape: more load than usual, at one instant, and no more capacity than yesterday. A purge-all during Christmas week is lesson 008's cold Redis at a new radius, with the difference that you pressed it yourself while trying to fix something smaller.

Which is why the answer is the one Marlow already had. Lesson 009 renames a changed cover to /covers/9780140449136-b41f9c.jpg and changes the reference on the page. A new name is an object nobody has ever held, so there is nothing to invalidate anywhere in the world, including the browsers you cannot reach. It also earns the long expiry honestly: a URL whose name changes whenever its bytes change can carry a year and immutable without lying to anyone.

A stable URL buys you a short TTL and a purge you hope about. A versioned URL buys you a year and no message at all.

What an edge moves, and what it cannot

Galewatch collects a reading from each of nine hundred wind turbines every two seconds, and its engineers watch a dashboard that has to be seconds fresh. Lesson 008 ruled that read uncacheable at any TTL worth having. A hundred edges do not change it. An edge cache moves bytes. It does not move decisions.

Except that the edge still earns its place on that dashboard.

Lesson 004 counted four round trips before byte one: a name lookup, a TCP handshake, a TLS handshake, and then the request itself. From the site office at Tarrow Ridge, a farm of sixty turbines on a fixed wireless link 45 milliseconds from the Galewatch region, that is four times 45, which is the 180 milliseconds lesson 002 charged to setup. Put an edge 10 milliseconds away and three of those four round trips now end at the edge instead of at the region.

direct, cold   DNS 45 + TCP 45 + TLS 45 + request 45   = 180 ms
via an edge    DNS 45 + TCP 10 + TLS 10 + request 55   = 120 ms

The name lookup still costs the full 45, since it does not go to the edge. The two handshakes cost 10 each instead of 45 each. The request itself costs 5 milliseconds out to the edge, 45 on the connection the edge already holds open to the region, taking the edge as no nearer the region than the office, and 5 back to the office, which is 55. Forty five and ten and ten and fifty five is a hundred and twenty.

A hundred and twenty milliseconds instead of a hundred and eighty, on a response that was never cached and never could be. An edge earns its keep on the requests it cannot cache, by ending the handshakes near the customer and keeping a warm connection open to you. It is lesson 004's "open it once", done on your behalf for strangers who have never visited.

Notice what is left. DNS is the one of the four that an edge does not move, and forty five of the remaining hundred and twenty milliseconds are a name lookup that has fetched nothing yet, which makes lesson 004's TTL dial suddenly interesting again.

Then the thing an edge quietly makes worse, and it is lesson 012's.

Lesson 012 timed a customer posting a review on fibre in the same city as box A: the write reached the replica 23 milliseconds after it committed, and the customer's next read arrived 15 milliseconds after it committed. The copy lost the race by eight milliseconds. Lesson 012's sharpest finding was that the customers closest to you lose, because a short round trip is a short gap.

Cache that page at the edge for sixty seconds and you have made every customer a close one. The gap is still fifteen milliseconds for the ones on fibre, and now it is being measured against sixty thousand milliseconds of copy rather than twenty three. An edge cache is replication lag you chose, with none of the instruments lesson 012 gave you: no log position to carry, no synchronous standby, no heartbeat row. So never let an edge cache a document that a person's own write is supposed to change. That is what private is for, and more often it is the document split from the Anand section, one more time.

Last, the term you just added to your availability product. Lesson 003 put a CDN at 99.99% in Stagefront's five component purchase path, which was honest of it: a cache in front of your service is a component in front of your service. 99.99% is 53 minutes a year. On 8 June 2021 a single customer's configuration change at Fastly triggered a bug that had been sitting in software deployed the previous month, about 85% of its network began returning errors, and a long list of very large sites were unreachable for the best part of an hour. One incident, near enough one year's entire budget. Lesson 027 owns what you do about that.

Marlow did not buy an edge for speed. Lesson 012's customers came back a median of 45 milliseconds after their own writes, which is a country rather than a planet. It bought one to take 1,161 gigabytes and six and a half million requests a month off an origin that had already filled its own disk once. The latency was a rounding error on top.

Recap

A CDN is lesson 008's third row, rented. Same keys, same TTLs, same evictions, run by somebody else in many cities. The two new facts are the origin behind it and the fleet in front, and the fleet is smaller than the map.

An edge cache's hit rate is your traffic's concentration divided by the number of edges. Marlow's covers got 62% with a one year expiry on files that never change, because the long tail of a 1.2 million title catalogue is asked for about once a month at any one edge. There was no dial, because neither term in that sentence is a setting. An origin shield puts the traffic back together before it reaches you, and turns one cold object into one origin request rather than one per edge.

Your TTL is a maximum, not a reservation. You are renting a shelf you share with every other customer of that CDN, and eviction is their decision.

Every cache key is a bet about what the response depends on. Bet too small and two hundred people meet somebody else's name. Bet too big and you build a cache with no hits and a bill. When the key gets awkward, fix the document instead: the split you do for your own cache is the split that lets you use somebody else's.

A purge is a message and a new name is not. A purge has to reach every node, never reaches a browser at all, and hits your origin with one miss per edge at the same instant. A versioned filename has nothing to deliver, which is why it can honestly carry a year.

An edge moves bytes, not decisions. It still pays for itself on requests it cannot cache, by ending three of lesson 004's four round trips near the customer. And it is replication lag you chose, so keep it away from anything a person's own write is meant to change.

Check your understanding

1. Your API's busiest endpoint gets 900 requests a second spread over about 300 distinct keys, and you are considering a two second edge TTL. Work out what the origin sees per edge afterwards, say what you need to know about the fleet before you can finish that calculation, and name the cost you are accepting.

2. An endpoint returns a list of titles that differs by country, and your application works out the country from a header the CDN adds to every request. Say what has to be in the cache key, what a customer sees if you forget, and what getting it right costs you in hit rate.

3. Marlow's founder wants to cache the /gifts category page from lesson 008 at the edge. Given what lesson 008 established about that page, say which Cache-Control directives you would set and why, and what you would expect to happen at the first expiry without stale-while-revalidate.

4. You run a purge-all at 11:00 on your busiest day because a stale price shipped to one product page. Describe what your origin experiences in the following minute, which earlier lesson already priced that shape, and two things you could have done that morning to make the purge unnecessary.

5. Galewatch's year long reports cover periods that have already ended and can never change. Say what TTL you would put on them, whether you would use a stable or a versioned URL, and which of today's two invalidation problems that choice makes irrelevant.

Next lesson

023 Consistency, Intuitively: Strong, Eventual and Everything Between. Today put a copy of your data in nine cities and called the delay a setting; next lesson gives you the vocabulary for what those copies are actually promising each other, and what it costs to ask for more.