Stagefront is an event ticketing service invented for this course, where a stadium show goes on sale at 10:00 and two hundred thousand people press the same button in the same minute. Lesson 048 left it in an embarrassing position: its configuration service had been dead for eight days, nobody had noticed, and six of its twenty application boxes had served an on-sale with their local cache switched off because a restart had handed each of them a compiled-in default of zero.
The platform team already had an answer, because they had been building it since September. Every service in the building would get a proxy of its own, running beside it on the same box, carrying its traffic in and out. One control plane would tell all those proxies what to do. No registry clients, no poll loops, no fail static rule written four times in four languages. The ticketing service went in on Friday 16 October.
On Tuesday 20 October, at 10:00, forty thousand seats went on sale.
By 10:00:20 the twenty boxes were at the 75% CPU lesson 006 measured for that minute, each taking its share of 3,333 arrivals a second, which is 166.65 apiece. Seat maps came back in single digit milliseconds. Nothing was wrong.
At 10:00:52 one box answered its health check slowly, because it was busy, and the process manager killed it. That had happened six times in the eight days lesson 048 described, because lesson 006's question about which one your process manager thinks it is doing had never been answered at Stagefront.
The box came back up at 10:01:00 and started two processes. The ticketing service was ready in about two seconds. Its proxy had no routes, no upstream addresses and no certificate, so it asked the control plane for all of them, at the busiest moment that control plane ever sees.
Lesson 006 wrote that fleet's check cadence down: every two seconds, three strikes. About six seconds. The proxy needed nine.
At 10:01:07 the same check killed the same box again, and the wait started over from zero.
Nineteen boxes now, at 175 arrivals each.
Two more crossed the same line at 10:01:40, two more forty seconds after that, and not one of the five ever came back on its own. By 10:02:30 fifteen boxes were carrying 222 arrivals a second apiece, checkout latency went through the two second bound lesson 003 set for the on-sale window, and the window was scored a fail. The show finished selling at about 10:11 rather than in the eighty seconds lesson 017 recorded.
At 10:06 an engineer on the call turned the liveness probe off. Every one of the five boxes was serving ninety seconds later, with correct configuration, on the first try.
Nothing in the ticketing service had failed. Five boxes were killed while waiting for permission to route.
Two shapes, and the two words underneath them
An API gateway is one proxy tier that every request from outside passes through before it reaches a handler. Lesson 029 already met it in passing, as whatever box terminates connections in front of your handlers, and Marlow's twenty dollar managed balancer from lesson 006 is one with a short feature list. The defining property is that it is shared: a few of them, in front, carrying everybody's traffic.
A sidecar is a proxy process that runs beside a single application process and carries that one process's traffic. Not in front, alongside. One per process, so as many of them as you have processes.
A service mesh is all of those sidecars plus the thing that configures them.
That last clause hides the split the whole industry organises itself around, and lesson 048 leaned on half of it from start to finish without ever stopping to define it. The data plane is the set of things that actually carry requests: the gateways, the sidecars, the balancer. The control plane is whatever tells them how to do it, by handing out routes, policies, certificates and the list of healthy addresses.
Read those two definitions against lesson 048 and you will find you have already studied this. The control plane is a registry and a configuration store. The data plane is the fleet of clients reading them. Every rule in 048 therefore applies here with nothing changed, and the hook above is what became of the one about fail static.
What is actually in the box, when a vendor sells you a mesh: routing, retries, timeouts, circuit breaking, rate limiting, outlier detection, health checking, encryption with a certificate at both ends, and a uniform set of metrics for every hop. Lesson 006 owns the routing, the health checks and the outlier detection, lesson 020 the retries and timeouts, lesson 042 the breaker, lesson 021 the limiter, lesson 026 the metrics, and lesson 048 the registry the routing reads. Exactly one item on that list is new to this course, and it is the second certificate, the one at the client's end. For everything else, the location is the entire product.
So why move them out? Lesson 048's closing paragraph is the honest answer and it is a good one. Every language and framework in the building needs its own registry client, its own poll loop, its own version check and its own fail static rule, and a proxy is written once, in C++ or Rust, by people who think about nothing else. Every language gets it by not having it. If you run four languages, that argument is strong enough on its own, and I think it is the only one here that is unambiguously good.
Most teams who install a mesh wanted three of its features and bought forty. The rest of this lesson is the price list.
A policy is priced by how many requests its decision needs to see
Lesson 042 handed something forward to this one. A circuit breaker inside the application divides its evidence by the size of the fleet, it said, and the fix is to move the state somewhere shared, which for a breaker means the gateway lesson 049 owns.
The answer turns out to be yes for a gateway and no for a sidecar, and the reason is worth the rest of the section.
Marlow Books, the four person online bookshop that also exists only in this course, takes one order every fifty seconds, which is 1.2 calls a minute to its payment provider. Lesson 042 counted the shop's eight application processes, four on each of its two boxes, and divided: one call per process every six minutes and forty seconds. Any percentage threshold over a sixty second window there is a coin toss, because almost every window holds either no calls or one. Meanwhile the balancer the shop has had since January judges each box on 1,200 responses a minute. Same three states, 8,000 times the sample.
Now install a sidecar. You get eight sidecars, because the shop has eight processes, and each one sees one call every six minutes and forty seconds.
The breaker left the application's code. Its evidence did not move at all.
A sidecar is a per-process thing wearing the word infrastructure. Lesson 006 found the same shape pointing the other way and said it better: a connection pool is a fleet wide number wearing a per box disguise, and every machine you add divides it again.
Lesson 021 owns the mirror image. A limiter inside the application gives every caller one full allowance per copy, which 021 wrote up as two boxes meaning twice the limit and twenty boxes meaning twenty times. It was counting boxes; with the counter inside the process and 042's eight processes, Marlow's real multiplier is eight. The conclusion is the same either way, and 021 drew it: a limiter that counts per caller needs one counter every box can see, which is why limiters live at the balancer or the edge rather than in your handler.
The mesh's own manual agrees, which is worth knowing before the sales meeting rather than after. Envoy's local rate limit filter is enforced inside each proxy with no calls to anything, so a limit of a hundred requests a second on a route gives you a hundred a second per proxy, and your fleet's real limit is that multiplied by however many proxies are running. To get one number for the whole mesh you deploy a separate rate limit service and have every proxy ask it, over gRPC, about every request. Which is 021's one counter every box can see, with a network hop in front of it.
| What the decision needs to see | Example | Where it can live |
|---|---|---|
| this request only | timeout, routing, mTLS | anywhere, sidecar included |
| many of your own calls | breaker, outlier detection | where the samples pool |
| every caller's calls | rate limit, quota | one counter all of them see |
As a sentence, since the audio skips tables: a decision about one request can be made anywhere, including in a proxy beside one process; a decision that needs a statistic has to sit where enough traffic passes to produce one; and a decision about a caller's total needs one counter shared by everything that counts.
Sort a policy by how many requests its decision needs to see, and you know whether a proxy beside your process can hold it. The first row moves for free. The second moves only if the proxy is shared by enough traffic to have an opinion, which a sidecar is not and a gateway often is. The third never moves anywhere. It only ever gets a new client.
Which settles Marlow, and the verdict is no. Eight processes, forty requests a second, and a balancer that already does outlier detection for twenty dollars a month on 1,200 responses a box a minute. A mesh would hand the founder eight more processes to restart on a Friday afternoon and not one new fact about the payment provider.
Whoever owns the clock owns the outage
Lesson 035 wrote the sentence this section needs, about something much smaller. A Postgres advisory lock is released when the connection closes, which for a box that loses power means whenever the server notices the socket is gone, and with default TCP keepalives that can be hours. Set tcp_keepalives_idle, 035 said, and look at the number you chose, because the clock did not disappear. It moved into the kernel, where nobody argues about it.
A mesh is that sentence one radius out. Your timeouts, your retry counts and your breaker thresholds leave your code and move into a configuration file in another team's repository, where nobody argues about them either.
Retries are where that costs the most. Lesson 020 established that retries multiply across layers, that three layers of three attempts is twenty seven times your baseline load, and that you offer that multiplier at the exact moment the far end can least take it. Its rule was to retry in exactly one layer, turn it off everywhere else, and cap the whole thing with a budget, which gives you 1.1 rather than a number between one and three that you do not control.
A mesh turns retries into a default that one person can switch on for everybody, in one line, with good intentions. Now 020's exactly one layer is a claim about two teams' configuration rather than about your code, and the layer nobody can see is the one that never comes up in review. Your client library retries three times, your sidecar retries three times, the gateway in front retries three times, and 020's twenty seven is live in production without any single commit having done anything wrong.
The tell is a dependency whose traffic during an incident is some multiple of your own request rate that nobody can account for. Lesson 020 put it more usefully: a dependency's load spike during an incident is as likely to be the consequence as the cause.
Timeouts go the same way and the sentence is 020's again. A timeout is a claim about somebody else's latency distribution, so a platform default timeout is a claim nobody made about a distribution nobody measured. The proxy is the right place to enforce a timeout and the wrong place to decide one, and that distinction is not an org chart detail: the people who should choose the number are the ones who can name the far end's p99 and say what their own request does when it fires. Keep the policy in the caller's repository even when the enforcement is in the mesh. And prefer 020's deadline to a duration, because a proxy on every hop is the one component present at every hop, which makes it the natural carrier of a deadline that shrinks as it travels.
Then there is the one policy that is strictly better in a proxy. Mutual TLS, or mTLS, is a TLS connection where both ends present a certificate, so each one knows who the other is rather than trusting the network the bytes arrived on. In a sidecar no application author can forget it, which is most of its value. Lesson 004 priced it: TLS 1.3 needs one round trip, lesson 002's round trip inside a data centre is half a millisecond, and connection reuse pays that once per connection instead of once per request. A proxy holding its upstream connections open makes mTLS nearly free. One opening a connection per request makes it the most expensive thing in the call. The certificates have to come from somewhere and be rotated, which is lesson 053, and what per-service identity does to your trust model is lesson 057.
The last cost in this section is the one that makes the news, and lesson 003 has already told you the story. On 2 July 2019 Cloudflare pushed a new managed rule to its web application firewall, to every machine, everywhere, at once. The regular expression in it backtracked catastrophically, CPU went to 100% across the network, and traffic through Cloudflare fell by roughly eighty percent for about half an hour until an engineer used the global kill switch to turn the firewall off. In 003 it is evidence for the shared cause is the ceiling, since redundancy removes only the failures that are not shared, and every machine was redundant while the deploy was not. What belongs to today is why the rule arrived that way at all. Managed firewall rules went out in one step on purpose, skipping most of the graduated rollout the company used for other changes, because the whole value of a managed rule is that it reaches everybody before the threat does.
Hold that against 003's description of the failure shape: not two disks dying on the same Tuesday, but one change arriving everywhere in the same second. Today is the lesson that can name the component whose entire product is that sentence. A control plane's selling point and its blast radius are the same mechanism, and the data plane did exactly what it was told, instantly, everywhere.
A configuration push is a deploy, and it usually has none of a deploy's manners: no canary, no percentage, no bake time, and frequently no review. Lesson 054 owns deploys and this is the sentence to carry into it. The global kill switch that ended that half hour is lesson 027's kill switch at network scale, built for 027's reason: Marlow's switch lives in a file on each box because a switch has to work on the day the thing it switches off is the problem.
The sidecar has to be born
Lesson 048's result was that fail static buys you the fleet you have and nothing you are about to start. A process holding the value is unaffected for as long as it lives; the process that starts during the outage never had the fact. The six week old process is fine and the thirty second old one is the whole failure surface.
A mesh doubles the number of processes that have to be born on every box, and the new one is the one with nothing compiled in. That is on purpose. Its entire configuration arrives over the wire, which is the point of buying it, and lesson 048's compiled-in default of zero is exactly the thing everybody agreed to stop doing.
So here is the shape of the title. A gateway is a hop in the request path. A sidecar is a step in the startup path.
A gateway is a component in lesson 003's chain, and lesson 029 already priced one extra hop there: an identity service as good as Stagefront's own application tier took the purchase path from 19.3 hours of expected downtime a year to 23.6. A sidecar is a different animal. It shares a box and a lifecycle with the process it serves, so it does not multiply availability the way a separate tier does. What it adds instead is a new way for the pair of them to fail to start.
Envoy writes the mechanism down. A proxy subscribing to its configuration has an initial_fetch_timeout, fifteen seconds by default, and when that expires the proxy moves on to the next phase of initialization whether or not the configuration ever arrived. Moving on with nothing means a listener with nowhere to send anything. Until then it reports itself not ready, with a 503 on the endpoint whose job is to answer that question, and a liveness probe pointed at that endpoint will restart a proxy for the crime of still waiting, which resets the clock it was waiting on.
Which brings lesson 006's two questions back for the third time in this course, with a state they do not cover. Liveness asks whether a process is broken beyond recovery so that a kill is the only thing left. Readiness asks whether this box should be sent traffic right now. A proxy waiting for its control plane is neither broken nor merely busy. It is unborn, and the single action guaranteed to make it worse is the one a liveness check takes.
A kill is only ever right for a process that will not get better by waiting.
Now the arithmetic of the collapse, which I do not have to do, because lesson 006 published it in a code block about this exact fleet.
20 boxes at 75% remove one
19 boxes at 78.9% the next one tips
18 boxes at 83.3%
17 boxes at 88.2%
16 boxes at 93.8%
15 boxes at 100%
Those percentages treat CPU as proportional to arrival rate, which is the pessimistic end, since some of that 75% is work a box does whether or not anybody is buying. In arrivals it is 166.65 a second each at twenty boxes, 175.4 at nineteen and 222.2 at fifteen, off the 3,333 lesson 048 counted. Five boxes to the wall. If a quarter of that 75% is baseline rather than per request, fourteen boxes come to 99.1% and the wall waits for the sixth, which is also the honest answer to why Tuesday stopped where it did. Nothing stopped it. Fifteen boxes in that model are slow enough to go on failing checks and not quite pinned, so the sixth was somewhere in the next few minutes when the engineer turned the probe off at 10:06. A ratchet does not have a resting place. It has however long you take to notice.
Five boxes is also what 006 got, and the sentence it wrote under the table is the one worth rereading: thirty seconds, five boxes gone, and not one of them was ever broken. The panic threshold does not help, because fifteen healthy of twenty is comfortably above half. It stops the wipeout, not the spiral.
So Tuesday added one word to a failure this course has had on the shelf since lesson 006. In 006 those five boxes were ejected: taken out of the pool by a balancer, still running, still holding their configuration, and back in service the moment they answered a check inside a second. The spiral had an exit, and 006's two fixes, a check served off its own thread and a timeout set against p99 under load rather than p50 at three in the morning, were cheap in the ordinary sense of being easy to type. Tuesday's five were killed, and a killed box has to be born again, which now means two processes and a conversation with a control plane that is busy. Same table. Nothing at the bottom of it.
That is the whole of what a mesh changed about a failure mode much older than itself, and it matters more than anything on the feature list: it turned a transient into a ratchet.
Three things fix Tuesday, and the first one is free.
Point liveness at the process and readiness at the configuration, and set the liveness threshold above the proxy's own startup budget. Lesson 006 asked its readers to go and check which one their process manager thought it was doing, and Stagefront still has not. Run the morning through it, and let the first kill stand even though pointing liveness at the process should have prevented that one too, because the result is stronger without the rescue: the box is killed at 10:00:52 for being busy, is back at 10:01:00, spends nine seconds collecting routes and a certificate, is not killed while it does so, and is serving by about 10:01:12. Nineteen boxes for twenty seconds at 78.9% CPU, which nobody would ever find on a graph. The entire incident lived in the gap between a probe's six seconds and a proxy's nine.
Then ask which of your two processes has something to be static about. Lesson 048's cheap half was to write the last good value to disk, so a process starting during a control plane outage has something to be static about at all. That fix covers the application and says nothing about the proxy, and the proxy is now the thing that must be born. Whether yours can start from a last known good configuration, and what exactly it does when that fifteen second timer expires with nothing in hand, are two questions with real answers. I would rather know those two numbers than anything in the vendor's benchmark.
And do not let a Tuesday on-sale be the first time a box is born. There is an ordering problem that arrives before any of the above: the application process starts, makes its first outbound call, and its proxy is not ready yet, so the call fails in a way that looks exactly like the far end being down. Meshes have a switch to hold the application until its proxy is up. It is the kind of switch you find out about on a Tuesday.
None of which argues against what Stagefront bought. The compiled-in zero is gone, the local TTL now reaches every box in every language, and nobody writes a poll loop any more. That is real, and it is what they paid for. The bill was that a configuration problem stopped being a wrong value and became a missing box, and those two arrive at different times of day.
Recap
A mesh is relocation, not invention. Routing, retries, timeouts, breakers, rate limits, health checks and per-hop metrics are lessons 006, 020, 021, 042, 026 and 048 moved out of your code and into proxies, and mTLS is the one thing on the list the course had not already taught you. The data plane carries requests, the control plane tells it how, and that pair is lesson 048's registry and configuration store wearing the names the industry actually uses. The one unambiguously good reason to do it is that a proxy is written once and every language in the building gets it by not having it.
Sort a policy by how many requests its decision needs to see. A decision about this request can live anywhere. A decision that needs a statistic has to live where enough traffic passes to produce one. A decision about a caller's total needs one counter everything can see, and never moves. So a sidecar is a per-process thing wearing the word infrastructure: Marlow's eight processes become eight sidecars holding eight breakers that each see one call every six minutes and forty seconds, and Envoy's local rate limit of a hundred a second is a hundred a second per proxy. You moved the code, not the counter.
Whoever owns the clock owns the outage. Lesson 035 watched an expiry move into the kernel where nobody argues about it, and a mesh moves your timeouts and retry counts into another team's configuration file. 020's twenty seven times goes live without any one commit doing anything wrong. Enforce in the proxy, decide in the caller's repository, and pass a deadline rather than a duration.
A configuration push is a deploy. One change reaching every proxy in seconds is the product and the blast radius both, which is 003's Cloudflare regex of July 2019 read from the component's end rather than from redundancy's. Stage it like code, and keep the off switch off the path it disables.
A gateway is a hop in the request path and a sidecar is a step in the startup path. A gateway joins 003's chain, which 029 priced at four extra hours a year for one hop. A sidecar shares its neighbour's fate instead, and adds a birth: a second process per box whose whole configuration arrives over the wire, with a fifteen second default timer and nothing to fall back on. Fail static has to be arranged twice now, and a kill is only ever right for a process that will not get better by waiting, because restarting a wait restarts the wait. Five boxes took Stagefront's fleet to the wall, and the spiral 006 warned about had lost its exit.
Check your understanding
Your platform team offers to move your service's retries and timeouts into the mesh and delete them from your code. Say what you would want in writing first, which of lesson 020's rules gets harder to verify afterwards, and the one measurement you would add before agreeing.
A colleague proposes using the mesh's rate limiting to cap a noisy internal caller at 50 requests a second. Your service runs 24 pods. Work out what that limit actually enforces, then give two designs that deliver the number asked for and say what each costs.
Your control plane is unreachable for forty minutes. No request fails and no alert fires. Write down which processes in your system are now carrying a risk, what single event converts that risk into an outage, and the cheapest instrument that would tell you how exposed you are.
Stagefront wants mTLS between its services and the security review asks for the latency cost. Produce the number using lessons 002 and 004, state the assumption it depends on, and name the condition under which your answer is wrong by two orders of magnitude.
A box's proxy is answering its readiness endpoint with a 503 and has been for four minutes. Give two causes that look identical from outside, the question you would ask to tell them apart, and say which one a restart helps.
Marlow's founder reads about meshes and asks whether the shop should run one. Answer with arithmetic rather than taste, then name the one feature on the list the shop genuinely lacks and the cheapest way to get only that.
Next lesson
050 Multi-Region: Active-Passive, Active-Active and the Speed of Light. Every control plane today sat close enough that a proxy could reach it in a millisecond; next lesson puts an ocean in the middle and finds out which of today's answers were really about distance.