Deployment Strategies: Blue-Green, Canary and Feature Flags

Marlow Books is a four person online bookshop that exists only in this course, and on Friday 9 October its founder shipped a change and, for the first time, stopped halfway through to look at it.

The change was small and overdue. The shop had always kept a delivery address in one free text column, which is fine until you want to print a label that fits or sort a morning's parcels by postcode. So he split it: name, two lines, town, postcode. New columns, a backfill over the old orders, the checkout writing the new fields, the admin pages and the label printer reading them. One release, the way this shop has always shipped.

He rolled it out the way lesson 006 told him to in January, and for the first time one box at a time. Mark box B unready, wait six seconds for the balancer to actually stop choosing it, then stop the process. Box B was serving the new code at 16:42, nothing was dropped, and the shop stayed open throughout, which had never happened before.

Then he did the thing he had read about and never had the nerve to try. He left box A alone, and watched box B.

Ten minutes. Orders landing, no errors, checkout latency where it always sits. Lesson 026 built those graphs and every one of them was green. At 16:52 he deployed box A, both boxes were new by 16:54, and he went home.

On Monday morning the person who packs the parcels found seven labels with no address on them.

Box A had spent that whole window doing what it had done for months, which is write the delivery address into the old column, because box A had the old code and the old code was correct. The backfill had already run, so nothing was coming back for those rows. And after 16:54 nothing in the shop ever read that column again.

So the damaged orders were the ones taken by the box he had not touched, and they were damaged by the box he had. Every graph he watched for ten minutes was pointed at the new code. The failure was in the control group.

The window runs from the backfill finishing at about 16:40 to box A being drained at 16:52. Box A served all of the first two of those twelve minutes, while box B was restarting, and about half of the other ten. At lesson 027's rate of one order every fifty seconds, carried from lesson 009's Christmas conversion onto an ordinary afternoon and never checked by anybody at the shop, that is about eight orders. He picks that hour precisely because the orders go quiet, so seven landing in the packing room is what you would expect.

The repair was one UPDATE, run before the kettle boiled, because the old column was still sitting there holding every address. Nothing was lost. What the morning actually cost is the thing worth a lesson: he had done the careful version, and the careful version is what made the window twelve minutes wide instead of ninety seconds.

Four handles, and a word nobody writes down

A rolling deploy replaces the fleet a piece at a time: take a box out, upgrade it, put it back, repeat. Marlow's Friday is one, with a fleet of two.

Blue-green runs two complete environments. Blue is live, green gets the new version, something in front moves traffic from one to the other, and the old environment stays up and idle for as long as your nerve holds. One switch, one moment, and a rollback is the same switch the other way.

A canary is a small share of real traffic sent to the new version, with a rule written down in advance for what sends it back. The name comes from the bird carried down a coal mine to be affected before the miners were, which tells you what the strategy thinks of the share it picks.

A feature flag is a branch in running code that a configuration value decides. Ship the code to everybody with the new path switched off, then switch it on for one percent of customers, or for your own account, or for the country you can afford to be wrong in.

And bake time is how long you leave the new version live before you believe it. It is the fifth word here, and the one nobody writes down.

Two of those are already running in this course under other names. Galewatch, the service that takes a reading every two seconds from each wind turbine on the farms it monitors, pushes configuration to its four ingest boxes one box at a time, twelve minutes end to end, and lesson 031 is the morning that cost it. Nobody there calls that a rolling deploy, which is what it is. Marlow has a feature flag too, and calls it a kill switch. Lesson 027 built it after a September afternoon when turning checkout off needed a deploy: two fields in the small uncached request every page already makes, read from a file on each box, changeable in thirty seconds over ssh.

Here is what the four have in common. Each one is a bet that you will notice something while there is still something cheap to do about it.

Read the last column as a sentence, since the spoken version skips the table: rolling back the way you came, one switch in seconds, routing the share home, a configuration change with no deploy at all.

None of that matters if nothing tells you to pull the handle.

So the first question is never blue-green against canary. It is what signal would turn this around, how long that signal takes to appear, and whether it is pointed at the half of the fleet that can break. Lesson 026 argued that an alert comes from a tolerance rather than from a number somebody liked. The rollout version is sharper, because a deploy has a deadline: the instrument has to produce its answer before the window shuts.

A canary is a statistical instrument, and most canaries have no sample

Unless something in front is splitting individual requests, the smallest canary you can run is one box. Marlow has two application boxes, so the shop's finest possible canary is fifty percent of its traffic, which nobody would call a canary. Stagefront, the ticketing service where a stadium show goes on sale at 10:00 and two hundred thousand people press the same button, runs twenty boxes during a window, so its floor is five percent. If you want one percent you need a hundred boxes, or something that routes by request, which is the proxy tier lesson 049 took apart and one more input to the mesh question it put to you.

A canary answers one question: is the error rate on the new version worse than on the old one? That is a comparison of two proportions, and a proportion needs events. Marlow's canary on box B sees one order every hundred seconds. Suppose the regression breaks one checkout in twenty. In ten minutes of watching, the canary sees six orders, and the chance that at least one of them fails is one minus 0.95 to the sixth, which is about twenty six percent.

Three times in four, the founder's ten minutes of watching show him a clean canary over a change that is breaking one order in twenty.

Turn it round and ask how long he would have to watch to be ninety percent sure of seeing one failure. You need 0.95 to the n below 0.1, so n is forty five orders. At one order every hundred seconds that is seventy five minutes. An hour and a quarter of a Friday evening, for ninety percent confidence in a regression big enough to be a scandal.

Stagefront does the same arithmetic in a different universe. Lesson 004 put the peak of its purchase path at five hundred payment calls a second, which is how lesson 017 got a forty thousand seat stadium selling out in eighty seconds. Against Marlow's one order every fifty seconds that is twenty five thousand to one, and the two shops are running the same strategy out of the same handbook. Five percent of five hundred a second is twenty five purchases a second, so Stagefront's canary clears forty five events in under two seconds, and even a one percent canary gets there in nine and still beats Marlow's hour and a quarter by a factor of five hundred.

Now the catch. That five hundred a second is a peak, measured inside the on-sale window. Stagefront is otherwise quiet, so its canary is only an instrument for twenty minutes, forty times a year, which lesson 050 counted as 800 minutes out of 525,600. The one window in which a Stagefront canary can learn anything is the one window nobody will authorise you to deploy into, and lesson 050 was right to say nobody should.

Lesson 042 asked this same question of the shop's circuit breaker and got a coin toss. A canary is that breaker with a human holding it. A canary at one percent, on a metric whose ordinary noise is five percent, is a deploy with a ceremony attached.

What should a shop like Marlow do instead? Not a rate. At thirty six orders an hour on one box you cannot measure a rate before the evening is over, so instrument the events whose honest baseline is zero. A NULL arriving where the code requires a value. An exception type that has never been thrown in production. One occurrence of either is a signal, because one is infinitely more than none. The other instrument that works at this volume is a synthetic order placed by a script once each box is up, which tests the path you thought of and nothing else, and at Marlow it has to be sent to both boxes or it checks the half the bug was never in.

The quiet hour is the cheapest window to deploy in and the one that teaches you least. Both are true at once, and most teams budget only for the first.

Bake time is a count of events, not a length of time

Lesson 049 put Stagefront's ticketing service behind its new sidecar proxy on Friday 16 October. The on-sale was Tuesday 20 October, and five boxes were killed inside the first two and a half minutes of it.

Four days of bake time. Here is what those four days contained: a weekend, a Monday, and not one on-sale. Stagefront is scored between 09:55 and 10:15 on forty on-sales a year, which lesson 050 priced as 800 minutes of the year that matter out of 525,600. Forty on-sales spread over 365 days means a four day window holds 0.44 of one, on average. The usual outcome of a four day bake at Stagefront is that nothing the company exists for happens inside it.

So the mesh baked for four days, met the only load that could break it on day five, and broke. Nothing about that is bad luck. The bake measured the wrong thing: elapsed time, which was never the risk.

Bake time is the number of the events you are afraid of that happened while the new thing was live. For Stagefront that is on-sales, and unless somebody schedules it the answer is zero. Marlow's arrives once a year and is called Christmas.

And some readers never arrive inside any window you would tolerate. Lesson 052's publisher payout job runs at two in the morning on the first Sunday of the month, which is a gap of 28 or 35 days, and it found a migration's damage 26 days after the migration. No bake time anybody would sit through covers a reader that is asleep, so for those you wait for the slowest one on purpose rather than hoping a window stumbles into it.

What makes a rollback a rollback

Lesson 052 built the machinery this section needs and spent it on a different question.

A change is backward compatible when the new code can read data the old code wrote. It is forward compatible when the old code can read data the new code writes. Both are said from the new code's point of view, and lesson 052 found that the two commonest changes anybody makes, adding a field and removing one, are backward easily and forward only if your old readers are tolerant of what they do not recognise.

Now put each strategy against that pair.

A rolling deploy and a canary both run the two versions at once, so they need forward compatibility to work at all. The old boxes are still reading, and now there is new data in front of them. Blue-green does not need it to run, because the switch is a single moment. It needs exactly the same property to roll back, because a rollback puts old code in front of everything the new code wrote while it was live.

One property, two different days. And the commonest change you will ever ship does not have it.

Marlow's Friday broke in the other direction, and that is the one a backfill hides. The new readers coped with every old row, because the backfill had just rewritten them all, so the release looked backward compatible and for one instant it was. What a backfill cannot reach is the rows an old writer is still producing behind it. That gap is a property of the window rather than of the schema, so it closes on its own the moment the last old box goes, and nothing the founder watched on box B for ten minutes was ever going to show it to him.

A canary is not a safer deploy. It is a longer one, and what it buys in detection it pays for in the time two versions of your code have to agree with each other. Marlow's careful Friday is the whole argument. Run that rollout straight through and box A keeps writing the old shape only while box B restarts, about two minutes, which is two orders. The ten minutes of watching turned two orders into eight.

Lesson 031 is the same shape with a worse ending. Galewatch's shard constant went from 2 to 3, the config rolled out one ingest box at a time, and for twelve minutes four ingest boxes disagreed about which machine a turbine's readings belonged on. The rollout was the bug.

Its remedy is worth copying, and what makes it work is where the check sits. Lesson 031 puts a constraint on each shard saying a row's serial has to hash to that shard, and gives the three databases the new divisor before the four ingest boxes get it. A box that has not been told yet collects an error from Postgres instead of writing a clean row in the wrong place, and the receiving end works ownership out for itself rather than trusting what the writer computed, because a stale writer is wrong about exactly the thing you would be asking it. Widen the side that accepts, then move the side that produces, and the mixed window fails loudly rather than quietly.

And now the thing nobody at Marlow had noticed they were buying.

Lesson 006 showed in January how two boxes and a balancer turn lesson 003's 2.6 hours of deploy downtime into nothing, one box at a time. Then lesson 027 prices a deploy at twelve minutes to three at about ninety seconds of downtime, and lesson 052 prices a migration at twenty past eleven at the same ninety seconds, both of them in September, because the script still stops both boxes together. The shop bought the balancer and never rewrote the deploy, which is eight months of a saving sitting on a shelf.

Here is the part that is not just sloppiness. For as long as every deploy took the site down, exactly one version of the code was ever running. The ninety seconds were buying a guarantee, and it was the guarantee this whole lesson is about. A deploy that costs no downtime is a deploy during which two versions of your code are both live. That is the trade, it is never written on the invoice, and the Friday the founder collected his saving is the Friday the shop acquired a class of bug it had not been able to have.

One step still has no handle at all. The balancer can move traffic between two application boxes. It cannot move Postgres, which has been one machine on box A since lesson 005 and, with lesson 011's replica behind it, is still the only machine the shop can write to. A deploy that restarts an application is reversible. A deploy that rewrites a 240 megabyte table and all its indexes is not, and no rollout strategy here touches it.

So the unit that decides your exposure is not the deploy. It is the last point you could still get back to, and a schema migration, a queue drain and an outbound email each move that point past the handle you were counting on.

Lesson 050's failover drill passed at two in the morning and measured the half nobody was worried about. A rollback has that disease and worse, because it lives in a runbook rather than in a schedule. A rollback that has never been run is a paragraph, not a plan. The cheapest fix is to roll one deploy back on purpose, at a quiet hour, with nothing wrong, and time it.

The flag is the only handle that works without a deploy, which is also why it outlives its reason

Lesson 027 is the one that matters: a switch that needs a deploy is not a switch. The person who notices first at Marlow is in the packing room with a tape gun, and nobody holding a tape gun is going to ship a release.

That is why flags win, and the win is real; lesson 048 showed how the value reaches every box.

It also carries 027's caveat, which gets quoted less often. Marlow's switch lives in a file on each box, so two boxes can disagree, and 027 was explicit that this is fine for a banner and a disabled button and would not be fine for anything that decides. A kill switch is allowed to be inconsistent in a way a stock count is not. The moment a flag decides something, by routing a payment or choosing which column to write, that tolerance has quietly expired and you own a distributed configuration problem instead of a file.

And flags rot. A flag is lesson 052's expand and contract with nobody to run the fifth step, which is precisely what lesson 053 found about rotating a credential: the first four steps make your exposure strictly worse and the last one is the only one that takes anything away, and almost nobody runs it. Three lessons, one shape. The contract step is the one that never gets done, because it is the only step with no new capability in it.

My opinion, held harder than most things in this course: a flag gets a removal date in the commit that introduces it, or you have written a permanent branch and should name it as one.

The deploy nobody calls a deploy

Lesson 049 left one sentence for today. A configuration push is a deploy, and it usually has none of a deploy's manners: no canary, no percentage, no bake time, frequently no review.

Look at what a configuration service quietly takes away. Galewatch pushes a value to its four ingest boxes one at a time, twelve minutes end to end, which is a staged rollout whether or not anybody there calls it one. Move that value into a store every process polls every five seconds, which is lesson 048's design and better in nearly every other way, and lesson 048 says what happens: twelve minutes becomes five seconds for the whole fleet. You have deleted the only control that rollout had, and nobody books it as a loss, because what you deleted was a delay. Nobody writes a rollout plan for a value.

Lesson 048 had the other half of this. Fail static buys you the fleet you have and nothing you are about to start, and a configuration outage is collected one restart at a time, by deploys. So a deploy is the event that turns a quiet control plane outage into a real one, and the first box you restart is the cheapest probe you will ever have for whether that control plane is healthy. One box, upgraded, watched for a minute, is a rolling deploy doing a job nobody asked it to do.

Almost nobody I have asked can tell me what their canary would have caught. The strategy is in the pipeline, the percentages are in a YAML file, and which signal would have moved has never been written down anywhere, which makes the rollout a shape rather than a control.

Which brings the Friday afternoon back round.

Lesson 003 did the arithmetic on Marlow's deploys when they still took the site down: 104 deploys a year at ninety seconds each, 2.6 hours, six percent of the shop's annual availability budget. It argued the real prize was the batching rather than the hours, because a founder who shipped twice a week shipped half a week of work at a time, and the fix was to make a deploy cheap and ship a fifth of the risk five times as often.

It was a good prediction and it has not happened yet. January made the saving available and nobody collected it, and in September the founder shipped the checkout, the price import, the admin form, the page template and the JSON response in one release, which is lesson 052's migration and five changes in one batch, and the Sunday that caught it was twenty six days later.

That is not a story about discipline. The batching was never only about the downtime. A deploy at Marlow is cheap to perform and still expensive to believe, because thirty six orders an hour on a box cannot tell you anything in ten minutes, and if the only confirmation available is Monday in the packing room then you may as well put half a week of work behind one Monday. Make the deploy observable and the batches come apart on their own. Leave it unobservable and no pipeline you buy will change what the founder does on a Friday.

Recap

Every rollout strategy is a claim about what you would notice. The four differ in what they cost and in how fast the handle works, and all four are worthless if no instrument answers before the window shuts. So ask what signal turns this around, how long it takes to appear, and whether it watches the half of the fleet that can break. Marlow's Friday had a green canary over a broken release because every graph was pointed at the new code and the damage was in the control group.

A canary is a statistical instrument and most canaries have no sample. Your finest share is one box unless something routes requests, which is fifty percent at Marlow and five percent at Stagefront. A one in twenty regression takes seventy five minutes to reach ninety percent confidence on Marlow's canary and under two seconds on Stagefront's, because twenty five thousand to one is the gap between those two shops. At low volume stop measuring rates and instrument the events whose baseline is zero, such as an exception type never seen in production. And the quiet hour is both the cheapest window to deploy in and the one that teaches you least.

Bake time is a count of events, not a length of time. Stagefront's mesh baked for four days containing no on-sale, which is the normal outcome when forty on-sales are spread over 365 days, and then broke at the first one. Count the events you are afraid of, not the hours.

A canary is not a safer deploy, it is a longer one. Rolling and canary need lesson 052's forward compatibility to run at all, since the old boxes are still reading; blue-green needs the same property to roll back. The commonest changes anybody ships, adding a field and removing one, do not have it. Marlow turned a two minute mixed window into twelve by being careful, which cost it six orders, and Galewatch's twelve minutes of disagreeing shard maps is the same window with one farm's readings going to two machines at once.

A deploy that costs no downtime is a deploy during which two versions of your code are both live, and for eight months Marlow's ninety seconds were buying the opposite without anybody writing it down. The handle also stops at the first irreversible step. A restart is reversible and a table rewrite is not, because the balancer can move traffic between two application boxes and cannot move the one Postgres on box A. And a rollback that has never been run is a paragraph, not a plan, so run one on a quiet Tuesday and time it.

A configuration push is a deploy with none of a deploy's manners. It is usually the fastest rollout and the widest blast radius you own, and it skips every control because it does not feel like shipping. A flag is the only handle that works without a deploy, which is both why it wins and why it outlives its reason: it is expand and contract with nobody to run the fifth step, exactly as lesson 053 found for a credential.

Check your understanding

1. Marlow's founder, having had that Monday, proposes deploying box B and waiting an hour before box A, so the canary has a real sample. Say what that buys him using this lesson's arithmetic, what it costs that the ten minute version did not, and the one change to the release that would make the hour safe.

2. Your service runs six boxes and the pipeline offers a canary at one box for ten minutes. Your p99 latency varies by about eight percent hour to hour, and the regression you are most afraid of would add fifteen percent. Say whether that canary can see it, what you would measure instead, and what you would tell the team that built the pipeline.

3. Lesson 052's expand and contract gives the order for a change like this one, so take the order as given: a release adds a required field to a row the next release will read, and removes a column nothing has read for a year. Say which of the two changes decides whether you may roll the fleet gradually, what happens if somebody rolls back twenty minutes after the last box, and how long you would keep the old environment warm.

4. Galewatch wants to ship new ingest code to its four boxes with a canary, rather than the twelve minute roll it uses for config. All four boxes read the same shard map and write rows according to it. Say what signal you would put on the canary box, how long you would hold it, and what lesson 031's constraint on each shard does to your answer.

5. A colleague wants to put the new checkout behind a feature flag at one percent of customers, and the flag is read from the configuration service. Say what the flag gives you that the canary in question 2 does not, which of lesson 027's three demands it fails, and what you want in place before the percentage goes above zero.

6. Your pipeline does blue-green with an automatic rollback if the error rate exceeds one percent in the first five minutes. A release ships that corrupts one row in two hundred, with no error anywhere. Say what the automatic rollback does, what the bake time was actually measuring, and the cheapest instrument that would have caught this one.

7. You are asked to cut deploy frequency from daily to weekly after a bad release, by a manager who is not wrong about the bad release. Argue the other side with lesson 003's figures, and name the one thing you would have to deliver first for your argument to be honest.

Next lesson

055 Capacity Planning and Autoscaling. Today's canary needs boxes to put traffic on and lesson 048's cold start arrives one restart at a time, so lesson 055 is where the fleet starts changing size on its own, with every deploy's mixed window inside it.