Sagas and Distributed Transactions

Marlow Books is an online bookshop run by four people, and it exists only in this course. On a Thursday in May, three weeks after the replay lesson 038 opens with, it sold a novel it did not have, refunded the money, apologised, and ordered another copy. Not one line of its code did anything it was not written to do.

The shelf said one copy. Here is the evening, from the logs, and it is the least dramatic timeline in this course.

At 20:41:06 a customer the order table calls Arjun pressed buy. One transaction, the shape lesson 035 had the founder ship before February's train: insert the pending order row carrying lesson 019's idempotency key, and update books set stock = stock - 1 where isbn = ... and stock > 0. One copy became zero copies. Committed in under two milliseconds.

Then the card, with no transaction open, for the reason lesson 015 spent a lesson on. The provider answered in about the three hundred milliseconds lesson 015 published, and a second short transaction moved the row to paid and committed at 20:41:06.4.

The checkout published order_placed, the message lesson 038 renamed on the first Tuesday in April. Lesson 036 measured the email consumer at a 240 millisecond median, so the email provider had the message before the page had finished reloading. It said the order was confirmed, because that is what the template has said since December.

reorder_watch, one of April's two twelve line consumers, read the stock for that ISBN, got zero, and emailed the two buyers from lesson 010, because zero is under three. One of those buyers filtered April's flood into a folder they do not read. The other still reads them. At 21:10 they added the title to the distributor order that goes out overnight.

Friday morning, packing. The slot was empty.

Nobody knows when it emptied. There was no bug to find: a number in a row had been one too high for that title for some unknown stretch of time, and the only instrument the shop owns for that error is somebody walking to a shelf. This is the third time in this course the catalogue has promised a copy the shelf did not have, after lesson 009's April and the February that lesson 035 opens with, and the first time nothing anywhere was broken.

So the founder went backwards through the evening, and found that most of it does not go backwards.

The paid row became cancelled, which is one statement. The money needed a refund, which is not an undo: it is a second payment in the other direction, through the same provider, and the customer's bank decides which day it lands. The confirmation email cannot be unsent, so a human wrote a second email contradicting the first. And the stock count wanted the inverse of the forward step, stock + 1, which would have put a copy that does not exist back on the shelf. That is exactly the phantom lesson 035 spent a Friday on, arriving this time from the front door. The founder set the row to zero by hand.

The distributor order stayed. It was triggered by a step in a flow that failed, and it is the only thing that evening produced which the shop actually needed.

Five steps. Two have an undo that is one statement, one has an undo that is a second transaction with somebody else, and two have no undo at all.

A rollback is a property of a log you own

Lesson 015 named what a transaction gives you. Atomicity is the one that feels like magic: write four rows, crash before the commit, and the database will have none of them when it comes back.

Worth knowing how. Postgres wrote those rows into its write ahead log, the same log lesson 011 ships to the replica, and each row carries the id of the transaction that wrote it. ROLLBACK does not go and erase anything. It writes one more entry, in a small structure Postgres calls the commit log, saying that transaction is aborted, and every later reader checks that entry before believing a row. The undo is a new fact appended to a log, and it works because one machine owns that log and every reader of those rows has to come through it.

Now put two logs in the room. Marlow's Postgres owns one. The payment provider owns the second, and the shop can write to it only by asking. There is no entry Marlow can append to the provider's ledger that makes the charge never have happened, because the provider's own readers have already read it: their reconciliation, their fraud model, the customer's bank. The best available is a new entry in that log, in the other direction, with its own id, its own fee, its own clock.

Same for the inbox. Same, in the end, for a shelf.

There is no rollback across machines, only a second operation that apologises. The word for that second operation is a compensation: a real forward operation whose business effect undoes as much of an earlier one as the world allows.

The textbook alternative is two-phase commit, which lesson 013 introduced in one clause. A coordinator asks every participant to prepare, meaning write the change durably but commit nothing and hold whatever locks it needs; once all of them have said yes, the coordinator records the decision and tells them to commit. It is real, in daily use between databases and brokers inside one organisation, and it gives genuine atomicity across them.

It asks for two things Marlow cannot supply. The first is a participant that can hold a prepared change open until the coordinator comes back. Cards look like they have that verb, and it is worth being exact about why they do not: an authorisation really does hold the money, and a capture really does complete it later. What is missing is who owns the clock. The hold expires on the issuer's schedule rather than your coordinator's, nothing lets the provider sit and block on a decision that never arrives, and an authorisation is a promise about funds rather than a promise that the capture will succeed. Authorise and capture are two local transactions with a void for a compensation, which makes them today's pattern wearing two-phase commit's clothes.

The second thing is harder, and lesson 015 refused it: every lock stays held for the slowest participant's round trip. That lesson priced the refusal at Stagefront, this course's fictional ticketing service, where a stadium show goes on sale at exactly ten in the morning and two hundred thousand people press the same button in the same minute. The figures are lesson 004's: five hundred payment calls a second at a hundred and fifty milliseconds each is seventy five of them in flight, so charging inside the transaction means seventy five open transactions holding seat locks and waiting on somebody else's data centre, in the minute of the year the business exists for.

The in doubt row is the famous one. If the coordinator dies after the prepares and before the decision, every participant sits there holding locks and is not allowed to guess. Lesson 033's consensus is the honest fix: put the coordinator's decision log on three machines and the window stops being unbounded. That removes the outage and not the cost. The locks were always the price.

Marlow has had a saga since January and nobody called it one

A saga is a sequence of local transactions, each committing on its own machine, with a compensation written for each step that might need taking back. Nothing holds anything between steps. The flow's progress lives in data.

step                    commits where       undo
1 hold copy, pending    Marlow's Postgres   stock back, row abandoned
2 charge the card       provider's ledger   refund, days later
3 row to paid           Marlow's Postgres   row to cancelled
4 send the email        somebody's inbox    none
5 pick off the shelf    the back room       none

In prose, because the shape matters more than the layout: two steps commit inside the shop and are cheap to take back, one commits in a ledger the shop does not own so taking it back is a second movement of money on somebody else's schedule, and two commit in the physical world, where nothing takes anything back.

Every saga needs something that notices a half-finished one, and at Marlow that something already exists. Lesson 019's sweeper runs every two minutes over orders pending for longer than two minutes, asks the provider what happened to each key, and either completes the order or releases the copy and refunds. A row has to age two minutes and then wait for the next run, so close to four minutes can pass before the job reaches it, and lesson 027 then watched a dozen provider questions at ten seconds each fill the sweeper's whole schedule. That sweeper is the shop's saga coordinator. It is a scheduled job in lesson 018's table with a deadline written beside it, and a lock that lesson 035 showed was the wrong kind.

Which brings the uncomfortable part of compensations. They are work, so they fail, so they get retried, and lesson 036 applies to them exactly as it applies to the forward path. At Marlow the second attempt does not even arrive as one of 036's redeliveries. It arrives as the next sweep, two minutes later, which is nobody's idea of a rare event. A compensation has to be idempotent.

That is harder than it sounds, and this shop paid for the lesson in February. update books set stock = stock + 1 is commutative, so releasing two different orders in either sequence gives the right answer. It is not idempotent, and lesson 019's rule says why: a relative or conditional write is safe to repeat only when its own effect falsifies its condition, and adding one does not. Two sweeps settling the same row put the copy back twice. The forward step at least had a where stock > 0, which lesson 019 showed refuses a repeat on the last copy and nowhere else. The compensation had no condition at all, because nothing in it said "and this order has not already been released".

Marlow has the fix, and it is the one to carry into any saga you write. Lesson 035 ended on it:

update orders set status = 'abandoned'
 where id = $1 and status = 'pending';

with the increment conditional on that statement's row count, in the same transaction. The second sweeper matches zero rows and puts nothing back. Lesson 035 called that the row's own status is a fence with two values, which is lesson 019's first pile doing the work: the already idempotent one, where nothing needs naming because the write refuses itself. In today's vocabulary that column is the whole of the compensation's safety, and it is the cheapest you will ever get, because you were already storing it.

A compensated transaction is not an aborted one

An aborted transaction never existed. Lesson 015's isolation levels are promises about exactly that: no other reader ever sees your half-done work, whatever happens to it. A saga makes the opposite promise, and makes it by design. Step one commits. From that instant every reader in the system can see it, and will carry on seeing it until the compensation commits, which may be four minutes later or Friday morning.

So a saga gives up isolation. Not turned down a notch, the way lesson 015's dial turns down. There is none, and no setting anywhere turns it on.

Marlow's version is the four minutes a copy spends off the shelf while a card call that will fail is still being asked about. Price it honestly. The shop sees forty requests a second on an ordinary Tuesday, so about 9,600 requests in four minutes, and if every single one were a book page for a different title out of 1.2 million, a given title gets looked at 0.008 times. One chance in a hundred and twenty five, with an assumption generous in the direction that makes the number bigger. Lesson 030 chose to hold at intent knowing that cost, on the grounds that refusing one sale wrongly annoys a customer the shop has 1.2 million other titles for, while overselling twelve signed first editions is a letter from a solicitor.

Then notice the assumption that is doing the real work, which is that attention is spread evenly over the catalogue. Lesson 008 refused that model in its own words: a few thousand titles carry most of a bookshop's day, which is why its cache gets 94 percent where an even spread would get almost nothing. Call it three thousand titles carrying most of those 9,600 requests, and a moving title is looked at about two and a half times in the four minutes rather than one time in a hundred and twenty five. A factor of three hundred, and the title whose last copy has just sold is in the small set by definition. Lesson 012 made the same complaint about medians. That does not overturn lesson 030's choice, because the other side of its asymmetry is still a solicitor. It does mean the cheap side was never free, and nobody at Marlow has counted the customers who met a zero that was going to be refunded four minutes later.

Three countermeasures are worth the words, and you have met one of them already.

The pending status is a semantic lock: a column that tells every other reader a saga is in flight over this row, so they can refuse, wait, or hide it. It is not a database lock. Nothing enforces it except every piece of code that remembers to look, which is both the appeal and the catch. It also makes the waiting explicit in your own schema, which is more than you get from four minutes of silence.

Commutativity is the weak one. Adding and subtracting stock survives any order of arrival; setting it to a value does not.

Reordering is the one that actually fixes things. Put the step most likely to fail first, so most failed sagas never reach the expensive steps at all. Marlow does the opposite: the step that fails most often, a human finding the copy, is last.

The pivot step, and why your confirmation email is worded like that

Steps sort into three kinds, and this split is the standard way the pattern is taught.

Before some point in the flow, every step has a compensation you would actually be willing to run. That is the compensatable stretch. Then comes one step after which you are committed, either because nothing undoes it or because undoing it is worse than finishing: the pivot. Everything after it has to be retriable, meaning the only acceptable outcome is eventual success, so it gets retried until it succeeds and it had better be idempotent.

Marlow's pivot is the email at step four. The shop's failure is now a one line diagnosis: its most failure-prone step sits after its pivot.

Two fixes, and this course keeps watching the shop pick the cheaper one.

Move the pick earlier, which means somebody takes a book off a shelf before the card is charged. For a four person shop that packs in the mornings and sells at lesson 027's one order every fifty seconds, that is a business change pretending to be an architecture, and it is not happening.

Or move the pivot, which costs nothing and is what the rest of the world already does. Go and read any confirmation email you have received this month. It says your order has been received, or placed, or that the shop is getting it ready. Few of them claim the shop has the thing in its hands. The wording of the email is where the pivot actually is, and somebody in e-commerce worked that out a long time before anybody drew a saga diagram. Send "we have your order" at step four and the pivot moves to dispatch, by which point the copy is in somebody's hand and the shop knows it exists. The email becomes compensatable, because the apology is now a cancellation notice rather than a contradiction.

At Stagefront, where an oversold seat is a lawsuit, the pivot is the charge, which is why lesson 015's shape is the right one: the seats are claimed and committed before the money moves, so everything compensatable happens first, and the only thing after the pivot is issuing a ticket, which can be retried all day.

What a saga would have cost Stagefront

Lesson 013 put this choice in front of Stagefront, named both tools in one clause, and left the answer in a check question. Today it gets priced.

Shard the seats and a purchase of four adjacent seats spans up to four machines. There is no transaction. Either two-phase commit, or a saga: claim 14F on shard 2, claim 14G on shard 3, claim 14H on shard 1, and if 14J, on shard 4, is already gone, release the three you are holding.

The latency is nothing. Four claims at lesson 015's millisecond plus lesson 002's half millisecond round trip is about six milliseconds, and three releases add four and a half. Nobody notices ten milliseconds in a minute where the seat map on the screen is already up to ten seconds stale, because lesson 008 gave it a ten second TTL on purpose and lesson 009 explained why invalidating it would be worse.

The cost is what a lost release leaves behind. If the process dies between the third claim and the fourth, or the compensation times out, three seats stay claimed by nobody. Lesson 018 already published what that costs here, and the chain is worth re-reading: nothing errors anywhere, every abandoned seat leaves the sale permanently, the show sells out with rows of empty seats in it, and the first instrument to notice is a photograph of the stadium.

So the by-seat cut does not buy a latency problem. It buys four sweepers, each needing lesson 032's election to decide which box runs it and lesson 035's fence so that it does not matter when two do, in a business where the failure mode is a lawyer's letter either way. And the choice lesson 013 offered has an answer today that lesson 013 could not give: when every participant is a database you run, two-phase commit is the honest tool, and a saga is the wrong one. Sagas exist for the participant you do not own. Stagefront's four shards are all its own Postgres, so the real comparison was never saga against 2PC. It was 2PC's held locks against lesson 024's refusal to distribute the row at all, and lesson 024 chose the ceiling lesson 013 declined to remove.

Galewatch, which takes a reading every two seconds from each of the wind turbines on the farms it monitors, has no saga anywhere in its ingest path, and the reason is a design lever rather than an accident: every operation in that path is an append. An append needs no compensation. If you can turn a step into an append, and derive the thing people actually read from the appended facts, the saga problem does not arise, which is half of what lesson 037's log is for. Lesson 059 has more to say about records you only add to.

Orchestration, choreography, and a row that remembers

Two ways to run a saga, and you have seen both shapes already.

Choreography is lesson 038's architecture pointed at a transaction. Each service listens for the events it cares about and publishes its own. Nobody owns the flow. Adding a step is free.

Follow it through at Marlow and the appeal thins out. Suppose the charge moved into a payment service of its own, which is the shape choreography assumes. The checkout publishes order_placed, the payment service charges and publishes payment_failed, and now something has to release the copy. The only service that knows how is the checkout, so the checkout subscribes to its own consumer's failures. Lesson 038's whole selling point was a producer that stops knowing who is listening; the compensation path hands that knowledge straight back, because every participant has to understand which later step's failure obliges it to undo its own. The flow then exists nowhere except in the union of several subscriptions, which lesson 038 priced: the broker's group list is the only honest inventory of who depends on you, and it is stale the next time anybody deploys.

Orchestration puts one component in charge. It holds the state machine, calls each step or sends it a command, records where the flow got to, and runs the compensations in reverse when a step refuses. One place to look when somebody asks where Thursday's order got to, which is the question lesson 038 left you a forest of unparented traces to answer.

My rule, and the reason under it matters more than the rule. Orchestrate anything with money in it, or more than about three steps, or a compensation chain. Choreograph notifications, derived views, and anything where a failed consumer is somebody else's problem to retry. The reason is not elegance: it is that a compensation path is the code that runs least often in your entire system, and the only way it ever gets tested is if somebody can point at the thing that owns it.

You do not need to buy software for this. Temporal, AWS Step Functions and Netflix Conductor all exist to be exactly this component, and what they sell is real: the state machine survives the process dying, and timers and retries stop being your crontab. What they cost is a new piece of infrastructure to run and your business logic rewritten as their workflow.

Your orchestrator is a row that remembers which step you are on. Marlow's is an order table with a status column and a cron line, and it got the shop from January to May. What it lacks is a step counter. pending and paid say what the money did rather than which step the flow is on, and today those happen to coincide. Add a step that moves no money, a dispatch after the pick, and paid has to mean both waiting for a shelf and picked but not sent, with the sweeper unable to tell the two apart. That is one column, and it is cheaper before the third step than after it.

Recap

There is no rollback across machines, only a second operation that apologises. A rollback is an entry in a log you own, trusted because every reader of those rows comes through you. The moment a step commits in somebody else's log, a provider's ledger, an inbox, a shelf, the only undo available is a new entry going the other way, on their terms and their clock.

A compensation is not an inverse. stock + 1 was the inverse of Marlow's forward step and the wrong answer, because the copy never existed. The question a compensation answers is what the business wants to be true now, and that is a decision somebody has to make rather than a sign somebody can flip. Marlow kept the distributor order for the same reason.

A compensated transaction is not an aborted one. An aborted transaction never existed; a compensated one was visible to every reader for its whole length, and somebody may have acted on it. A saga gives up lesson 015's isolation completely, and the pending column is the only lock you get: a semantic lock, enforced by every piece of code that remembers to look.

Every compensation is at-least-once too. The second attempt arrives as a redelivery or as the next sweep, and lesson 036's gap applies to the undo path exactly as it does to the forward one. Sort a compensation into lesson 019's piles part by part: Marlow's stock write sits in the free pile behind the status guard, and the refund call needs a name of its own, the same mechanism the charge has carried since lesson 019. Commutative is not idempotent, and lesson 035's two phantom copies are what that distinction costs in a back room.

Everything before the pivot must be undoable and everything after it must be driven to success. Marlow's most fragile step, a human and a shelf, sits after its pivot, which is the confirmation email. The cheap fix is not architecture. It is the sentence every e-commerce email you have ever received is carefully worded to say: we have received your order.

Sagas are for the participant you do not own. When all four participants are your own database shards, two-phase commit is the honest tool and its bill is held locks; Stagefront paid lesson 013's ceiling instead, and lesson 024 is where it decided that. An append needs no compensation at all, which is why Galewatch's ingest path has none.

Your orchestrator is a row that remembers which step you are on. Choreography makes the flow free to extend and leaves the compensation path owned by nobody, which is the code in your system that runs least and is tested least.

Check your understanding

1. A colleague proposes two-phase commit between your service's Postgres and your payment provider's API, having read that it gives real atomicity. Say what you would ask them first, and what you would expect each participant to be able to do.

2. Your sign-up flow's third step sends a welcome email, and its compensation is a second email saying to ignore the first. Say what is wrong with that, and where you would move the pivot.

3. Design the compensation Marlow actually needed on the Friday: a paid order for a copy that never existed. Say which of lesson 019's three piles each part of your compensation sits in, what the stock write should be, and how you make the whole thing safe to run twice.

4. Stagefront's new engineer proposes lesson 013's by-seat shard cut with a saga over four shards. You have one on-sale to gather evidence. Say what you would measure during those eighty seconds, what you would insist on building before the next sale, and which of today's two tools you would argue for.

5. A choreographed saga spans four services and one compensation event is lost on a Tuesday. Say how you would find out at all, what you would expect to notice first, and which earlier lesson each instrument you add comes from.

6. The founder reads about durable workflow engines and wants to buy one for Marlow's order flow. Argue both sides with this lesson's figures, then say what you would ship this week instead, and what would have to change at the shop before the answer flips.

Next lesson

040 The Outbox Pattern: Writing to a Database and a Queue Safely. Today's saga assumed each step's event actually gets published once the step commits; next lesson takes the gap between Marlow's paid commit and the message leaving the building, which is the one failure in today's flow that nothing above fixes.