Idempotency: Safe to Retry

Marlow Books is a four person online bookshop that exists only in this course, and every January the founder sits down with December's card settlements and the order table and checks that the two sides agree.

This January they did not. Seventeen charges in December had money against them and nothing in the order table behind them.

Eleven of the seventeen belonged to customers who had a second, successful order for the same title a minute or two later. Six did not. Those six people paid for a book, saw an error page, and went to buy it somewhere else.

One of the eleven was a customer called Priya, on the Saturday before Christmas at 20:14. She pressed buy. Marlow's checkout works in the shape lesson 015 published: take the stock row with a single conditional UPDATE, call the payment provider from inside that same transaction, then write the order.

The provider usually answers in about three hundred milliseconds. That evening it took twelve seconds. Marlow's HTTP client gives up at ten.

So the call raised, the transaction rolled back, the decrement was undone, and no order was ever written. For those ten seconds nobody else could buy that title either, which is lesson 015's lock queue at thirty times the length it was priced at. Priya got an error page. She pressed buy again, the provider answered in 280 milliseconds that time, and her book went out on the Monday.

The provider's own records show both charges. The first was authorised at 20:14:01, nine seconds before Marlow stopped listening, and the reply left the provider's building at 20:14:12, two seconds after there was nobody left to read it. The work happened. The receipt is what went missing.

Lesson 001 said this in one sentence, and the whole of today is that sentence. A call that returns an error has told you something. A call that times out has told you nothing at all.

What the word actually promises

An operation is idempotent when doing it twice leaves the system in the same state as doing it once. That is the entire definition, and two neighbouring words get tangled up with it constantly.

Safe means the operation changes nothing at all. Reading a book page is safe. Deleting an order is not.

Pure means it returns the same answer every time. A search over the catalogue is pure until somebody adds a title.

An operation can be idempotent without being either. DELETE /order/55 changes the world the first time you send it, so it is not safe. Send it again and there is nothing left to delete, so the state afterwards is identical, which makes it idempotent. It also returns a different status code the second time, so it is not pure. None of that matters. Idempotency is a claim about state, not about answers.

HTTP has had opinions about this column since the 1.1 specification. Lesson 004 pointed out that the specification names GET, HEAD, PUT and DELETE idempotent and pointedly leaves POST out. That is not a description of what your code does. It is a promise your handlers are expected to keep, and the specification has no way to make you keep it.

Lesson 004 also gave you the cleanest demonstration of why this stops being academic. TLS 1.3 lets a returning client send its request in the very first packet, saving a whole round trip, and the price is that anyone who captured those bytes can send them again later with the server unable to tell the copy from the original. A feature down at the transport layer turns into a question about your handlers: which of your requests are safe to receive twice? Every layer below you is allowed to duplicate. None of them will ask first.

Lesson 015 fixed the other bug

Here is the part that catches people out, and it is the reason July's fix did nothing at all for December.

Lesson 015 spent a section on Marlow's July flash sale, twelve signed first editions that sold thirteen copies, and the fix it shipped was one statement:

UPDATE books SET stock = stock - 1
 WHERE isbn = '9780571358939' AND stock > 0;

That statement is atomic. The read and the write are the same operation, so there is no window between deciding and acting for a second buyer to climb into. It is the right fix for the bug 015 was looking at and it is still the first thing I reach for.

It is not idempotent. Run it twice and you have sold two books.

Atomic and idempotent are answers to different questions. Atomic asks whether somebody else can interleave with you. Idempotent asks whether you can happen twice. Concurrency and retries are separate hazards and each has its own fix, and there is a real mean streak here: the relative decrement that cures the first one is the thing that creates the second. Lesson 015 told you that a relative update cures the classic lost update, which is true, and the same relative update is exactly what makes a retried checkout sell a book you do not have.

Now put a conditional update from a different part of the course beside it. Stagefront, the ticketing service where a stadium show goes on sale at exactly 10:00 and two hundred thousand people press the same button, turns a seat hold into a sale like this:

UPDATE holds SET status = 'sold'
 WHERE hold_id = $1 AND status = 'held';

Same shape. Conditional update, check the row count. Run it twice and the second one matches nothing, because the row already says sold. That one is idempotent.

The difference is worth saying as a rule, because it is checkable in ten seconds on any statement you write. A conditional update is idempotent when its own effect falsifies its condition. Setting status to sold destroys status = 'held'. Subtracting one from stock does not destroy stock > 0.

Well, once. It destroys it on the last copy, when the stock goes to zero and the second run matches nothing. So Marlow's decrement is idempotent precisely on the sale that empties the shelf, and wrong on every other one, which is a lovely way to have a bug that passes the test you wrote for it.

Three kinds of work

Sort your operations into three piles. The piles have very different prices and most teams find out which pile something is in during an incident.

Already idempotent, for free. Writing an absolute value. A PUT that replaces a delivery address with the one the customer typed. A state transition guarded by the state it is transitioning out of, like Stagefront's hold. You do not have to do anything to these, and you should still write down that they are in this pile, because the next person to add a counter to that handler will quietly move it into another one.

Idempotent once you give the work a name. This is the big middle pile and it is where the interesting engineering lives. If you can name the thing being done, uniquely, and make the database refuse a second row with the same name, the second attempt fails at the constraint and you catch that failure and call it success.

The course already has one of these and has had it since lesson 007. The payout job's claim is an insert on (job_name, run_date) with on conflict do nothing, and the reason it works is that the run has a name and no two runs can share it. Lesson 018 built the table and left a debt sitting in it: finished_at as a lease, so a second box can retake a run the first box abandoned. That debt is due today, and the answer is not comfortable.

The payout job loads a year of order lines and emails each publisher a CSV. Emailing is in the third pile. So you split the job: write each publisher's CSV to object storage under a key of (publisher, month), the shelf lesson 016 moved Marlow's covers onto in December, where writing the same key twice overwrites harmlessly, then stamp a sent_at on a row per publisher before the send. Two boxes racing produce one file each publisher and one email each publisher, and the job as a whole is safe to run twice because the only part that was not has been fenced behind a row.

Stamping before the send is the uncomfortable half, and it is a choice rather than a fix. A box that dies in the gap leaves a row saying sent and a publisher who was not. Stamp after the send instead and the same crash emails that publisher twice. There is no third option here, and lesson 036 is where that stops being a surprise and becomes a thing you pick on purpose.

Galewatch, which collects a reading from each of nine hundred wind turbines every two seconds, has the cheapest version of this in the whole course. Lesson 017 mentioned in one clause what a turbine does when a write times out: it sends the reading again a minute later. Some of those writes landed before they timed out, so Galewatch has been storing duplicate readings for as long as it has existed. The fix is that the data already names itself. A reading is a turbine and a moment.

insert into readings (turbine_id, recorded_at, power_kw)
values ($1, $2, $3)
on conflict (turbine_id, recorded_at) do nothing;

The index that makes this possible is the one lesson 010 already praised for having its columns in the right order, (turbine_id, recorded_at), built so an engineer can open one turbine's panel. Make it unique and the same index does a second job for nothing. That is as good as this gets, and it is only available because the reading carries its own identity. There is a seam in it: the timestamp comes from a clock inside the turbine, so the uniqueness is exactly as trustworthy as that clock, and clocks are lesson 034.

Not idempotent at any price. Charging a card. Sending an email. Posting to somebody else's API. You cannot make another organisation's handler idempotent by being careful on your side, and no amount of design in Marlow's codebase makes the payment provider forget it has seen Priya's charge. Everything you get to do about this pile happens before the call or after it, never inside it.

The key names the intent

For the middle pile you need a name, and for anything a customer initiates the only party who knows what one intent looks like is the customer's own browser.

An idempotency key is a unique string the caller makes up to name an action it intends to perform, and sends with every attempt at that action. Not a hash of the request. A name for the intention.

The generation point is where this gets built wrong, and it is the difference between working and looking like it works.

Generate the key when the checkout page renders and put it in a hidden field. Priya presses buy, the call times out, she presses buy again, and the second request carries the same key as the first. That works.

Generate the key when the button is pressed and every press makes a fresh name, so the server sees two different intentions and honours both. You have added a column, a table and an index, and changed nothing at all. A retry that invents a new key has built nothing.

The server side is one table and one insert.

create table idempotency_keys (
  key         text primary key,
  fingerprint text not null,
  status      text not null,
  response    jsonb,
  expires_at  timestamptz not null
);

insert into idempotency_keys (key, fingerprint, status, expires_at)
values ($1, $2, 'in_progress', now() + interval '24 hours')
on conflict (key) do nothing
returning key;

If that insert returns a row, this attempt owns the work and goes ahead. If it returns nothing, somebody has been here before, and what happens next depends on what the existing row says.

A row saying done is the good case. You do not do the work again. You return the stored response, byte for byte, and the caller cannot tell whether you just did the work or remembered doing it. That is the point: an idempotent endpoint does not merely avoid a second charge, it gives the retry the same answer the lost reply would have given. Store the response or you have built a deduplicator rather than an idempotent endpoint, and the client is still stuck.

A row saying in_progress is the awkward one. Another attempt is running right now, you do not know its answer, and you must not invent one. Say so. HTTP has a code for exactly this, 409 Conflict, and lesson 004's rule applies without modification: put the truth in the status line, because everything between you and your caller reads the number and never opens the body.

The fingerprint column is a hash of the request body, and it is there to catch a client bug rather than a duplicate. Same key, different body, means the caller has reused a name for a different action, and returning the stored answer to a request you never actually looked at is a worse outcome than an error. Refuse it loudly.

One more thing about in_progress, which brings lesson 007 back around. If the process that claimed the key dies mid way, that row sits there forever and the customer can never complete this purchase at all. So the row needs an expiry after which another attempt may take it over, which is a lease, which is the exact thing lesson 018 said would require the work to be safe to run twice. The recursion is real and it does not resolve. You cannot escape needing a lease. You can only make the work behind it safe, which is the three piles again, arriving from the other direction.

What it costs to remember, and for how long

Two questions, and the second one is more interesting than it looks.

The storage first, because it is small and people assume it is not. Lesson 009 established that Marlow packs about one order every five seconds at the Christmas peak, so call it a fifth of an order a second. A day at that rate is 17,280 orders. A row here is a key of thirty six characters, a hash of sixty four, a status, a small stored response of a hundred and twenty bytes or so, a timestamp and Postgres's row header, which lands around three hundred bytes, the same order as the two hundred byte session rows lesson 007 measured.

17,280 times 300 bytes is about five megabytes, plus a megabyte or so of primary key index.

That is a day of keys priced at the busiest hour of the busiest week of the year and then applied to all twenty four hours, which is deliberately the wrong way round. Lesson 007's session table is 8 megabytes and lives entirely in the 8 gigabyte buffer pool lesson 010 spent a section inside. The complete memory of every intention every customer has expressed in a day comes to less than the sessions of the people expressing them.

And it does not grow. The expiry makes this a rolling window rather than a ledger, so five megabytes is not the first day's bill, it is the bill on the worst day, forever.

Now the expiry, which is not a storage question at all.

The right retention is: how long can a retry of this request still arrive? For a browser, that is a person's patience, so minutes. For lesson 018's email consumer, it is the retry deadline, which 018 set at four hours. Set the expiry shorter than that and the last retry finds no key, decides the request is new, and does the work again. Your key's lifetime has to outlive your own retry deadline, and 018 already told you most people have never worked out what that deadline is.

Then the same question one hop further out, where it gets genuinely nasty.

When Marlow sends an idempotency key to the payment provider, the window that matters is the provider's, not Marlow's. Lesson 018 made the consumer retry to a four hour deadline. Suppose the provider remembers keys for one hour. Every retry after the first hour is a brand new charge as far as they are concerned, and the safety Marlow believes it bought is gone at exactly the moment a long outage is happening. Your retry deadline and the far end's dedupe window are the same number, and you probably picked neither of them.

Borrow a real retry schedule to work it on: 018's published Tuesday. The email provider was down 10:00 to 12:00, and the message placed at 10:56 attempted at 10:56, 10:57, 10:59, 11:03, 11:11, 11:27, 11:59 and finally 13:03. First attempt to last is two hours and seven minutes. Against a twenty four hour dedupe window, fine. Against a one hour window, the key expires at 11:56 and the last two attempts are outside it.

Except no duplicate email was sent that Tuesday, and the reason is the sharpest thing on this page.

Every one of those attempts came back 503. A 503 is information. Lesson 004 put that code in the retry contract as out of room right now rather than broken, which is a server telling you it never reached your request. Take it at its word and nothing landed, so nothing could be duplicated, and a far end that is cleanly dead is the safest far end there is.

The one that hurt Marlow was the provider being slow. Priya's charge went through and the answer did not come back. Lesson 018's Friday, when the email provider was sluggish and the queue backed up, is the same weather. A dependency that is down is safe. A dependency that is slow is the one that duplicates. That inverts how most of us rank third party failures, and it is why the outage that produces a support queue full of double charges is never the one that made the dashboard red.

The half you do not own

Which leaves the pile that cannot be made idempotent, where Marlow's seventeen charges live.

You cannot fix the provider. You can do two things, and Marlow did both in the second week of January.

Ask what the far end offers. Any payment provider worth taking money through accepts a key on the charge call and will tell you in its documentation how long it remembers one. Marlow's says twenty four hours. Nobody at the shop had ever read that sentence, which is the ordinary way this goes: the feature exists, it costs one header, and it is on page four.

Then write the intent down before you act on it. This is the reshaping lesson 015 argued for and handed forward, and it is finally built. The order row is inserted and committed as pending, carrying the idempotency key, before the card is touched. The card call happens with no transaction open. A second, short transaction moves the row to paid.

Count the states now. Charge succeeded and row updated: done. Charge refused: mark the row failed. Charge timed out: the row is still sitting there saying pending, and that row is the thing December did not have. It is evidence that Marlow intended to take money, written down before the money moved, so a timeout is no longer a fact that exists only on somebody else's server.

A sweeper runs every two minutes over orders that have been pending for longer than that, asks the provider what happened to each key, and either completes the order or refunds the charge. That is the founder's January reconciliation, except it runs seven hundred and twenty times a day instead of once a year, and it is a scheduled job, so it goes in the job_runs table lesson 018 built with a deadline beside its schedule.

There is a name for writing your intention into your own database before you send it anywhere, and it is the outbox pattern, and it is lesson 040. Lesson 036 has what a broker can and cannot actually promise about delivering a message once. Both of those go deeper than Marlow needs in January.

The honest limit on all of this is worth saying plainly, because it is the boundary of what idempotency means. Marlow's key stops Priya being charged twice for one intention. It does not stop her opening a new tab an hour later and buying the same book again, and it must not, because that is a second intention and a real sale. Only the caller knows which is which. The server cannot infer intent from a request that looks identical, and any scheme that tries will eventually refuse somebody who genuinely wanted two.

Priya got her refund in January, four minutes of the founder's time, because she had written in and could be found. The six who never came back took an afternoon, since finding a customer from a settlement line when there is no order to join it against is a job for a person and a spreadsheet.

That afternoon is what the pending row abolishes. It does not stop the provider being slow and it does not stop a charge going astray. It makes the next one a row somebody can find in two minutes, instead of a discrepancy that waits for January.

Recap

Idempotent means doing it twice leaves the same state. Not harmless, not read only, not deterministic. DELETE is idempotent and destructive at once, and the HTTP specification has been telling you which of your verbs owe this promise since before any of us started.

Atomic is not idempotent. Lesson 015's conditional UPDATE closes the window between reading and writing and does nothing whatever about the same request arriving twice. A conditional update is idempotent only when its own effect falsifies its condition, which is why turning a hold into a sale is safe to repeat and decrementing stock is not.

Work sorts into three piles: free, named, or impossible. The middle pile is the one you build, and building it means giving the intended action a unique name and letting a constraint refuse the second copy, which is lesson 007's on conflict do nothing pointed at a customer instead of a cron job.

The key names the intent, not the attempt. Generate it where the intention forms, which is the page, not the press. A retry that invents a new key has built nothing, and an endpoint that dedupes without storing the response has left the caller exactly as stuck as before.

Your key's lifetime has to outlive your retry deadline, and theirs has to outlive yours. Lesson 018 found the sum of a backoff schedule nobody had computed. This is the same arithmetic meeting a number a third party chose and published, and the two are almost never compared.

A dependency that is down is safe. A dependency that is slow is the one that duplicates. A 503 tells you nothing happened. A timeout tells you nothing at all, and Priya's twelve second charge was authorised nine seconds before Marlow stopped listening.

Write the intent down before you act on it. A pending row committed before the card call turns an unanswerable question into a row you can sweep, which is the difference between finding seventeen charges in January and finding one in two minutes.

Check your understanding

1. A process claims an idempotency key, writes in_progress, and is killed by a deploy before it does anything else. Describe what the customer sees on their next attempt, what you would put on that row so they are not stuck forever, and why your answer drags lesson 007's lease back into a lesson that was supposed to have solved it.

2. A Galewatch turbine's clock drifts and it stamps two genuinely different readings with the same recorded_at. Say what the unique constraint on (turbine_id, recorded_at) does in that moment, whether silently discarding the second reading is the right behaviour for this product, and what you would change if it is not.

3. Marlow's email provider offers no idempotency key of any kind. Using lesson 018's retry schedule for that queue, describe what you would build so a customer is not emailed twice about one order, and then say honestly which duplicate your design still cannot prevent.

4. Take the payment provider's twenty four hour dedupe window and Marlow's four hour retry deadline and say whether a duplicate charge is possible. Then redo it for a provider that remembers keys for fifteen minutes, and say which of the two numbers you would try to change.

5. A colleague proposes skipping idempotency keys entirely: hash the request body, treat any identical body from the same customer within five minutes as a duplicate, and return the earlier response. Give the strongest version of their argument, then the customer it fails, then say what it would take to make their scheme safe.

Next lesson

020 Timeouts, Retries and the Retry Storm. Today earned you the right to retry safely; next lesson is about the retry that finishes off the service it was waiting for, and how to choose a timeout you could defend to the person whose service you just flattened.