Idempotency Keys in Practice

Intermediate · 17 min

How a client-generated idempotency key, a dedup table, and one transaction make an unsafe retry like "charge this card" safe.

Before thisTransactions and Isolation Levels in Application Code

Arpit Bhayani · 17 min

Idempotency Keys in Practice

The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.

Taught by Arpit BhayaniReport a problem with this video
Not started
Set this lesson's status

What you'll learn

  • A network timeout doesn't tell you whether the request it was attached to actually succeeded on the server — the server might have charged the card and the response just never made it back.
  • Retrying a non-idempotent operation blindly is how one real payment becomes two: the safe assumption after a timeout is that it maybe worked, not that it definitely failed.
  • An idempotency key is generated by the client once per logical operation and reused on every retry of that same operation — generating a new key per retry defeats the entire mechanism.
  • The server has to record the key and perform the operation inside the same transaction, or a retry that lands in the gap between 'recorded the key' and 'charged the card' can still double-charge.

The problem: a timeout doesn’t mean “it failed”

Picture a client calling POST /charges to bill a customer’s card. The request reaches the server, the payment processor charges the card successfully, and then — a network blip, a server restart mid-response, a load balancer timeout — the confirmation never makes it back to the client. From the client’s point of view, the request simply failed to respond in time.

Here’s the uncomfortable part: the client genuinely cannot tell, from a timeout alone, whether the charge happened or not. “No response” and “the operation never ran” are not the same fact, even though they feel the same from where the client is sitting. A client that’s written to retry on failure — which is good, normal practice for a transient network error — will resend the exact same charge request. If the original charge actually went through, the customer is now billed twice for the thing that happened once.

This is specifically a problem for operations that aren’t naturally idempotent. Fetching a record twice is harmless — you just get the same data back. Charging a card, creating an order, or sending a transfer is different: running it twice has a real-world effect the second time that you didn’t want. The fix isn’t “don’t retry” — retries are essential for a system that has to tolerate real network failures — the fix is making the operation safe to retry.

The mechanism: a key the client owns, a record the server keeps

An idempotency key is a unique identifier the client generates once, before making the request, representing “this specific logical operation” rather than “this specific network attempt.” A UUID generated client-side works well. The client sends it alongside the request, usually as a header:

POST /charges
Idempotency-Key: 7f3c9e2a-91d4-4b8f-9c11-2e5d6a8b0f31
Content-Type: application/json

{"amount": 5000, "currency": "usd", "customer": "cus_12345"}

If the request times out, the client retries — same endpoint, same body, and critically, the same idempotency key, not a freshly generated one. That reuse is the entire mechanism: without it, the server has no way to tell this retry apart from a brand-new charge request, and the whole scheme does nothing.

On the server side, this means keeping a record — commonly a dedicated table — mapping each key to the outcome of the operation it was attached to:

CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  response_status INT NOT NULL,
  response_body JSONB NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

When a request arrives, the server’s logic is: look up the key. If it’s already there, skip the actual work entirely and return the stored response — the retry gets back exactly what the original call would have returned, with no new charge, no new order, nothing re-executed. If the key isn’t there, this is a genuinely new operation: do the real work, then record the key and the result before responding.

Why it has to be one transaction, not two steps

The part that’s easy to get wrong is treating “do the work” and “record the key” as two separate steps run one after the other. Picture this ordering: charge the card, then insert the idempotency key row. A retry that arrives in the gap between those two steps — maybe because the server crashed right after the charge succeeded but before the insert ran — finds no existing key, concludes this is a new request, and charges the card again. The dedup table existed, but the gap between the two operations left a window where it didn’t protect anything.

The fix is to perform the operation and record the key inside a single database transaction, using exactly the kind of transaction boundary covered in transactions-and-isolation-levels-in-application-code:

BEGIN;

-- unique constraint on `key` makes this fail fast if a concurrent
-- request already inserted the same key
INSERT INTO idempotency_keys (key, response_status, response_body)
VALUES ($1, 200, $2);

-- the actual side-effecting work: create the charge, decrement
-- inventory, whatever this operation actually does
INSERT INTO charges (customer_id, amount_cents, idempotency_key)
VALUES ($3, $4, $1);

COMMIT;

Now either both rows exist or neither does. There’s no gap for a concurrent or retried request to land in: if the transaction committed, the charge happened and the key is recorded as having happened, atomically, as one fact. A unique constraint on the key column also gives you a second layer of safety — if two requests carrying the same key somehow race each other and both try to insert, the database rejects the second insert outright rather than letting both charges through.

It’s worth being precise about what this does and doesn’t protect against. It makes a retry of the same client request safe — the case this lesson is built around. It does not make two genuinely different requests that happen to represent “the same” real-world intent (a user double-clicking “Pay Now” and generating two separate requests with two separate keys, because the client didn’t dedupe on its own side) safe automatically — that’s a client-side concern, handled by generating the key once per user action and disabling the button on click, not something the server’s dedup table can fix after the fact. The server-side mechanism only works if the client actually holds up its end: one key, created once, reused on every retry of that one operation.

This lesson is the implementation: the table, the transaction, the constraint. Idempotent APIs and Exactly-Once Semantics is the contract-level question this builds on — what it even means for an API to promise idempotency, and why that promise matters at the architecture level before you write a single INSERT.

Check yourself

  1. 01Why does a network timeout create a genuine ambiguity that a simple retry can't resolve on its own?

    A timeout means the client stopped waiting for a response — it says nothing about whether the server actually finished processing the request before the connection dropped. The charge might have gone through and only the confirmation got lost in transit. A client that treats every timeout as 'it failed, try again' will, some fraction of the time, be retrying an operation that already succeeded.

  2. 02Why must the client reuse the same idempotency key on a retry instead of generating a fresh one?

    The key is what lets the server recognize a retry as the same logical operation rather than a brand-new one. If the client generates a new key every time it retries, each attempt looks like an unrelated request to the server, and the entire point of the mechanism — collapsing retries of one operation into one effect — never happens. The key has to be created once, before the first attempt, and carried through every retry of that specific operation.

  3. 03Where does the server need to store the idempotency key, and why does that storage need to happen inside the same transaction as the actual operation?

    The server records the key, typically in a dedicated table with a unique constraint on the key column, alongside the result of the operation. If recording the key and performing the operation (charging the card) happen as two separate steps rather than one transaction, a retry can arrive in the narrow gap between them — after the charge happened but before the key was saved — and see no existing record, so it charges the card again. Wrapping both in one transaction means either both happen or neither does, closing that gap.

  4. 04What should happen when a retry arrives with a key that's already recorded?

    The server looks up the key, finds the stored result from the original attempt, and returns that same result immediately without re-running the operation at all — no second charge, no second order created. The retry gets a response that looks identical to what the original request would have returned had its response not been lost, which is exactly what makes the retry safe from the client's point of view.

  5. 05How does this lesson's scope differ from idempotent-apis-and-exactly-once-semantics in the system-design roadmap?

    The system-design lesson is about the API contract itself — what it means for an API to promise idempotency, why that promise matters for clients building retry logic against it, and the broader exactly-once-semantics problem across a distributed system. This lesson is the concrete implementation underneath that promise: the actual dedup table, the actual transaction boundary, and the actual code path a server runs to make a specific endpoint honor it.

Go deeper

Notes

Private to you

Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark