Is a unique index on the reference not enough?

It stops the duplicate row and nothing else. The second attempt still updates the balance, still emits the event, still sends the receipt, and only fails at the very end. It also turns a benign retry into a server error, which most providers read as failure and answer by retrying harder.

What should the idempotency key be?

The provider's own transaction reference, scoped by tenant. Not your own generated id, which differs per attempt. Not a timestamp. Not a hash of the payload, because providers reformat payloads and the hash then changes for the same event, which fails while looking like it works.

How far back should the reconciler look?

The fast sweep needs a couple of minutes to catch crashed workers. The daily pass should cover at least a full settlement cycle, because its job is finding events that never arrived, which the fast path cannot detect by definition.

Do you need a queue for this?

You need somewhere durable to put the event and something that will process it later even if this process dies. A queue is the usual answer. A table plus a sweep is a perfectly good one and has fewer moving parts.

Solution

How to make a payment webhook safe to receive twice

Every payment integration works in testing. The one that matters is what happens when the provider retries because your acknowledgement was slow, and a customer is credited twice for one payment.

The short answer

Acknowledge in milliseconds having persisted only the raw event. Do the real work in a worker, keyed on the provider's own reference scoped by tenant. Claim the key before acting rather than relying on a unique constraint. Then run a sweep that assumes the worker died, and a daily comparison that catches events which never arrived.

Three layers of a payment webhook, the fast path which receives, persists raw, acknowledges, claims an idempotency key and commits one transaction, a sweep every two minutes catching workers that died, and a daily comparison against the provider statement catching events that never arrived — Three layers, each catching what the layer above it structurally cannot. Most implementations build only the first. Tap to open full size.

The failure, stated precisely

A provider sends you a payment notification. Your handler validates it, looks up the account, posts to the ledger, updates the balance, sends a receipt, and returns two hundred.

That took eight hundred milliseconds. The provider’s timeout was five hundred. It never saw your response, so it assumes failure and sends the same event again.

Now two handlers are running concurrently against the same payment. Both find the account. Both post. The customer is credited twice, and the first person to notice is either an accountant at month end or the customer, who will not mention it.

Everything below exists to make that impossible.

Step 1. Split acknowledgement from work

The acknowledgement is a promise that you have the event. It is not a promise that you have acted on it. Conflating those two is the root cause of most double posting.

So the handler does exactly two things: verify the signature, and persist the raw body. Then it returns.

POST /webhooks/payments
  verify signature            reject early if it fails
  INSERT raw event            id, provider, raw_body, received_at, status=RECEIVED
  return 200                  target: under 50ms

Persist the raw body, before parsing. If your parser has a bug, or the provider changes a field, you still hold the original and can reprocess. An event you failed to parse and did not store is an event that is gone.

Verify this step: put an artificial two second delay in your worker and confirm the endpoint still answers in under fifty milliseconds. If acknowledgement time moves when the work gets slower, they are not actually separated.

Step 2. Choose the identity, which is the whole design

Idempotency is not a library. It is one decision: what makes two events the same event.

Use the provider’s own transaction reference, scoped by tenant.

idempotency_key = tenant_id + ':' + provider + ':' + provider_reference

Each part of that is load bearing.

The provider’s reference, because it is the only identifier stable across their retries. It is theirs, it does not change, and it is the same value they will quote when you have to reconcile manually.

Not your own id, which is generated per attempt and therefore different for a retry, which is the exact case you are defending against.

Not a timestamp, for the same reason.

Not a hash of the payload. This one deserves emphasis because it looks correct and fails subtly. Providers reformat payloads: a field order changes, a null becomes an empty string, a numeric becomes a string. The hash changes, the event looks new, and you post it again. It is idempotency that works in testing and fails in production after a provider deploys.

Scoped by tenant, because providers reuse reference formats and two tenants can legitimately produce the same one. Without the tenant in the key, one customer’s payment silently deduplicates against another’s, which means a payment vanishes and the symptom points nowhere near the cause.

Step 3. Claim the key before doing the work

The instinct is a unique index on the reference. Have one, but do not rely on it as the mechanism.

A unique index protects the row and nothing else. The second attempt still runs the balance update, still emits, still sends the receipt, and only fails at the insert. Worse, it fails with a server error, which the provider interprets as failure and answers by retrying harder. Your protection has become the source of the load.

Instead, claim first:

BEGIN
  INSERT INTO processed_events (idempotency_key, status)
  VALUES ($key, 'CLAIMED')
  ON CONFLICT (idempotency_key) DO NOTHING
  RETURNING id
COMMIT

if nothing was returned:
    this event is already handled or in flight, return success, do no work

Now the second arrival does nothing and returns two hundred, which is what the provider needs to stop retrying. The unique constraint underneath remains as the backstop for the race you did not think of.

Verify this step: fire the same event ten times concurrently and assert exactly one ledger entry and exactly one receipt. If your test only fires them sequentially it is not testing this.

Step 4. Commit the effect and the record together

The write to the ledger, the update to the balance, and the transition of the event to a terminal state all commit in one transaction.

BEGIN
  INSERT ledger entry
  UPDATE balance
  UPDATE processed_events SET status='DONE'
  INSERT outbox row for the receipt        -- not a direct send
COMMIT

The outbox row matters. Do not send the receipt inside the transaction. If you call the messaging provider mid transaction and then roll back, you have told a customer about a payment that did not land, and they have the message as proof. Write the intent to send, commit, and let a separate worker deliver it.

Verify this step: kill the process between the ledger write and the commit. The database should show no ledger entry, no balance change, no outbox row, and the event still claimed but not done, so the sweep in step five picks it up.

Step 5. The sweep, which is the part people skip

Everything so far handles duplicates. None of it handles a worker that died.

An event claimed but never completed is invisible: the provider got its two hundred and will not send again, and no error was ever raised. Without a sweep it sits there forever.

every 2 minutes:
  find events claimed more than 5 minutes ago and not DONE
  release the claim and reprocess

Because the work is idempotent, reprocessing is safe. This is what turns the death of a worker from a correctness problem into a latency problem.

Pick the stale threshold above your realistic worst case processing time. Too short and you race a worker that is merely slow. That race is safe, because of step three, but it wastes work and it muddies the logs.

Verify this step: claim an event, kill the worker, and confirm it completes on its own within the sweep interval without anybody touching it.

Step 6. The daily comparison, which catches absence

The fast path cannot detect an event that was never delivered. Absence produces no signal. Only a comparison can find it.

Once a day, pull the provider’s own settlement report and compare it against your records for the same window. Three outcomes:

In both, matching. Nothing to do.
In theirs, not in yours. An event never arrived. Fetch and process it.
In yours, not in theirs. More serious. Either a timing boundary or a payment you posted that they have no record of, and the second one needs a person.

Do not auto correct anything except the second case. Automatic correction hides the rate at which discrepancies happen, and the rate is the signal. A system quietly self healing twenty times a day has a problem nobody will look at until it is a hundred.

Step 7. Make the rate visible

Put the daily discrepancy count on a screen somebody actually sees, trending over time.

The target is not zero on any given day. It is a number trending toward zero rather than sitting at an accepted constant. A constant means you have stopped treating it as a defect and started treating it as weather.

What this costs

More moving parts than a direct handler: a table, a worker, two scheduled jobs. Latency between receipt and effect, usually under a second, which matters if a user is watching a screen and waiting. And you now own a queue or a sweep, which is operational surface that did not exist before.

It is worth it the moment money is involved, because the alternative failure is not an error page. It is a customer being charged twice and finding out before you do.

The test that proves it works

Not a unit test. Replay a full day of production events three times in a row, in a random order, with the worker killed at random points. The ledger must be identical after each run, and the receipt count must equal the payment count.

If that holds, the design is right. If it does not, one of the seven steps above was skipped, and it is almost always step three or step five.

Questions people actually ask

Is a unique index on the reference not enough?: It stops the duplicate row and nothing else. The second attempt still updates the balance, still emits the event, still sends the receipt, and only fails at the very end. It also turns a benign retry into a server error, which most providers read as failure and answer by retrying harder.
What should the idempotency key be?: The provider's own transaction reference, scoped by tenant. Not your own generated id, which differs per attempt. Not a timestamp. Not a hash of the payload, because providers reformat payloads and the hash then changes for the same event, which fails while looking like it works.
How far back should the reconciler look?: The fast sweep needs a couple of minutes to catch crashed workers. The daily pass should cover at least a full settlement cycle, because its job is finding events that never arrived, which the fast path cannot detect by definition.
Do you need a queue for this?: You need somewhere durable to put the event and something that will process it later even if this process dies. A queue is the usual answer. A table plus a sweep is a perfectly good one and has fewer moving parts.