At least once is the only honest guarantee

The thing that cannot be sold

Two systems, a network between them. A sends a message. B receives it and acts. B sends back an acknowledgement. The acknowledgement is lost.

A now has to choose. Send again, and B might act twice. Do not send again, and B might never have acted at all.

There is no third option, and no amount of protocol removes the choice, because the failure is that A cannot distinguish “B never received it” from “B received it and the reply was lost”. Those two states look identical from where A is standing, and they always will.

Any system claiming exactly once is doing one of two things: giving you at least once with deduplication somewhere you cannot see, or restricting the guarantee to inside its own boundary where it controls both ends. The second one is real and it evaporates the moment the effect leaves that boundary, which for anything interesting it does.

The reframe that fixes it

Stop trying to make the message arrive exactly once. Make acting on it twice have the same result as acting on it once.

That is a local problem, entirely within your control, and it is solvable. It does not require coordination, consensus, or trusting anybody’s marketing.

Once you have it, the whole anxiety dissolves. Retry freely. Replay a day of events. Kill a worker mid run. None of it matters, because the second run is a no operation.

Identity is the whole of it

Idempotency is not a library and it is not a flag. It is one decision: what makes two operations the same operation.

Get that right and the rest is bookkeeping. Get it wrong and you have a system that looks idempotent and is not, which is worse than one that obviously is not, because you will trust it.

The rules I now apply without thinking:

Use an identifier that is stable across retries. For a webhook that is the provider’s own reference. For a scheduled job it is the thing the work is for, meaning this customer for this period, not the moment the job fired. A fired-at key makes every retry look like new work, which is exactly backwards.

Never use a hash of the payload. This is the trap I want to warn people about most, because it looks correct and it fails late. Providers reformat payloads. A field order changes, a null becomes an empty string, an integer arrives as a string. The hash changes. The event looks new. You act again. Your idempotency worked perfectly in testing and broke silently after somebody else’s deploy.

Include the tenant. In a multi-tenant system, two customers can legitimately produce the same reference. Without the tenant in the key, one customer’s operation silently deduplicates against another’s, and the symptom is that something quietly never happened, with no error, pointing nowhere near the cause.

Claim, do not rely on the constraint

The natural instinct is a unique index. Have one. Do not make it the mechanism.

A unique index protects the row and nothing else. The second attempt still updates the balance, still emits the event, still sends the message, and fails only at the very end. It also fails with an error, which most callers interpret as a failure and answer by retrying harder. Your protection has become the source of the load.

Claim the key first, in its own small transaction, and if the claim is already taken, return success without doing anything.

INSERT INTO processed (key) VALUES ($1) ON CONFLICT DO NOTHING RETURNING id

nothing returned  ->  already handled, return 200, do no work

The second arrival now does nothing and reports success, which is precisely what a retrying caller needs to stop. The unique constraint stays underneath as the backstop for the race you did not anticipate.

The part everybody skips

At least once handles duplicates. It does nothing about the opposite failure: an operation that was claimed and never completed, because the process died in between.

That one is invisible. The caller received its acknowledgement and will not retry. No error was raised. Nothing is in a queue. It simply never finished, and it will sit there forever.

The only thing that finds it is a sweep that looks for work claimed but not completed, and reprocesses it. Because the work is idempotent, reprocessing is safe.

every 2 minutes:
  find claimed more than 5 minutes ago and not done
  reprocess

This is what converts a dead worker from a correctness problem into a latency problem, and it is routinely left out because the happy path works fine without it.

And the one after that

Even the sweep cannot detect an operation that never arrived at all. Absence generates no signal.

For that you need a comparison against an external source of truth, on a slower cycle. A daily reconciliation against the provider’s own statement. That is the only mechanism that can find something that is missing rather than something that is wrong.

Three layers, and each catches what the previous one structurally cannot:

claim key       catches duplicates
sweep           catches crashed workers
daily compare   catches events that never arrived

How to know it works

Not a unit test. Replay a day of real events, three times, in random order, killing the worker at random points.

The resulting state must be identical after every run. The count of side effects must equal the count of distinct operations.

If that holds, you have at least once with idempotent handling, which is the strongest honest guarantee available. If it does not, the failure is almost always the claim step or the missing sweep.

Why this is better than what you wanted

Exactly once, even if you could buy it, would be a guarantee you have to trust. This is a property you can test, on your own machine, in an afternoon.

I would rather have a property I can prove than a guarantee somebody has promised me.