How to design a reconciliation engine that assumes the worker died

The shape of the problem

Money arrives from a system you do not control, over a network that drops things, from a provider whose retry policy is not negotiable and is usually more aggressive than you expect.

Three things are true at once and any design that assumes otherwise will fail in production:

• The same event can arrive more than once.
• Events can arrive out of order.
• An event that was sent can never arrive at all.

The first two get most of the attention. The third is the one that costs you a customer, because a payment that was made and never posted means somebody who paid is being chased for a debt.

The fast path is not the design

The instinct is to do the work in the webhook handler. Receive, validate, post to the ledger, respond. It reads cleanly and it is wrong for a specific reason: the provider is holding a timeout open, and your ledger write is not fast enough to promise inside it.

When the acknowledgement is late, the provider assumes failure and sends again. Now two handlers are posting the same payment concurrently.

So split it. Acknowledge as fast as the network allows, having done nothing but persist the raw event. Then process asynchronously.

receive  ->  persist raw event, ack in milliseconds
             |
          enqueue
             |
          worker: parse, resolve account, post, record
             |
          sweep: find anything not in a terminal state, run it again

The acknowledgement is a promise that you have the event, not a promise that you have acted on it. Those are different promises and conflating them is the root of most double posting.

Identity is the whole game

Idempotency is not a library, it is a decision about what makes two events the same event.

Use the provider’s own reference for the transaction, scoped by tenant. It is the only identifier that is stable across their retries, because it is theirs.

Do not use your own generated id, which differs per attempt. Do not use a timestamp, for the same reason. Do not hash the payload, because providers reformat payloads and a reformatted payload is a different hash for the same event, which is the worst possible failure: it looks idempotent and is not.

Scoping by tenant is not optional in a multi tenant system. Providers reuse reference formats, and two tenants can legitimately produce the same reference. Without the tenant in the key one customer’s payment silently deduplicates against another’s, which is a bug that is almost impossible to find from the symptom.

Why the constraint is not the mechanism

A unique index on the reference feels like the answer. It is a backstop and it should exist, but relying on it has two problems.

It only protects the row. The second attempt still runs the balance update, still emits the event, still sends the receipt, and only fails at the end.

And it turns a benign retry into a five hundred, which most providers read as failure, so they retry harder. Your protection has become the cause of the load.

The mechanism is a check and claim before the work: take the key, and if it is already taken, treat this as already done and return success. The constraint sits underneath as the thing that catches the race you did not think of.

The sweep is the engine

The part people skip is the one that matters most, because it is the only part that handles the failure the fast path cannot see.

Run a sweep on a short interval that finds any event not in a terminal state and processes it again. Because the work is idempotent, running it again is safe. This is what recovers from a worker that died between claiming and finishing, and it is why the death of a worker becomes a latency problem rather than a correctness problem.

Then run a slower pass, daily, that compares your records against the provider’s own statement. That is the only thing that catches an event which was never delivered. The fast path cannot detect absence. Only a comparison can.

A reconciler that only runs on receipt is not a reconciler, it is a handler.

Report the difference, do not silently fix it

When the daily pass finds a discrepancy, the temptation is to correct it automatically. Resist that for anything involving money.

Automatic correction hides the rate at which discrepancies happen, and the rate is the signal. A system quietly self healing twenty times a day has a problem that nobody will look at until it is a hundred. Surface it, make somebody acknowledge it, and let the number be visible.

The exception is a discrepancy with exactly one possible explanation and a reversible fix. Everything else is a person’s decision.

What good looks like

You can kill the worker at any point and rerun with no double effect. You can replay a whole day of events and land in the same state. You can answer “did this specific payment post, and if not why not” without reading logs. And the number of discrepancies per day is on a screen somebody looks at, trending toward zero rather than sitting at an accepted constant.

None of that is exotic. It just has to be designed in at the start, because retrofitting idempotency onto a system that already has duplicates means first working out which duplicates were real.