How to design a notification engine with one ledger and real lanes

Why this gets hard

The first version is fifteen lines: a function that takes a phone number and a string and calls a provider. It works. Six months later the same product has messages triggered from eleven places, no record of what was sent, a monthly bill nobody can attribute, and a support queue full of people who say they never received anything.

Notifications are hard for three structural reasons. They cost real money per unit, so waste is directly measurable and directly your fault. They fail asynchronously, long after the code that triggered them returned successfully. And the person affected by a failure is usually not a user of your admin interface, so they cannot report it in any useful form.

One ledger, written before the send

Every message, on every channel, gets one row. The row is written before the send is attempted, not after.

id, tenant_id, recipient, channel
template, payload
dedupe_key            deterministic: recipient + template + subject period
status                QUEUED SENT DELIVERED FAILED SUPPRESSED
provider_reference    once known
cost, attempts, last_error
created_at, sent_at, settled_at

Writing first is what makes the ledger a lock rather than a diary. Construct the dedupe key from the recipient, the template and whatever period the message is about, then insert. If the row already exists, this message has been handled and the send is skipped. A retry storm becomes a series of no operations instead of a series of texts.

One table across channels rather than one per channel, because every question a human asks is cross channel. Has this person heard from us today at all. How many times this week. Which channel actually reached them. A per channel design answers none of those without a join that nobody keeps current.

Status is a state machine, not a boolean

Sent is not delivered. The provider accepting a message means it left your building, and nothing more.

QUEUED to SENT to DELIVERED or FAILED, plus SUPPRESSED for messages deliberately not sent. Delivery receipts arrive later and asynchronously, so the row has to be findable by the provider’s reference, which means storing it the moment it is issued.

Suppressed deserves its own state rather than being deleted. The most useful debugging answer is often “we did not send it, and here is exactly why”, and that is only possible if the decision left a trace.

Lanes, because urgency does not average

The single most consequential design decision here. With one queue, a bulk send of forty thousand messages puts every subsequent message behind forty thousand others. A one time password that arrives four minutes late is not late, it is broken, because the login screen already timed out.

CRITICAL   one time passwords, security, anything a person is waiting on
HIGH       time sensitive but not blocking
NORMAL     the default
BULK       campaigns and fan out, deliberately slow, never blocks the others

Separate lanes with separate workers, and the bulk lane deliberately rate limited. The point is not throughput, it is that the important lane stays empty.

Metering before the send, not after

If a channel costs money, check the balance before dispatch and debit as part of the send, in the same transaction as the ledger update. Debiting afterwards creates a path where a failure has spent money without delivering anything, and reconciling that later is guesswork.

This also gives you the thing every operator eventually asks for: a projection of when the balance runs out at the current rate. That number prevents more outages than any alert, because it turns a sudden failure into a scheduled task.

The floor conditions

Every notification engine needs rules it cannot violate, no matter what a template or a campaign says. They are not features, they are safety.

• Never message somebody the condition no longer applies to. Evaluate at send time, not at campaign creation time, because a person who settled their affairs yesterday must not receive a chasing message today.
• Never let a template decide the recipient list. Templates are text. Audience is a separate decision, made by code that can be tested.
• Respect quiet hours for everything except the critical lane.
• Cap the number of messages one person can receive from one system in a day, and make the cap visible when it fires.

Every one of these exists because its absence produces a message that damages trust more than the message could ever have helped.

What good looks like

A single screen answers: what was sent, to whom, on which channel, what it cost, whether it arrived, and if not, why not. A retry produces no duplicate. A bulk campaign never delays a password. And somebody can be told, with certainty, that a message they say they never received was in fact never sent, which is far more valuable than insisting it was.