A scheduler holding ten million future jobs on flat memory

The situation

Overnight work had grown the way it always does. One cron entry, then four, then somewhere past twenty across several concerns. Each one a script, each one logging in its own style, none of them able to answer the two questions that actually matter: did this run, and what did it do.

Then a process died partway through a nightly run and the answer to “which half completed” was nobody knows. That is the point where the convenience has become a liability.

The obvious design, and where it stops

The first replacement was a proper queue. Every scheduled job becomes a delayed job in a Redis backed queue, workers pick them up, retries and dead lettering come free. That is one line of configuration in BullMQ and it is genuinely the right answer for a large range of systems.

It stops working for a reason that is obvious in hindsight and invisible while you are building it: a delayed job occupies memory for the entire delay.

Most of the scheduled work in this system was not for tonight. It was for the end of a period, or a date months away. Those jobs sat in RAM doing nothing at all, and the number of them only ever went up. Redis is expensive precisely because it is fast, and using it to store cold data is paying for speed you will not use for eleven weeks.

The decision, routing by time on the way in

Split the future into three tiers at the point work is scheduled:

• Immediate, due within about fifteen minutes, goes straight to the queue. Anything else adds latency for no benefit.
• Hot, due within a day, goes to both the queue and the durable table, so dispatch is fast and the record survives a flush.
• Cold, beyond that, exists only as a row in Postgres. It costs disk and nothing else.

Then a promotion sweep runs hourly, finds cold rows coming due, and moves them into the hot tier. A second sweep handles recurring work. A third catches anything overdue, which is the net for the case where the queue lost something.

The effect is that queue memory tracks the next twenty four hours of work rather than all future work. The table grew past ten million rows and Redis did not notice. That single decision was worth roughly ninety percent of the memory.

What promotion forces you to fix

Promotion means a job can legitimately be enqueued more than once, which means idempotency stops being good practice and becomes a precondition.

The key had to be the thing the work is for, scoped by tenant, rather than the moment it fired. A fired-at key makes every retry look like new work, and in a system that charges people, new work means a second charge.

That was the part that took the time. The routing is a switch statement. Making several dozen kinds of work safe to run twice is the actual project, and it is worth saying plainly because the diagram makes the routing look like the hard part and it is not.

Where it landed

Ten million tasks held in the table. Around a hundred thousand executions an hour sustained. Dispatch latency under a second. No lost jobs.

The number I care about more than any of those is that the answer to “did this run last Tuesday” went from reading logs to reading one row. That is what turns overnight work from a source of anxiety into something you can stop thinking about.

What I would do differently

I would build the durable table first and the queue second. I did it the other way, because the queue works in ten minutes and the table feels like paperwork, and I then spent real effort retrofitting a record onto a system that was already running.

The table is the system. The queue is a delivery mechanism, and delivery mechanisms are replaceable.