How to design a scheduler engine that holds ten million future jobs

Cron is a trigger, not an engine

What most teams call a scheduler is a cron entry pointing at a script. That works until the day one of five things happens, and all five eventually happen.

The process dies halfway through, and nothing knows which half completed. Two instances of the app are running, so the job fires twice. The job fails, and there is no record beyond a log line that rotated away. Somebody asks whether last Tuesday’s run actually happened, and there is no answer. The list of jobs grows to forty and no two of them report success the same way.

Every one of those is a memory problem. Cron has none. An engine is what you get when you add memory, and the design work is deciding what to remember and where.

The three tier time router, and why it matters

The obvious implementation puts every scheduled job into a Redis backed queue with a delay. It is one line with BullMQ and it works beautifully until the number of pending jobs is large.

The problem is that a delayed job occupies memory for the whole delay. A reminder scheduled for three months from now sits in RAM for three months doing nothing at all. At a few thousand jobs this is invisible. At a few million it is the whole hosting bill, and Redis is the wrong place to keep cold data because Redis is expensive precisely because it is fast.

So route by time on the way in:

scheduleTask({ type, payload, runAt, tenantId })
          |
     time router
          |
  +-------+--------+------------------+
  |                |                  |
IMMEDIATE        HOT                COLD
under 15 min     under 24 hrs       beyond 24 hrs
  |                |                  |
straight to      queue AND          Postgres only,
the queue        Postgres           sleeps until due

Immediate work goes straight to the queue, because holding it anywhere else adds latency for no benefit. Work due within the day goes to both, so the queue can dispatch it with low latency while the durable record still exists if Redis is flushed. Everything beyond that lives only in Postgres as a row, costing nothing but disk.

A promotion sweep then runs hourly, finds the cold rows coming due, and moves them up into the hot tier. A second sweep runs every minute for recurring work, and a third catches anything overdue, which is the safety net for the case where the queue lost a job.

The result is that queue memory tracks the next twenty four hours of work rather than all future work. The table can grow to ten million rows and Redis does not notice. That single decision was worth about ninety percent of the memory.

Durability lives in Postgres, not in the queue

The row is the truth. The queue is a delivery mechanism, and delivery mechanisms lose things.

A scheduled task row needs the identity of the work, the schedule, the payload, and the history:

id, tenant_id, task_type
schedule_type     ONE_TIME | RECURRING
scheduled_for     next_run_at, always UTC
cron_expression   plus the source time zone, stored separately
payload           JSON, plus metadata
priority, status, enabled
attempts, max_attempts
last_run_at, last_error

Status moves PENDING to QUEUED to PROCESSING to COMPLETED or FAILED, and every transition is written. That is what lets the system answer the four in the morning question without a human reading logs.

The rule that follows from this: if the queue and the database disagree, the database wins. Flush Redis entirely and the sweep rebuilds the queue from rows. Lose the database and you have lost the system, which is why the durable side is the one that gets the backups.

At least once, and the identity that makes it safe

Nobody can sell you exactly once delivery across a network. The honest guarantee is at least once, and the engineering job is making a second run harmless.

That comes down to identity. The wrong key is the moment the job fired, because a retry has a different moment and therefore looks like new work. The right key is the thing the work is for: this tenant, this subject, this period. Two runs then collide on the same key and the second becomes a no operation instead of a duplicate charge.

In a multi tenant system the key has to carry the tenant, or one customer’s job silently deduplicates against another’s. That is a bug you find in production and never forget.

Lanes, because urgent and bulk cannot share

One queue is one lane, and one lane means a one time password waits behind a bulk send of forty thousand messages. Split by priority and give each lane its own workers:

HIGH     priority 8 and up      time sensitive, user is waiting
NORMAL   4 to 7                 the default
LOW      3 and below            housekeeping
BATCH    bulk fan out           deliberately slow, never blocks the others
DEAD     terminal failures      needs a human

The dead letter lane is not optional. Work that has exhausted its retries has to go somewhere a person will look, because the alternative is that it disappears and the failure is discovered by a customer.

Handlers, so the engine never knows what the work is

The engine should not contain business logic. It owns scheduling, dispatch, retry and recording. Everything else is a handler registered against a task type.

interface TaskHandler {
  validate(ctx): Promise<void>;      // refuse bad work before it runs
  beforeExecute(ctx): Promise<void>; // claim, lock, mark
  execute(ctx): Promise<Result>;     // the actual work
  afterExecute(ctx): Promise<void>;  // record, emit, schedule the next run
}

New work becomes a new handler and a row, not a change to the engine. That is what stops a scheduler from slowly becoming the place all the business logic lives, which is the failure mode every long running system meets eventually.

The lifecycle split matters more than it looks. validate is what lets the engine reject work that can never succeed instead of retrying it three times first. afterExecute is where a recurring job computes its own next run, which means the schedule advances only when the work actually finished.

Multi tenant execution

Scoping is part of the schedule, not an afterthought. A task runs for one tenant, for every tenant of one product, or globally. The engine resolves the tenant set, switches context per tenant, runs the handler inside that scope, and aggregates the outcomes.

The trap is partial failure. Ninety eight tenants succeed and two fail, and if the engine records a single result then the whole run either lies or is retried in full. Per tenant outcomes, retried individually, is the only version that survives contact with real data.

What I would tell someone building this today

Build the record before the queue. It is tempting to start with BullMQ because it works in ten minutes, and the table feels like paperwork. The table is the system. The queue is replaceable.

Then, in this order: make it idempotent, give it lanes, give it a dead letter, and only then make it fast. Every one of those is painful to retrofit and cheap to design in.

And measure the tail. A scheduler with a good average and a bad ninety fifth percentile is a scheduler that is quietly late for a small number of people, every day, forever.