Skip to content

Half one of two

Engines

How to design the piece of a system that runs when nobody is watching. Written from having built them and operated them, at the level of pattern rather than implementation.

Every page here answers the question in its first forty words, then shows the reasoning. Nothing on these pages describes any client's internals. A drawing of how a reconciliation loop works in principle is craft. A drawing of a specific one is a map, and maps do not go on the internet.

What counts as an engine

The part of a system that does work on its own schedule, with nobody watching it happen. A scheduler firing overnight jobs, a reconciliation pass comparing yesterday against a provider statement, an approval chain moving a request through four people, a rules engine deciding what a fee should be. Nobody sees it run. They see the result the next morning, and they only look closely when it is wrong.

That is what makes engines a different discipline from request handling. A web request that fails is visible immediately and someone retries it. An engine that fails quietly keeps failing, and by the time a human notices, the wrong data has been sitting in the system for eleven days and nobody can say which rows are affected.

Why the same five keep appearing

These five are not a survey of everything a backend can do. They are the ones that recur across unrelated products, because they solve problems that come from the domain rather than the stack. Anything with money needs reconciliation. Anything with more than one person approving needs an audit trail that survives the approver changing role. Anything that grew past one cron entry needs a scheduler.

Each page states the design in the first forty words, then works through what breaks at scale, what the trade off costs, and which failure mode the design is actually buying protection against. They are patterns, not implementations, which is why they hold in a language I have not used.

Where to start

Most transferable
Three tier time routing, why exactly once is a promise nobody can sell you, and how to keep memory flat while the table grows into the millions.
If the numbers have to be right
Written from the assumption that the worker died halfway. That single assumption changes the whole design.
If somebody asked to make it configurable
Including the case for not building one, which is the right answer more often than it is popular.

Common questions

Can I copy these designs directly?
The reasoning transfers, the numbers do not. Every threshold on these pages, the twenty four hour promotion window on the scheduler for instance, came from measuring one workload. Copy the structure and the failure analysis, then measure your own traffic before fixing any constant.
Why are the diagrams generic rather than the real systems?
Because a diagram of a real system is a map of where its weak points are. The drawings here show how a pattern works in principle, which is the part that is useful to somebody solving the same problem, and none of them describe any client deployment.
Which one best shows how you work?
The scheduler page. It has the clearest example of a decision that was expensive to get right and would have been much more expensive to retrofit, which is the kind of call an engineer is actually hired to make.

The other half of the job is the screen people use the engine through. Interface work .