Half one of two
Engines
How to design the piece of a system that runs when nobody is watching. Written from having built them and operated them, at the level of pattern rather than implementation.
Every page here answers the question in its first forty words, then shows the reasoning. Nothing on these pages describes any client's internals. A drawing of how a reconciliation loop works in principle is craft. A drawing of a specific one is a map, and maps do not go on the internet.
What counts as an engine
The part of a system that does work on its own schedule, with nobody watching it happen. A scheduler firing overnight jobs, a reconciliation pass comparing yesterday against a provider statement, an approval chain moving a request through four people, a rules engine deciding what a fee should be. Nobody sees it run. They see the result the next morning, and they only look closely when it is wrong.
That is what makes engines a different discipline from request handling. A web request that fails is visible immediately and someone retries it. An engine that fails quietly keeps failing, and by the time a human notices, the wrong data has been sitting in the system for eleven days and nobody can say which rows are affected.
Why the same five keep appearing
These five are not a survey of everything a backend can do. They are the ones that recur across unrelated products, because they solve problems that come from the domain rather than the stack. Anything with money needs reconciliation. Anything with more than one person approving needs an audit trail that survives the approver changing role. Anything that grew past one cron entry needs a scheduler.
Each page states the design in the first forty words, then works through what breaks at scale, what the trade off costs, and which failure mode the design is actually buying protection against. They are patterns, not implementations, which is why they hold in a language I have not used.
How to design a scheduler engine that holds ten million future jobs
A scheduler is not a trigger. Cron fires and forgets, which is fine until the process dies mid run, or the same job fires twice, or somebody asks what happened at four in the morning and nothing can answer.
How to design an approval engine that survives an audit
Every system eventually grows an action nobody should be able to take alone. Refunding money, releasing results, deleting a customer, messaging forty thousand people at once. The usual answer is a permission, and a permission is the wrong tool.
How to design a reconciliation engine that assumes the worker died
Every payment integration works in testing. The interesting question is what happens when the provider retries because your acknowledgement was slow, and the customer is credited twice for one payment.
How to design a notification engine with one ledger and real lanes
Notifications look like the easiest thing in a product and become one of the hardest, because they cost money per message, they fail silently, and the person who needed one has no way to tell you it never arrived.
How to design a rules engine, and when not to build one
Somebody asks for a threshold to be configurable. Then a second condition. Then an exception for one group. Six months later you have built a programming language with no debugger, no tests and no error messages.
Where to start
- Most transferable
- Three tier time routing, why exactly once is a promise nobody can sell you, and how to keep memory flat while the table grows into the millions.
- If the numbers have to be right
- Written from the assumption that the worker died halfway. That single assumption changes the whole design.
- If somebody asked to make it configurable
- Including the case for not building one, which is the right answer more often than it is popular.
Common questions
- Can I copy these designs directly?
- The reasoning transfers, the numbers do not. Every threshold on these pages, the twenty four hour promotion window on the scheduler for instance, came from measuring one workload. Copy the structure and the failure analysis, then measure your own traffic before fixing any constant.
- Why are the diagrams generic rather than the real systems?
- Because a diagram of a real system is a map of where its weak points are. The drawings here show how a pattern works in principle, which is the part that is useful to somebody solving the same problem, and none of them describe any client deployment.
- Which one best shows how you work?
- The scheduler page. It has the clearest example of a decision that was expensive to get right and would have been much more expensive to retrofit, which is the kind of call an engineer is actually hired to make.
The other half of the job is the screen people use the engine through. Interface work .