Solutions
Somebody with a problem does not search for my name. They search for their problem. These pages answer the question in the first forty words, then show the production system where I solved it.
If a page here has no production story behind it, it does not exist. That rule is why there are sixteen of these and not fifty.
What counts as a solution here
Every page answers one question that somebody types into a search box at the point of being stuck. Not a topic, a question. How to make a payment webhook safe to receive twice. Why adding workers made the queue slower. How to change a schema while people are using the system.
The answer comes first, inside the first forty words, because somebody debugging at two in the morning should not have to read an introduction. What follows is the reasoning: the wrong fix that looked right, why it failed, and the fix that held. The wrong turn is included deliberately. It is the part that is hard to fake and the part that tells you whether the writer actually did the thing.
Why these are backend problems with an interface half
Most of these are correctness problems rather than performance problems. Money counted twice, a job that ran again after a retry, a tenant seeing another tenant's row, a migration that locked a table at the wrong moment. Those fail quietly, which is what makes them expensive: nothing alerts, and the damage is discovered by a person rather than a monitor.
The ones that are about speed are usually about the tail rather than the average, because the slowest request is somebody's request and the average hides it.
How to build multi-tenant SaaS where isolation cannot be forgotten
Every multi-tenant system has a tenant filter on every query. It works for years, because people are careful, and then one query written under time pressure exposes one customer to another.
Database per tenant against a shared schema, decided properly
This decision gets made on a feeling about isolation and then lived with for a decade. The variable that actually decides it is what happens on the day you have to change the schema.
How to make a payment webhook safe to receive twice
Every payment integration works in testing. The one that matters is what happens when the provider retries because your acknowledgement was slow, and a customer is credited twice for one payment.
How to fix a queue that is slower after you added workers
The queue is backing up, so you double the workers. Throughput does not double. You double them again and it gets worse. At that point you are not scaling, you are adding contention, and every further worker makes it slower.
How to add realtime without four sockets per browser tab
Realtime features arrive one at a time, each built sensibly by somebody solving their own problem, and the result is four persistent connections per tab and four reconnection strategies that behave differently.
How to make a slow Postgres query fast, in order
The report takes eleven seconds. Somebody suggests an index. Somebody else suggests caching it. Both might help and neither is a diagnosis, and one of them will make the write path slower forever.
How to build role based access that survives five years
Access control starts as three roles and an if statement. It ends as a spreadsheet nobody trusts, where the only safe way to give somebody a capability is to copy an existing user and hope.
How to build an approval workflow an auditor cannot break
An auditor asks who approved a payment in March, under which policy, on what evidence. If the answer requires reconstructing it from a status column and a changelog, the answer does not exist.
How to change a database schema while people are using it
The migration that takes the site down is rarely the complicated one. It is the single line rename that somebody ran on a Friday because it was obviously safe.
When to build an API and when to build the web app
Almost every project eventually asks whether to build the API first. The honest answer depends on one thing, and it is not architectural purity, it is whether a second consumer actually exists.
How to move off a legacy system without a big bang
The rewrite that replaces everything on a chosen weekend is the most reliably disastrous plan in software, and it keeps being chosen because the alternative sounds slower.
How to make a site that ranks, verified against the output
A site can be perfectly built, thoroughly reviewed, and rank as a single page for a year, because the thing that is wrong is only visible in the output and everybody was reading the source.
How to stop one website being four websites
Most sites are quietly served at four or more addresses. The content is identical, the URLs are not, and a crawler treats each one as a separate page competing with the others.
How to build for connections that drop, on devices that are old
The people most in need of a service are frequently the ones on the worst connection to reach it. Designing as though everybody has fibre is not an oversight, it is excluding the users who need you most.
How to ship alone without the wheels coming off
Working alone removes the reviewer, the second opinion, the person who remembers why, and the colleague who takes the call at two in the morning. Each of those has to be replaced by something, or the system slowly becomes unmaintainable while still working.
How to make a system explain itself when you are the only one asking
Observability advice assumes a team, a budget and a platform. Alone, the goal is narrower and more useful: turn every two hour investigation into a two minute query, because you are the one paying the difference.
Where to start
- If money is involved
- Idempotency is the whole game, and the mistake is nearly always keying the work on when it happened rather than what it was for.
- If you are building for more than one customer
- Tenant isolation is the one invariant that cannot be enforced by remembering to add a WHERE clause.
- If the database has to change while people use it
- Expand, migrate, contract, and never a rename. The order is what makes it safe.
- If you are the only engineer
- What you give up, what you must automate first, and what breaks when there is nobody to review the change.
Common questions
- Are these tutorials?
- No. A tutorial shows one path that works. These show the decision, the alternatives that were rejected and why, and the failure each choice is buying protection against. If you want a copyable snippet, the documentation for whichever library you are using is better than anything here.
- Do they depend on a particular stack?
- The examples are Node, TypeScript and Postgres because that is where I met the problems. The reasoning is not stack specific. At least once delivery, tail latency and tenant isolation behave the same way in Go or Python, and the pages are written at the level that transfers.
- Why is there no client architecture on any of them?
- Describing how a specific production system is put together is a map, and a map helps an attacker more than it helps a reader. Every page here is written at the level of the problem and the decision, which is also the level that is useful to somebody working on a different system.
Where two reasonable options exist and the answer is it depends, the reasoning is in decisions .