How to make a system explain itself when you are the only one asking

The goal is narrower than observability

The industry framing assumes a team, a budget and somebody whose job is the platform. Alone, none of that applies, and chasing it produces dashboards nobody looks at.

The useful framing is four questions. Each one, when it cannot be answered, costs a specific and predictable amount of your evening.

1. Is it working right now?
2. Did this specific thing happen?
3. Why did this one request fail?
4. What changed just before it broke?

Answer those four cheaply and you have most of the value of a platform for a fraction of the effort.

Question 1. Is it working right now

Not a dashboard of graphs. One endpoint that checks the things that actually break and says so in words.

GET /health

database          ok       12ms
redis             ok        3ms
queue depth       ok       41 jobs, oldest 8s
last nightly run  ok       03:04, 1,204 processed
payment provider  degraded  last success 6m ago

The rule that makes this useful: check the dependencies you cannot control, because those are the ones that fail for reasons unrelated to anything you did. Checking that your own process is running tells you what you already know if you are reading the response.

Two derived numbers worth including because they answer questions before they are asked: the age of the oldest queued job, which distinguishes a busy queue from a stuck one, and the timestamp of the last successful scheduled run, which is the fastest way to notice that overnight work quietly stopped a week ago.

Question 2. Did this specific thing happen

The one that produces the longest investigations, because background work fails silently by nature. There is no user watching to report a symptom.

Every background job gets a row. Not a log line, a row, because rows can be queried and joined.

job_runs
  id, type, tenant_id, subject_id
  status      QUEUED RUNNING DONE FAILED DEAD
  attempts, started_at, finished_at, duration_ms
  error, result_summary

Now “did the reminder for this customer go out last Tuesday” is one query instead of an evening. The subject id matters as much as the type, because the question is almost always about one customer rather than about the job in general.

The same principle applies to anything consequential: a message sent, a payment posted, a permission changed. If somebody might ask whether it happened, it needs a row rather than a log line.

Question 3. Why did this one request fail

Structured logs with a request id threaded through everything the request touches, including the background jobs it enqueues.

{ "req": "01HX3...", "at": "...", "level": "error",
  "msg": "payment post failed", "tenant": 7,
  "provider_ref": "ABC123", "err": "duplicate reference" }

Two habits make this work in practice.

Log the identifiers, not the objects. A whole serialised object is unsearchable and frequently contains data that should not be in a log file. The reference, the tenant and the subject id are what you actually search on.

Give every error surfaced to a user a correlation id. When somebody sends a screenshot with a code on it, the entire investigation is one query. Without it, the investigation starts with establishing who they are and roughly when it happened.

Question 4. What changed just before it broke

The single most useful correlation in operations, and the cheapest to obtain.

Record every deploy and every configuration change with a timestamp, and put that list next to the error rate. Most incidents follow a change, and the answer is usually visible in ten seconds if the two are on the same screen.

14:02  deploy  a3f91c  invoice rounding fix
14:07  errors  spike on /invoices

That is often the entire investigation. Without the first line it is an hour.

An audit trail is the same idea applied to data: who changed what, when, and what the previous value was, for anything touching money, permissions or customer records. It answers operational questions and it is also what you need the day somebody asks a question you cannot answer from the current state.

What to alert on

Very little, and the discipline is in what you leave out.

Alert on symptoms a user would notice, not on causes. High memory is not an alert. Requests failing is.

alert:      error rate above normal for 5 minutes
alert:      queue oldest job older than 15 minutes
alert:      no successful nightly run by 06:00
alert:      health endpoint failing for 2 minutes
do not:     CPU, memory, disk, unless they are the actual symptom

An alert you have ignored twice is retired, whether or not you admit it. Delete it, because its real effect is teaching you to ignore the next one, and the next one might matter.

The order to build it

1. A health endpoint that checks external dependencies and reports in words.
2. A row per background job, with the subject id.
3. Request ids threaded through logs, and a correlation id on every user facing error.
4. A deploy log next to the error rate.
5. An audit trail on money, permissions and customer records.
6. Four alerts, on symptoms.

That list is perhaps a week of work in total and it is the difference between running a system and being run by it.

The inversion worth stating

I learned this backwards. I treated all of the above as things to add after the feature worked, and paid for it during an incident diagnosed by reading the database directly, at length, at night.

Alone, the system’s ability to explain itself is a feature with higher priority than most product features, because you personally pay every hour it cannot. I would now build the record before building the thing that does the work.