Role
Senior Node.js engineer, six years, all of it in production
A background job has been failing quietly for eleven days. It retries, so nothing alerted. Some records are duplicated and some are missing, and nobody knows which. That is the week this role is actually for.
How I fit
I take the part of the system nobody wants to touch. The scheduled work, the payment integration, the thing that was written fast two years ago and is now load bearing. In the first month I would expect to be reading production behaviour rather than shipping features, because the fastest way to be useful in an unfamiliar system is to find out what it actually does rather than what it is documented to do.
A challenge from this role
A background job has been failing quietly for eleven days. It retries on failure, so no alert ever fired. Some customers now have duplicate records, some have none, and nobody can say which is which. The pressure is to fix the job and move on.
How I approach it
First hour, stop the bleeding without destroying evidence. Disable the job, do not delete anything, and take a snapshot, because a repair against an unknown state is unrecoverable. First day, establish the true state before touching it, working out which records are duplicates, which are missing, and which are correct, because the repair cannot be written until that is known. Then the repair, idempotent and run in preview first. Then the actual fix, which is that the work was not keyed on the thing it was for, so every retry looked like new work. Last, the alerting that should have caught it, because a retry that always succeeds eventually is indistinguishable from success.
What proves it
Six, currently as the only engineer on a multi-tenant platform used daily by 112 schools.
Backend and a multi portal frontend, both strict, with permission types generated from the database rather than hand maintained.
Reading the plan before changing the query, measuring the tail rather than the average, and testing at the scale it will meet.
A scheduler holding ten million future jobs at a hundred thousand executions an hour with no losses.
Role and portal scoped access where what a user is served is the intersection of several independent gates.
Deploys and migrations against a live multi-tenant system while customers are using it.
Why this scenario rather than a puzzle
The scenario above is deliberately unglamorous, because it is the shape of the week that actually separates a senior engineer from a strong mid level one.
The mid level answer is to fix the job. It is the obvious move, it is satisfying, and it is third in the correct order. Doing it first makes the data problem permanent, because the moment the job runs correctly over a corrupted state, you have lost the ability to tell what was corrupted.
Sequencing under pressure is the skill. Stop, understand, repair, fix, detect. Everything about that order is counter intuitive when somebody senior is asking how long it will take.
What I would want to know in week one
Not the architecture diagram. Three things: what breaks most often, what nobody wants to touch, and which number the business trusts least.
Those three questions locate the real system, which is always different from the documented one.
Questions people actually ask
- What if our stack is not Node?
- Then the framework list is not what you are buying. The loop transfers, and so does knowing what to be suspicious of. I would expect to be slower for a few weeks and no slower after that.
- Are you looking for contract work?
- No. Permanent and remote. The work I am proudest of took years, which is not a coincidence.