Why adding more workers made it slower

The situation

A nightly recalculation that touched a large number of records. It had been comfortably finishing in its window for a year, and then it stopped finishing, because the number of records had grown the way they do.

The queue depth graph looked exactly like you would expect. Jobs going in faster than they came out, a backlog forming, and the run bleeding into the working day.

The obvious move

Four workers to eight.

This is the correct instinct in most situations and it is what every piece of documentation implies. Queues exist so you can add consumers.

Throughput did not double. It went up by something like fifteen percent, which I noted as disappointing and then rationalised as some fixed overhead.

So I went to sixteen.

Throughput went down. Not flat. Down, by a noticeable margin, and the ninety fifth percentile job duration roughly tripled.

At that point I stopped, because a system that gets slower when you give it more resources is telling you that your model of it is wrong.

What was actually happening

Each job read a set of related records, computed a value, and wrote it back inside a transaction. Two jobs touching overlapping records would contend for row locks.

At four workers, overlap was occasional. At sixteen, overlap was constant. Every worker was spending most of its time waiting for locks held by other workers, and Postgres was doing deadlock detection work on top.

Then there was a second effect I had not considered at all. The connection pool held twenty connections. Each job held one for its whole duration, including while waiting on locks. With sixteen workers plus the web application sharing that pool, requests started waiting for connections.

So adding workers had made the user facing application slower, which was a consequence I had not predicted from a change to a background queue. That was the moment it became genuinely alarming rather than merely disappointing.

The distinction I did not have

There are two kinds of slow queue and they need opposite responses. I did not have this distinction in my head at the time and it is the entire lesson.

Waiting. Workers are idle, blocked on something external. A third party API, a slow network call. An idle worker costs almost nothing, so more workers is free throughput. Adding them works and throughput rises roughly linearly.

Competing. Workers are contending for one shared resource that is already saturated. Locks, connections, disk. Adding workers adds coordination overhead on top of a fixed capacity, so throughput is flat and then negative.

The symptom is identical from outside. In both cases the queue backs up. The response is opposite, and I had guessed.

The measurement that distinguishes them takes twenty minutes and I should have done it first:

2 workers   -> 420 jobs/min
4 workers   -> 610 jobs/min      still rising, but sub linear
8 workers   -> 700 jobs/min      flattening
16 workers  -> 480 jobs/min      falling. past the limit.

Rising roughly linearly means waiting. Flat means at the limit. Falling means every worker you added is making it worse, which is a genuinely counterintuitive thing to see on a graph for the first time.

What actually fixed it

Not workers. Three changes, in this order.

The transaction was too long. Each job opened a transaction, did its computation, and committed. The computation did not need to be inside the transaction at all. Moving it outside cut lock hold time by roughly eighty percent and that single reordering recovered most of the throughput on its own.

BEFORE                          AFTER
BEGIN                           read the data
  read the data                 compute            <- no locks held
  compute        <- locks held  BEGIN
  write                           write
COMMIT                          COMMIT

Batching. Instead of one job per record, one job per five hundred records, with a single bulk write. Five hundred transactions became one. This is where the actual order of magnitude came from.

The trade is real: a batch that fails fails five hundred records. I handled it by splitting a failed batch in half and retrying each half, so one bad record costs a handful of extra passes rather than five hundred failures.

A concurrency cap tied to the real constraint. Six workers, not sixteen, chosen from the connection pool rather than from optimism. And a separate pool for background work so the queue could never again starve the web application, which was the failure that had frightened me most.

The final numbers

The nightly run went from not finishing to finishing in about a fifth of its previous window, on fewer workers than it started with.

That is the part I found hard to internalise. The answer to a backlog was less concurrency and better batching, and I had spent two days going in the opposite direction with complete confidence.

What I took from it

Measure before scaling. Twenty minutes of measurement would have replaced two days of guessing. The measurement is trivial and I skipped it because the diagnosis felt obvious.

Utilisation is not throughput. Sixteen workers at a hundred percent utilisation looked like a system working extremely hard. It was working extremely hard at waiting for each other. Every dashboard I had was green.

Transaction duration is the first lever, and it is free. Before batching, before pooling, before tuning, ask what is being held while something else happens. It costs a reordering and it is very often most of the win.

Watch for effects outside the system you are changing. Making a background queue faster should not be able to slow down the web application, and it did, because they shared a resource I had not modelled. Shared pools are the connection between systems you think are independent.