How to fix a queue that is slower after you added workers

First, work out which problem you have

There are only two, and they need opposite responses.

Waiting. Workers spend most of their time idle on network calls, waiting for a third party API, an external service, a slow disk. Concurrency is free here, because an idle worker costs nothing. More workers is correct and throughput rises roughly linearly.

Competing. Workers spend their time contending for a shared resource, almost always the database. Locks, connections, disk. Concurrency is negative here, because the resource is already saturated and every extra worker adds coordination overhead on top of a fixed amount of capacity.

Guessing which one you have is how the mistake happens, because the symptom is identical: a backlog.

The measurement that settles it, and it takes twenty minutes:

run with 2 workers   record jobs completed per minute
run with 4 workers   record again
run with 8 workers   record again

Plot total throughput, not per worker throughput. Rising roughly linearly means you are waiting. Flat means you are at the limit of a shared resource. Falling means you are past it and every worker you added is making it worse.

If you are waiting, the fix is simple

Add workers up to the point where either throughput stops rising or you start straining the thing you are calling.

The one thing to watch: be a good client of whatever you are waiting on. If forty workers all call one third party API, you will hit their rate limit and start getting rejections, which converts a throughput problem into a retry storm. Cap concurrency per external dependency separately from the worker count.

If you are competing, adding workers is the wrong lever

Four fixes, in the order to try them.

Shorten the transaction

The most common cause and the cheapest fix. A job that opens a transaction, calls an external service, and then writes, holds locks for the entire duration of that network call. Ten workers doing that means ten transactions open across a network round trip.

Do the external call first, outside any transaction, then open the transaction, write, and commit.

BAD                              GOOD
BEGIN                            call the API
  call the API   <-- lock held   BEGIN
  UPDATE row                       UPDATE row
COMMIT                           COMMIT

Transaction duration is usually the single biggest lever, and it costs nothing but reordering.

Batch the writes

If a thousand jobs each update one row, you have a thousand transactions and a thousand round trips. One job updating a thousand rows in a single statement is dramatically cheaper.

per item:    1000 jobs  x  1 UPDATE   = 1000 transactions
batched:        5 jobs  x  200 rows   =    5 transactions

The trade is real and worth stating. Batching increases latency for any individual item, and it makes failure coarser, because one bad row can fail the batch. Handle that by splitting a failed batch and retrying the halves, so one bad row costs you a few extra passes instead of five hundred failures.

Batch the bulk lane. Do not batch the lane where somebody is waiting.

Cap concurrency at the real constraint

Work backwards from the scarcest shared resource rather than forwards from the worker count.

If the database connection pool has twenty connections and each job holds one for its whole duration, then more than twenty concurrent workers simply queues inside your own pool. That queue is invisible: the queue dashboard looks healthy while jobs sit waiting for a connection.

Size the workers to the pool, and size the pool to what the database can genuinely serve without its own latency climbing.

Split the lanes

Even at the right total concurrency, one queue means one bulk operation blocks everything behind it. A forty thousand item fan out puts every urgent job behind forty thousand others.

CRITICAL   somebody is waiting on this right now
NORMAL     the default
BULK       fan out, deliberately rate limited, never blocks the others
DEAD       exhausted retries, needs a human

Separate queues, separate workers, separate concurrency limits. The goal for the critical lane is that it is always empty.

The trap that hides all of this

A worker pool that is fully occupied and a worker pool that is fully productive look identical from outside. Both show a hundred percent utilisation.

So measure the work, not the workers:

• Jobs completed per minute, in total. This is the only number that matters.
• Time from enqueue to start. Rising means the backlog is real.
• Time from start to finish, at the median and the ninety fifth percentile separately. If the tail rises while the median stays flat, you have contention.
• Time spent waiting for a connection. If this is above zero, your worker count is above your real capacity and the queue is inside your pool.

That last metric is the one nobody has and the one that answers the question fastest.

The step by step, condensed

1. Measure total throughput at two, four and eight workers. Do not skip this.
2. Rising? You are waiting. Add workers, cap per dependency, stop.
3. Flat or falling? You are competing. Do not add workers.
4. Move external calls outside transactions. Re-measure.
5. Batch the bulk writes, with split on failure. Re-measure.
6. Cap worker concurrency at the connection pool, and size the pool to the database. Re-measure.
7. Split lanes so bulk cannot block urgent.
8. Add the connection wait metric permanently, so the next person does not have to rediscover any of this.

What good looks like

Total throughput rises when you add capacity and stops rising when you hit a resource you can name. The critical lane is empty. The bulk lane is deliberately slow and nobody minds. And when somebody asks why the queue is backed up, the answer is a metric rather than a theory.