Measure the tail, not the average

The number that reassured me

A dashboard page averaged around three hundred milliseconds. That is fine. Nobody complains about three hundred milliseconds and I had no reason to look at it.

Then somebody mentioned in passing that the page was “sometimes slow”. Not a ticket. A remark, made while talking about something else.

I looked at the distribution rather than the average for the first time.

p50    180ms
p75    240ms
p90    980ms
p95    2,400ms
p99    6,100ms

The average of three hundred was real and it was describing the wrong requests. Half of all loads were under two hundred milliseconds, which is where the average came from. One request in twenty took over two seconds, and one in a hundred took six.

Why nobody had reported it

This is the part that stayed with me.

Complaining about latency requires a comparison. If the page is always slow, people say the system is slow. If it is usually fast and occasionally slow, people conclude their connection is bad, or that they did something, or that it is just how it is.

Intermittent slowness is under reported to almost zero, and the effect is worse than that: it trains people to distrust the system without ever giving you the information to fix it. They start avoiding the page, or opening it before they need it, and none of that shows up anywhere.

Six thousand milliseconds happening one time in a hundred, on a page loaded a few thousand times a day, is dozens of people a day having a genuinely bad experience. Silently.

What the tail was

It was not one thing, which is typical.

Roughly two thirds of it was one query whose plan changed depending on the parameters. For most customers it used an index. For customers above a certain size the planner switched to a sequential scan, and those customers had that experience every single time. It was not random. It was deterministic per customer, which meant a small number of people were having a consistently bad time while the aggregate looked healthy.

The remaining third was cold cache after deploys, which is real, affects the first users after every release, and is invisible in any average.

What I changed about measuring

Four things, and I now do all of them by default.

Never look at an average alone. The median tells you the typical experience. The ninety fifth percentile tells you the bad one. If they differ by an order of magnitude, that gap is the story and the average is hiding it.

Segment by customer size. An aggregate across a multi-tenant system averages your largest customer with your smallest, and your largest customer is usually your most important. Their experience should be visible on its own.

Measure cold. The first request after a deploy is somebody’s request. If your only measurements are from warmed caches, you are systematically excluding the worst experience your product delivers, and you deliver it after every release.

Watch the tail when making a change. A change that improves the median and worsens the tail has made the system worse for the people already having the worst time. That trade is common and almost always unintentional, because the number on the summary went down.

The organisational version of the same mistake

The same shape appears everywhere once you look for it.

Uptime that averages across a month hides the four hour outage. A support satisfaction average hides the small group having a terrible time. A test suite that passes on average is not a thing anybody would accept, and yet the same logic is applied to latency without objection.

Averages describe the population you already served well. The tail describes the people you are losing.

The thing I would tell myself earlier

Look at the tail before anybody complains, because the people in the tail are the least likely to complain and the most likely to leave.

And when somebody mentions in passing that something is “sometimes slow”, treat it as a report. That sentence is what a tail problem sounds like when a person says it out loud, and I nearly did not follow it up.