Measuring latency without lying to yourself

Every latency argument I have witnessed started with two people quoting different numbers, both correct, about different things. ping measures the round trip of an ICMP echo. Your application measures the time between writing a request and reading the first byte of the response — and between those two definitions sits a DNS lookup, a TCP handshake, a TLS handshake, and whatever the server does before it speaks.

The first rule is to measure the thing users complain about. If the complaint is “the page is slow”, an ICMP graph is decoration. If the complaint is “the game lags”, the TLS handshake is irrelevant.

Percentiles, not averages

An average hides exactly the samples people notice. A service that answers in 20 ms for 95% of requests and 3 seconds for the rest has a beautiful average and an angry user. Record every sample, then look at p50, p95 and p99 separately. If p99 is far above p95, you are not looking at load — you are looking at a queue somewhere.

Any number that cannot be reproduced with a timestamp attached is an opinion.

Same clock, same path

Two hosts with unynchronised clocks will happily produce negative durations. Keep NTP running, and prefer timestamps taken on one side of the connection when you can. And when you compare two networks, run the measurement from the same machine at the same time — otherwise you are measuring the weather, not the route.

None of this is exotic. It is just the difference between a graph that explains an outage and a graph that starts an argument.