Describing performance
Response time vs throughput
Response time is how long one request takes, as the client sees it. Throughput is work done per second, like 3,000 requests a second. Near the hardware's limit, new requests wait behind earlier ones, and response time shoots up. Users feel response time; throughput sets the server bill.
Overload spirals
Slow responses make clients time out and retry, adding load: a retry storm. A system can stay stuck even after the load drops, until it is reset: a metastable failure. Clients help by waiting longer, with some randomness, between retries (exponential backoff), and by pausing calls to a failing service (a circuit breaker). Servers help by rejecting extra work early (load shedding) and by telling clients to slow down (backpressure).
Where time goes
Service time is time spent actually working on the request. Queueing delay is time spent waiting, for example for a free CPU. Latency is any time the request is not being worked on, including travel over the network. A server works on few requests at once, so slow ones hold up fast ones behind them: head-of-line blocking. The server's timer misses this wait, so measure at the client.
Percentiles over averages
The average helps with throughput sums but hides how many people waited, so sort the times. The median (p50) is the middle one: half are faster. High percentiles like p99 (99 in 100 are faster) show the slow tail. Network delay varies too (jitter). Slow requests often come from users with the most data, often the best customers. Past about p999, delays come from random events, so chasing them costs more than it gives.
Tail amplification and SLOs
A page needing 100 backend calls waits for the slowest, so a rare slow call makes many slow pages. An SLO is a target, like "p99 under 1 s". An SLA is a contract saying what happens if you miss it. Dashboards use compact histograms instead of sorting every time. To combine percentiles from many servers or minutes, add their histograms; never average the percentiles.
Remember
- Measure response time at the client.
- Use p50 for typical, p99 for the tail.
- Retry with backoff; shed load when full.