Monitoring

Monitoring tells you what the app is doing and when users are hurting.

Logs

A log is a written record of one event, like a request or an error. Each line has a time, a level (like INFO or ERROR), a message, and a request ID that ties together the lines of one request. Busy apps write a lot of logs, so they are kept only for a while. Never log passwords or card numbers.

Metrics

A metric is a number measured over time. Three key ones:

• Request rate: requests per second.
• Error rate: the share of requests that fail.
• Latency: how long requests take.

Metrics are small, so they are cheap to keep for months.

Percentiles

The average hides slow requests. p50 (the median) means half of requests take this long or less. p99 means 99% take this long or less, so 1 in 100 is slower. With 1 million requests a day, a p99 of 3 seconds means about 10,000 slow requests. So watch p99, not just the average.

Traces

One click can call many services. A trace follows one request across all of them. The first service gives the request a trace ID and passes it on. Each service records a span: one step and how long it took. The slowest span shows where to look.

Alerts and SLOs

An alert calls a person when something needs action. Alert on what users feel, like a high error rate or a slow p99, not on causes like a busy CPU. Too many alerts teach people to ignore them.

An SLO (service level objective) is a target, like "99.9% of requests succeed". The 0.1% left over is the error budget: out of 1,000,000 requests, 1,000 may fail.

Remember

• Logs say what happened, one event at a time.
• Metrics are numbers over time: rate, errors, latency.
• Watch p99, not only the average.
• A trace follows one request by its trace ID.
• Alert on what users feel; an SLO sets the target.