Load balancing

Clients only know the load balancer's address, so you can add or remove servers behind it freely.

Picking a server

Rules to pick a server:

• Round robin: each server in turn.
• Weighted round robin: a bigger server gets more turns.
• Least connections: the server with the fewest open connections, good when some requests take much longer than others.
• Hash by user: the same user always goes to the same server; with a plain hash, if the number of servers changes, most users move.

Health checks

Every few seconds, the balancer sends each server a small request, like GET /health. After a few failures in a row, it marks the server down and stops sending it requests. When the server passes again, it gets requests again. A good health check also tests what the server needs, like its database.

Layer 4 and layer 7

A layer 4 balancer sees only addresses and ports. It picks a server once per connection and passes the bytes along. It is simple and fast. A layer 7 balancer reads each HTTP request. It can send /api to API servers and /images to image servers. For HTTPS it must decrypt requests first, so it does more work.

Sticky sessions

Sticky sessions send a user to the same server every time. Teams use them when servers keep sessions in memory. But load gets uneven, and if that server dies, its users lose their sessions. With stateless servers and a shared store, you don't need them.

Two balancers

One balancer is a single point of failure: if it dies, the app is down. So run two: the active one handles traffic, and the standby watches it. If the active one stops answering, the standby takes over its IP address, so clients change nothing.

Remember

• Least connections suits requests of mixed lengths.
• A few failed checks in a row take a server out.
• Only layer 7 can route by path.
• Stateless servers make sticky sessions unneeded.
• Run two balancers, or the balancer is a single point of failure.