Staying available

Parts fail all the time. A good design keeps the app working anyway.

Counting nines

Availability is the share of time the system works. A year has about 8,760 hours.

Each extra nine cuts the allowed downtime to a tenth. If a page needs two services and each is up 99.9%, the page is up only about 99.8%.

No single point of failure

A single point of failure is a part with no spare: if it fails, everything fails. Redundancy means adding a spare that can do the same job. When one part fails, traffic moves to the spare: this is failover. Two servers that each fail 1% of the time, at separate times, are both down only 0.01% of the time.

Circuit breakers

A timeout stops waiting on a slow service after a set time. A circuit breaker watches calls to a service. When too many fail, it opens: calls fail at once without trying. After a wait, it lets one test call through (half-open). If that works, it closes and calls flow again. This stops calls piling up and lets the sick service recover.

Graceful degradation

When a side feature fails, keep the main feature working. If recommendations are down, show the product page without them, or show a fixed list of best sellers. Decide ahead which parts are critical, like payment, and which can be hidden.

Zones and regions

A zone is a separate data center with its own power and network. A region is a group of zones in one part of the world. Several zones survive one data center failing. Several regions survive a whole region failing, but cost more, and syncing data far away is slow.

Remember

• 99.9% is about 8.8 hours down a year; 99.99% is about 53 minutes.
• Find every part with no spare.
• An open breaker fails fast, so the sick service can recover.
• Hide a broken side feature; keep the main one working.