Reliability and fault tolerance

Users expect an app to do its job, cope with their mistakes, be fast enough, and block abuse. Reliable means it keeps doing so when things go wrong.

Faults vs failures

A fault is one part going wrong; a failure is the whole system missing its promise (SLO). A single point of failure is a part whose fault downs everything. A dead disk is a failure if it is the only one, but a fault if copies exist. A machine taking over a half-done post delivery must deliver it exactly once (exactly-once semantics).

Limits and testing of fault tolerance

No system survives every fault, only chosen ones like "one of three nodes down". Fault injection (chaos engineering) breaks things on purpose so rare error handling stays tested. Prevent faults with no cure, like stolen data.

Hardware faults and redundancy

Hard drives fail at 2-5% a year: about one a day in 10,000 disks. SSDs fail less but have more uncorrectable errors. About 1 machine in 1,000 has a CPU core that miscalculates. ECC memory still has errors, and some access patterns flip bits. Datacenters go down; a solar storm could wreck power grids and undersea cables. Small systems can mostly ignore this; at scale it is daily life. Redundancy (RAID, spare power) helps, but faults often come together. So cloud systems span availability zones and fail over in software, which also allows rolling upgrades with no downtime.

Correlated software faults

Every node runs the same code, so one bug hits them all. A leap second hung many servers; an SSD firmware bug killed drives at exactly 32,768 hours. Runaway processes, bad dependencies, and cascades do the same. Defend with many small steps: tests, isolation, restarts, monitoring.

Humans and blameless learning

Operator config changes cause more outages than hardware. "Human error" is a symptom of the system around people. Tests, fast rollback, and gradual rollouts help; blameless postmortems let people speak freely. Wrongly trusted software can ruin lives, as UK postmasters found. Short outages are often fine; lost data is not. A prototype may knowingly trade some reliability for cost.

Remember

• Fault is a part; failure is the whole.
• Software faults hit everything at once.
• Fix the system, not the person.