Failover and replication logs
A dead follower just catches up later. A dead leader stops all writes, so another node must take over.
Failover steps
Failover promotes a follower to leader, by hand or automatically. First, detect the failure: there is no foolproof test, so systems use a timeout. Next, choose the new leader, by a majority vote or a controller node. Pick the most up-to-date follower: the synchronous one, or else the one with the highest log position. Last, send writes to the new leader, and turn the old leader into a follower if it returns.
Failover dangers
With async replication, the new leader may miss the old leader's last writes. They are usually discarded, so confirmed writes are lost. This is worse when other systems used them, like a cache keyed by reused IDs. Split brain means two nodes both act as leader; fencing shuts out the old one. A long timeout slows recovery, while a short one causes needless failovers under load. These problems have no easy fix, so some teams fail over by hand.
Statement-based replication
The leader logs each write statement, and followers rerun it. This breaks on NOW() and RAND(), on statements that need the exact same order, and on side effects. The leader can swap in fixed values, but full determinism is hard to guarantee.
WAL shipping
The leader sends its storage engine's write-ahead log, and followers build exact copies. The log describes disk blocks, so all nodes usually need the same version. Upgrades then need downtime.
Logical logs and CDC
A logical log records row changes in its own format. Inserts carry new values; deletes and updates identify the row; a commit record ends each transaction. Being separate from storage, it allows mixed versions during upgrades. Outside systems can read it too, which is change data capture.
Remember
- Promote the most up-to-date follower, and fence the old leader.
- Async failover can lose confirmed writes.
- Statements need determinism; the WAL ties you to one version.
- Logical logs allow mixed versions and feed other systems.