Living with replication lag
Most apps read far more than they write. So writes go to one leader and reads spread over many followers. This only works with asynchronous replication: waiting on every follower means one dead follower stops all writes.
Eventual consistency
An async follower can be behind the leader, so the same query can give different answers on each. If writes stop, followers catch up: this is eventual consistency. "Eventually" has no limit. Lag is often under a second, but under heavy load or network trouble it can reach minutes.
Read your own writes
A user saves a change, reloads, hits a stale follower, and their change seems lost. Read-your-writes promises a user always sees their own changes, not other people's. Ways to get it:
- Read things the user can edit, like their own profile, from the leader.
- Read from the leader for a minute after the user's last write.
- The client remembers the log position of its last write, and only a replica that has reached it may answer.
Across regions, leader reads must go to the leader's region. Across devices, that last-write position must be kept on the server.
Monotonic reads
If a user's reads hit random replicas, a reply can appear and then vanish on refresh. Monotonic reads means a user never sees older data after newer data, though data may still be old. Fix: send each user to one replica, picked by a hash of their user ID.
Consistent prefix reads
If shards replicate at different speeds, a reader can see a reply before the message it answers. Consistent prefix reads means writes are seen in the order they happened. Fix: put related writes on one shard, or track what depends on what.
Choosing stronger guarantees
Ask what users see if lag reaches minutes. App-level tricks are hard to get right. A database with strong consistency and transactions lets you treat it like one machine. Weaker replication still fits when you must ride out network trouble with less overhead.
Remember
- Lag has no upper bound.
- Read-your-writes covers only your own writes.
- Monotonic reads stops time going backward, not staleness.
- Out-of-order reads are mostly a sharding problem.