How do distributed databases manage cross-datacenter replication?

Distributed databases handle cross-datacenter replication by synchronizing data across geographically separated nodes while balancing consistency, availability, and latency. This typically involves three key mechanisms: replication strategies, conflict resolution, and network failure handling. The goal is to ensure data remains accessible and consistent even if a datacenter goes offline or experiences delays.

First, replication strategies determine how data is copied. Synchronous replication requires all replicas to confirm writes before acknowledging success, ensuring strong consistency but introducing higher latency (e.g., Google Spanner uses synchronized clocks for global consistency). Asynchronous replication allows writes to propagate in the background, prioritizing speed but risking temporary inconsistencies (e.g., Amazon DynamoDB Global Tables). Many systems use a hybrid approach, like quorum-based replication, where a majority of nodes must confirm a write (e.g., Apache Cassandra’s tunable consistency levels). For example, a developer might configure a quorum of 3 nodes out of 5 to balance speed and reliability.

Second, conflict resolution is critical when concurrent updates occur across datacenters. Techniques include version vectors (tracking update timestamps), last-write-wins (using clocks to resolve conflicts), or application-defined logic (e.g., CRDTs for mergeable data types). For instance, Redis uses Conflict-Free Replicated Data Types (CRDTs) to handle counter or set conflicts without manual intervention. Some databases, like CockroachDB, employ hybrid logical clocks to order events globally even when system clocks drift.

Finally, handling network partitions or outages requires trade-offs. Systems often prioritize availability (AP in CAP theorem) during disruptions, allowing writes to continue locally and reconciling later. For example, Cassandra’s hinted handoff stores updates locally if a replica is unreachable and forwards them once connectivity resumes. Others use multi-leader topologies (e.g., PostgreSQL with logical replication) to let any datacenter accept writes, though this increases complexity. Monitoring tools like health checks and automated failover (e.g., etcd’s leader election) help maintain stability during outages. Developers must configure timeouts, retries, and consistency levels based on their application’s tolerance for stale data versus latency.