Distributed consensus and multi-region data

Consensus protocols such as Raft and Multi-Paxos choose one ordered log despite node failure. They usually require a majority of voting members to make progress. Put those voters across distant regions and every committed write waits for enough network round trips to reach a quorum. Put most voters in one location and write latency improves there, but a regional outage may remove the majority. There is no placement that gives zero latency, complete partition tolerance and immediate consistency at once.

The useful question is not whether the database is distributed. Ask which failures the voter layout can survive and where the acknowledgement path travels. A three-voter group placed across three sites can survive one voter loss, yet the write path still depends on communication with another site. Five voters permit two failures if a majority remains, but add infrastructure and often wider latency variance. Non-voting replicas can improve read locality and recovery without changing quorum size. Operators need the actual topology, not the product brochure.

Many enterprise workloads do not need concurrent writes everywhere. A record can have a home region selected by tenant, account or geography. Commands for that record travel to its home, while read models and events replicate outward. This arrangement makes ownership plain and keeps the consensus group near the write path. It also supports data-residency rules when the owning record must remain inside an approved location.

Home-region routing has costs. A user far from the home waits for the command round trip. Reassignment is a controlled migration, not a DNS trick. The routing directory becomes important infrastructure and must avoid sending the same writable record to two homes. Still, these are tractable costs. They are often easier than resolving two accepted updates that both spent the same balance or assigned the same scarce resource. Boring ownership beats magical conflict resolution.

The phrase strongly consistent read hides several choices. Does a user need to read their own write after a redirect? Must a report reflect every transaction accepted before a cutoff? Can a product catalogue trail by a few seconds? Linearizable reads contact a current authority or quorum and provide a simple mental model, but they pay network and availability costs. Local replica reads are fast, though they can return an older state.

Session guarantees solve some practical problems without making every read global. A client can carry a commit token and require subsequent reads to reach at least that position. Monotonic reads stop the same user from moving backward in observed history. Bounded-staleness rules can reject a local replica once its lag exceeds an agreed limit. Each guarantee needs telemetry. If the application cannot show replica position and age, it cannot prove that a bounded-staleness promise held during an incident.

An event-driven design can move work out of the synchronous request, but it does not erase ordering or atomicity. A service that commits a database change and then publishes a message has a gap between those actions. If the process fails inside the gap, state and downstream events disagree. The transactional outbox closes that gap by committing the state change and an event record together, then publishing from the outbox with retries.

Consumers should assume duplicate delivery. They can store processed message identifiers, use naturally idempotent operations, or guard state transitions with expected versions. Ordering should be scoped by a business key rather than claimed for the whole stream. A partition carrying all accounts through one ordered lane creates a bottleneck. Partition by account, shipment or another ownership key so related decisions retain order while unrelated work proceeds in parallel. Cross-key workflows need a saga, reservation or reconciliation step, not wishful global order.

Conflict-free replicated data types can merge concurrent changes without a central coordinator when the data has a mathematically defined merge operation. Counters, sets and some registers fit. A business decision usually does not. Two offices approving incompatible outcomes cannot be repaired by choosing the highest timestamp. Wall clocks drift, and the most recent packet is not necessarily the authorised decision.

Use automatic merge where the semantics are genuinely commutative. Keep approval, allocation, settlement and identity changes behind explicit authority or human conflict review. A hybrid model works well: local devices append independent observations, a server merges safe fields, and policy-sensitive conflicts enter an operator queue with both versions and their evidence. That queue is part of the system. Hiding it behind last-write-wins merely loses data more quietly.

Failover changes who may write. That makes it a data protocol, not only an infrastructure action. Promotion criteria should check replication position, quorum health, fencing and the status of the former writer. Without fencing, an isolated primary can keep accepting work while a new primary starts elsewhere. The result is split brain, followed by a painful argument over which accepted history to keep.

A planned regional transfer can drain writes, record a final log position, move authority and verify the new path. An emergency transfer has less evidence, so the business must know its permitted data loss and reconciliation procedure. Recovery objectives become useful only when tied to these steps. Test with real-sized state, blocked links and partial dependency failure. A script that works against an empty staging database says little about restoring the production service.

Regional design should expect variable cross-border latency, provider maintenance and routes that fail asymmetrically. A request may reach another region while the response takes a different, impaired path. Synchronous dependency chains amplify that variance. Keep interactive paths short, cache safe reference data locally, and queue non-interactive work. Timeouts must reflect the user operation and retry budget rather than a copied default.

Measure connections between the locations that actually carry production traffic. Public internet averages do not describe a private link, mobile path or specific provider route. Record packet loss, tail latency and outage duration, then test the consensus and replication design against those observations. Where links are weak, smaller write authorities with asynchronous outward distribution often behave better than a single global transaction stretched across every site.

The final architecture record should name the invariant, write owner, quorum placement, read guarantee, partition behaviour, retry policy and recovery process for each important data class. Include diagrams, but make them executable through tests and telemetry. A design review should be able to ask where a command goes after one site loses connectivity and receive a specific answer.

The hard part is deciding which operations may wait. Once that is explicit, technology choices get easier. Some data stays regional and moves through events. Some accepts local writes with later merge. A small set needs consensus across failure domains. Treat those paths differently. Multi-region systems become understandable when each fact has an owner, each acknowledgement has a meaning, and every failure mode leaves evidence an operator can inspect.