One database writes, ten read: primary and read replicas
One primary writes, read replicas serve reads. In our simulation 10 replicas gave 10.3x the reads, but at 500 ms lag your own post was missing 51.8% of the time.
Video · 60 secondsWatch the short versionSame diagrams and numbers. Tap to play. In this lesson you will learn
- How one primary copies every change to read replicas
- How much read capacity replicas add, and what they cost
- Why you sometimes cannot see your own post (replication lag), and the fixes
Builds on Lesson 11: Failover: active-passive vs active-active 18 min read + 60 s video Next: DNS: from a name to an address

Drafted with AI assistance, then checked and edited by a human engineer · How we write
Short answer
Most apps read far more than they write, so one primary database takes every write and copies each change to read replicas, which serve the reads. In our one-machine simulation, 10 replicas served 5000 reads/s against 488 for the primary alone (10.3x) and p95 read latency fell from 382 ms to 40 ms. The catch is lag: at 500 ms lag, reading your own post back was stale 51.8% of the time.
| The pattern | One primary accepts every write and streams its changes to read replicas; reads are spread across the replicas |
|---|---|
| Read scaling (our simulation) | 0 replicas 488 reads/s, 1 replica 500, 3 replicas 1500, 10 replicas 5000: 10.3x; p95 read latency 382 ms to 40 ms |
| Writes do not scale | Every write still hits the one primary; our model, which charges the primary per replica shipped, cut the write ceiling from 2000 to 1667 writes/s at 10 replicas |
| Stale read-backs (our simulation) | Read back 20 to 1000 ms after writing: 3.2% to 3.6% stale at 50 ms lag, 51.8% to 52% at 500 ms, 0% from the primary; replica count barely matters |
| The fix for your own writes | Read-your-writes: send a user's reads to the primary, or to a replica whose replay position has passed that user's write |
| Synchronous replication | PostgreSQL synchronous_commit = remote_apply waits for replay only on the required selected standbys or quorum; other listed standbys may lag; it costs write latency |
| Two primaries | With last-write-wins, our simulation silently lost 9.5% of writes at 50 ms lag and 37.6% at 500 ms |
Most applications read far more than they write, and a single database server runs out of time for reads long before it runs out of time for writes. Replication splits the job: one primary takes every write and copies each change to read replicas that serve the reads, which buys read capacity at the price of copies that are slightly behind. A common interview question asks exactly this: how do you scale reads, and what breaks when you do?
Prerequisites
Throughput, latency and queues. Throughput is how many requests a server finishes per second; latency is how long one request takes, including time spent waiting in line. When a server is busy, latency climbs even though each query is as fast as before. We worked through this in fast alone, slow in a crowd.
Copies side by side. Copies that can each do the whole job let you split traffic and survive a failure: the parallel case from two servers in a row. App servers are easy to copy. A database is not, because every copy must hold the same, constantly changing data.
Commits and the log. A commit is the moment a database promises a change is durable and visible. It keeps that promise by writing the change to a log on disk before replying. Replication reuses the log: another server that replays it in the same order ends up with the same data.
Definitions
Primary: the one database server that accepts writes from the application and decides their order. Older documentation calls it the master.
Replica: a server that receives the primary's changes and applies them, so it holds a copy of the data that is the same or slightly behind. A read replica serves read-only queries. PostgreSQL calls it a standby; older documentation calls it a slave.
Write-ahead log (WAL): the ordered record of every change, written before the change is applied to the data files. MySQL's equivalent for replication is the binary log (binlog).
Log sequence number (LSN): a position in the write-ahead log. Every commit has one; a replica's replay LSN tells you exactly which changes it has applied.
Asynchronous replication: the primary confirms a commit to the application without waiting for any replica. Fast for writes, but replicas trail behind and a crash can lose recent commits.
Synchronous replication: the primary waits for one or more replicas to acknowledge a change before it confirms the commit. What "acknowledge" means (received, written, flushed or applied) depends on the setting.
Replication lag: how far a replica is behind the primary, measured in time or in bytes of log not yet applied.
Stale read: a read that returns data older than a write that has already been confirmed. A stale read-back is the case where you read your own confirmed write and it is missing.
Read-your-writes consistency: the guarantee that after you write something, your own later reads always include it. Other users may still see the older value for a moment.
Multi-primary replication: two or more servers each accept writes and copy them to each other. Older documentation calls it master-master. It needs a rule for conflicting writes to the same row.
How it works: one log, many copies
The primary handles a write the usual way. PostgreSQL modifies pages in shared buffers and generates WAL records while the statement runs. With normal durable commit settings, it flushes WAL through the commit record before acknowledging the commit. Dirty data pages can reach disk later, during checkpoints or background writes, but their corresponding WAL must reach durable storage first (PostgreSQL docs: WAL). Replication adds one step: a walsender process on the primary streams new WAL records to each connected replica's walreceiver.
On each replica three things happen, and they happen at different speeds:
- Receive and write. The replica receives the log records over the network and writes them to its own log files.
- Flush. It forces those log files to disk, so the change survives a crash of the replica.
- Apply (replay). A replay process reads the log and changes the replica's data pages. Only now can a query on the replica see the change.
PostgreSQL exposes each stage separately. On the primary, the view pg_stat_replication shows, per replica, sent_lsn, write_lsn, flush_lsn and replay_lsn, and the matching times write_lag, flush_lag and replay_lag. The gap between flush and replay matters later: a replica can have your change safely on disk and still not show it to a query.
Read those time columns carefully. They measure how long recent WAL took to be written, flushed and replayed after the primary generated it, from the standby's acknowledgements. They are not an estimate of how long the replica needs to catch up, and once a standby is fully caught up and the primary goes idle, they eventually show NULL (PostgreSQL docs: pg_stat_replication). For backlog, use the byte gap pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) and divide by the replica's measured net replay rate.
This is physical streaming replication: the replica applies physical WAL redo records, including full-page images when required, so it is a full copy of the server. MySQL replicates through the binary log instead; with the default row-based format, the replica stores row changes in a relay log and an applier thread executes them. Either way there is one ordered stream of changes, so every replica moves through the same states as the primary, just later.
The application side is a routing decision. Writes, and reads inside a transaction that also writes, go to the primary. Plain reads go to a pool of replicas through a proxy, a connection pooler or the app's own data layer. A feed is the classic case: as in designing a news feed, one post is written once and loaded many times.
How much do read replicas add, and what do they cost?
In our lab's model the read side scales almost linearly. The model assumes each node is one serialized worker: it finishes one request before starting the next, so a node's capacity is exactly 1000 / t_read reads per second. Real servers run many queries concurrently across cores, so the formula describes the model's arithmetic and says little about a real server. Let t_read be the time a node spends on one read and R the number of replicas. If every read goes to a replica, the read ceiling is:
read ceiling ≈ R × (1000 / t_read) reads per second (t_read in ms)
= R × (1000 / 2) with t_read = 2 ms (our lab's assumption)
= R × 500
R = 1 → 500 reads/s
R = 3 → 1500 reads/s
R = 10 → 5000 reads/s
The first replica is the exception. With no replicas, the primary serves reads and writes, so it already gives you close to one node's worth of reads. In this model, the primary already spends nearly all its service capacity on reads, so moving reads to one identical replica adds little throughput: 0 replicas served 488 reads/s and 1 replica served 500. Production gains depend on concurrency, contention, replay work and resource saturation. If reads contend with writes for locks, buffer cache, CPU or disk on the primary, one replica can materially improve throughput by removing that contention. Keeping the primary in the read pool is another way to add the extra node's worth.
The write side does not scale at all. Every write still lands on the one primary, and the primary now has extra work per write: sending it to every replica. Our lab models that as 0.01 ms of primary time per replica per write, on top of 0.5 ms for the write itself:
write ceiling = 1000 / (t_write + R × t_ship) writes per second
R = 0 → 1000 / 0.50 = 2000 writes/s
R = 1 → 1000 / 0.51 ≈ 1961 writes/s
R = 3 → 1000 / 0.53 ≈ 1887 writes/s
R = 10 → 1000 / 0.60 ≈ 1667 writes/s
The falling ceiling follows from the model's assumption; we did not measure it on a production system. Direct replicas add WAL sender and network work on the primary (a sender process, bandwidth and WAL reads per replica). Their effect on write throughput depends on the bottleneck: offloading reads can improve writes, because the primary no longer spends CPU, buffer cache and disk on read queries, while saturated replication resources can reduce throughput. PostgreSQL's cascading replication, where a replica feeds other replicas, keeps that fan-out off the primary.
Replicas pay a hidden cost too: each must replay every write, and replay competes with queries for CPU, I/O and cache. Under heavy writes it can consume most of a replica's capacity. Our lab applies changes for free, so its read numbers are an optimistic ceiling; measure replica read capacity with replay active under your real write mix.
A production budget (assumed numbers). The lab leaves out bandwidth, disk and replay time, so size those from your own measurements. Take a primary that generates 20 MB/s of WAL at peak (measure it by sampling pg_current_wal_lsn() twice and taking pg_wal_lsn_diff over the interval):
replication egress, 10 direct replicas: 10 × 20 MB/s = 200 MB/s from the primary, before protocol overhead
WAL retained for a replica down 30 min: 30 × 60 s × 20 MB/s = 36,000 MB ≈ 36 GB
catch-up after it returns: replays at 30 MB/s while new WAL keeps arriving at 20 MB/s
net catch-up rate = 30 - 20 = 10 MB/s
36,000 MB / 10 MB/s = 3,600 s ≈ 1 hour still lagging
The primary's network card and the WAL sender processes must carry R × WAL rate; cascading replication cuts that by letting one replica feed the others. The primary's WAL disk needs headroom for the longest outage you want a slot to survive: budget about 36 GB for a 30-minute outage at 20 MB/s, plus operational headroom. Physical slots share one pg_wal directory and retain overlapping segments, so retention is governed by the oldest required restart_lsn across slots, rather than the sum of individual backlogs; ten replicas down for the same 30 minutes still need about 36 GB, not 360 GB (PostgreSQL docs: pg_replication_slots). And if a replica's replay throughput is only slightly above the WAL rate, catch-up time explodes: at 21 MB/s replay the same backlog takes 36,000 s, ten hours. Replay in PostgreSQL is a single startup process, so measure replay throughput under your real write mix rather than assuming it matches the primary.
Latency follows from queueing. With a fixed number of clients that send the next request as soon as the last one returns (a closed system), Little's law gives the average response time as clients divided by throughput:
0 replicas: 200 clients / (488 + 50) requests/s ≈ 0.372 s ≈ 372 ms average (p95 measured 382 ms)
10 replicas: 200 clients / (5000 + 552) requests/s ≈ 0.036 s ≈ 36 ms average (p95 measured 40 ms)
Each read still takes 2 ms; the waiting line got shorter because ten servers share it.
Why can't you see your own post?
Asynchronous replicas are behind by the lag. If you read from a replica sooner than the lag after writing, your own post seems to vanish. Nothing is lost, the primary has it, but to you the app looks broken.
The chance of a stale read-back is the chance that your read arrives before the change does. In our lab the read comes 20 to 1000 ms after the write, uniformly spread, so:
P(stale) ≈ P(gap < lag) = (lag - 20) / (1000 - 20)
lag 500 ms: (500 - 20) / 980 ≈ 49.0%
lag 550 ms: (550 - 20) / 980 ≈ 54.1% (top of the ±10% jitter)
measured: 52% (1 replica), 51.8% (3), 51.8% (10)
lag 50 ms: (50 - 20) / 980 ≈ 3.1%
lag 55 ms: (55 - 20) / 980 ≈ 3.6%
measured: 3.2% (1 replica), 3.6% (3), 3.3% (10)
The measured values sit between the plain and the top-of-jitter estimates because the lab, like real replication, applies changes in order: a change cannot overtake an earlier one that drew a longer delay, so many changes arrive near the slow end of the jitter.
The formula has no term for the number of replicas. Every replica runs the same lag, so adding replicas changes which one you hit while each stays just as far behind. They add capacity and do nothing for staleness.
Reading from the primary gave 0% stale read-backs, because the node that confirmed the write answers the read. Every read-your-writes technique builds on that:
- Read your own data from the primary. Always for pages showing the user's own profile or posts, or for a few seconds after a write; the time window is best-effort, because lag can outlast it.
- Track a log position. After a write, remember its commit LSN (PostgreSQL:
pg_current_wal_lsn()right after the commit, or MySQL's GTID of the transaction). For that user's next reads, use only replicas whose replay position has passed it, and fall back to the primary if none has. Receiving the change is not enough; the replica must have applied it. - Wait on the replica. MySQL has
WAIT_FOR_EXECUTED_GTID_SET(), which blocks on a replica until a given set of transactions has been applied, with a timeout. PostgreSQL applications can pollpg_last_wal_replay_lsn()on the standby.
A related guarantee is monotonic reads: once you have seen a value, you never see an older one. Round-robin across replicas with different lag breaks it (a comment appears, then disappears on the next refresh). Pinning a session to one replica, or tracking the highest position it has seen, fixes it. Pinning holds only while that replica stays available and its replay keeps moving forward; after a replica restart, validate the new target against the session's highest seen position before reading from it. After a failover, a bare number is not enough. LSN comparisons are valid only within compatible WAL history: a promoted standby starts a new timeline, and under asynchronous replication that timeline can advance past the session's LSN without ever containing the acknowledged write. Store cluster identity and timeline information with the token, and validate timeline ancestry through the failover controller. If promotion discarded the required write, report the consistency guarantee as unavailable; a numerically larger LSN cannot restore it, and falling back to the new primary cannot recover it (PostgreSQL docs: timelines).
Does synchronous replication fix it?
Only at one specific setting, and only on the standbys that commit actually waited for. PostgreSQL spells this out. synchronous_standby_names lists which standbys count as synchronous (for example FIRST 1 (replica1, replica2) or ANY 2 (...)), and synchronous_commit decides what the primary waits for before it confirms a commit:
| synchronous_commit | Primary waits until the synchronous standby has | A read on that standby right after commit |
|---|---|---|
| off | Nothing (not even a local flush) | Can miss the write |
| local | Nothing; only the primary's own disk flush | Can miss the write |
| remote_write | Received the change and written it to its operating system | Can miss the write |
| on (the default) | Flushed the change to its disk | Can miss the write |
| remote_apply | Applied the change, so queries can see it | Sees the write |
The word "synchronous" misleads here. With on, the commit is durable on two machines, which protects against losing data when the primary dies, but the standby may still be replaying, so a query there can be stale. Only remote_apply closes the gap. Without any synchronous standby configured, the remote levels give only the primary's own local flush, the same as on with no standby.
Each step down that table costs write latency: at least one network round trip to the standby plus its write, flush or apply time, and if the synchronous standby stalls, commits stall with it. Pay that only where it matters: synchronous_commit can be set per transaction with SET LOCAL, so a "publish post" transaction can use remote_apply while bulk logging stays asynchronous. Even then, coverage is narrower than the list in synchronous_standby_names. remote_apply waits for replay on the required selected standbys or quorum: with FIRST 1 (replica1, replica2) commit waits for one selected standby, and with ANY 2 (...) it waits for any two, so another listed standby can remain behind. Other listed standbys and all asynchronous replicas may lag (PostgreSQL docs: synchronous_standby_names). Check the target replica's replay LSN before routing a consistency-sensitive read there, and acquire a fresh query snapshot afterward (a new statement, or a new transaction under REPEATABLE READ), because a snapshot taken before replay reached your LSN will not show the change.
MySQL's semisynchronous replication is similar in spirit: the source waits until at least one replica has received and logged the transaction, not applied it, so reads on that replica can still miss it.
What about two primaries?
If one primary limits writes, why not two? Multi-primary replication lets each server accept writes and copy them to the other, which helps with far-apart regions or writing through a network split. The new problem is conflicts: both primaries can accept a write to the same row before either hears about the other's.
The simplest rule is last-write-wins (LWW): the newer timestamp wins and the other version is discarded. Both servers end up with the same value, so the data looks consistent, but a write the application was told had succeeded is silently gone.
Our lab ran two primaries taking 400 writes/s for 10 s against the same 100 keys, copying to each other after 50 ms and then 500 ms, resolving conflicts with LWW. Each write remembered the version it was built on, so at the end we could follow the chain back from each key's final value and count confirmed writes that were not on it:
lag 50 ms: 3943 writes, 374 lost → 374 / 3943 ≈ 9.5%, diverged keys 0
lag 500 ms: 4114 writes, 1548 lost → 1548 / 4114 ≈ 37.6%, diverged keys 0
"Diverged keys 0" is the uncomfortable part: the servers agree, nothing reports an error, and over a third of the writes vanished at half a second of lag. The lab had perfect clocks; real clock skew makes LWW pick wrong winners more often.
That loss comes from the LWW policy, not from having two primaries as such. Other multi-primary designs do better: MySQL Group Replication in multi-primary mode certifies each transaction and rolls back the one that conflicts, so the application gets an error it can retry; some systems merge values with data types designed for concurrent updates (counters, sets); and many teams give each key one home primary, so concurrent writes to the same row never happen. The lost-update problem is the same one from two fans clicking one seat, just spread across two servers instead of two transactions.
Worked example
The video uses one creator posting and a crowd of phones loading the post. We ran that shape as a discrete-event simulation on one machine (labs/sd-12, virtual clock, seed 12, Node.js v22.13.1). These are not real cluster numbers. Assumptions: 2 ms per read and 0.5 ms per write on any node, plus 0.01 ms of primary time per replica per write; 200 busy clients sending 90% reads and 10% writes; reads round-robin across replicas (the primary serves reads only when there are no replicas).
replicas reads/s writes/s p95 read ms write ceiling/s
0 488 50 382 2000
1 500 51 400 1961
3 1500 164 134 1887
10 5000 552 40 1667
speedup, 10 replicas vs primary alone: 5000 / 488 ≈ 10.25 → 10.3x
1 replica vs primary alone: 500 vs 488 → no real speedup
stale read-backs (write, read the same row 20 to 1000 ms later)
lag primary only 1 replica 3 replicas 10 replicas
50 ms 0% 3.2% 3.6% 3.3%
500 ms 0% 52% 51.8% 51.8%
Writes per second rise (50 to 552) even though the write ceiling falls, because clients finish reads sooner and return with the next write sooner. At 0.6 ms each, 552 writes/s keep the primary only about 33% busy (552 × 0.6 ms ≈ 331 ms of work per second).
Code
Run the lab. It needs only Node.js 22 and the standard library; time is simulated, so it finishes in a few seconds and prints the same numbers every time.
# from the repository root
node labs/sd-12/run.mjs
It prints the three tables above and a final RESULT {...} line with every metric as JSON. Try editing CFG at the top of run.mjs: raise shipMsPerReplica and the write ceiling falls faster; set stale.lagsMs to [5, 2000] and the stale rate tracks the lag; set stale.gapMinMs to 1000 and stale reads at 500 ms lag drop to zero, which is why slow demos hide the problem.
A read-your-writes router. This Python sketch has no real database. It shows the log-position technique: each write returns its position, the session remembers it, and reads go only to replicas that have applied up to that position.
# ryw_router.py: read-your-writes routing with log positions (a sketch, no real database)
class Primary:
def __init__(self):
self.lsn = 0 # position of the last change in the log
self.log = [] # (lsn, key, value), in commit order
self.data = {}
def write(self, key, value):
self.lsn += 1
self.log.append((self.lsn, key, value))
self.data[key] = value
return self.lsn # commit position, handed back to the app
class Replica:
def __init__(self, name):
self.name = name
self.applied = 0 # replay position: changes up to here are visible
self.data = {}
def catch_up(self, primary, upto):
for lsn, key, value in primary.log[self.applied:upto]:
self.data[key] = value
self.applied = lsn
class Router:
def __init__(self, primary, replicas):
self.primary, self.replicas = primary, replicas
self.rr = 0
def write(self, session, key, value):
session["min_lsn"] = self.primary.write(key, value)
def read(self, session, key):
need = session.get("min_lsn", 0)
n = len(self.replicas)
for i in range(n): # round-robin, skipping replicas behind this session
r = self.replicas[(self.rr + i) % n]
if r.applied >= need:
self.rr = (self.rr + i + 1) % n
return r.name, r.data.get(key)
return "primary", self.primary.data.get(key)
primary = Primary()
replicas = [Replica(f"R{i}") for i in range(1, 4)]
router = Router(primary, replicas)
writer_session, reader_session = {}, {}
router.write(writer_session, "post:1", "hello")
print("writer, right after write: ", router.read(writer_session, "post:1"))
print("reader, same moment: ", router.read(reader_session, "post:1"))
replicas[2].catch_up(primary, primary.lsn)
print("writer, after R3 applied it:", router.read(writer_session, "post:1"))
print("writer, again: ", router.read(writer_session, "post:1"))
for r in replicas:
r.catch_up(primary, primary.lsn)
print("writer, all caught up: ", [router.read(writer_session, "post:1")[0] for _ in range(3)])
What to observe: right after the write, the writing session's read goes to the primary and returns 'hello'. Another session, which has written nothing, is sent to R1 and gets None; that is ordinary staleness, and for someone else's post a short delay is usually fine. Once R3 has applied the write, the writing session's reads go to R3 only. When every replica has caught up, it is back in the normal round-robin: ['R1', 'R2', 'R3'].
In production, min_lsn would live in the session or a signed cookie. A time window ("read from the primary for 5 seconds after a write") provides only best-effort freshness: lag can exceed any fixed window during overload, paused replay or a network failure. For a guarantee, retain the required LSN and validate each chosen replica against it, falling back to the primary when necessary. Make the token (cluster id, timeline, LSN), not the LSN alone: LSN comparisons are valid only within compatible WAL history, so after a failover the router must confirm through the failover controller that the new timeline descends from the one that issued the token at or beyond that LSN. If promotion discarded the required write, report the consistency guarantee as unavailable rather than serving a read; a numerically larger LSN on the new timeline cannot restore it. Do not discard the token merely because several seconds elapsed; it costs nothing once every replica has passed it.
The same check in PostgreSQL. These are real functions and columns; run the first two on the primary and the third on a replica.
-- replication.sql
-- On the primary: how far behind is each replica, in bytes and in time?
SELECT application_name,
pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_bytes_behind,
write_lag, flush_lag, replay_lag
FROM pg_stat_replication;
-- On the primary, right after your write commits: remember this position
-- together with the cluster identity and timeline it belongs to.
SELECT pg_current_wal_lsn() AS lsn,
(pg_control_system()).system_identifier AS cluster_id,
(pg_control_checkpoint()).timeline_id AS timeline;
-- On a replica: has it applied up to that position yet?
-- Valid only if the replica is on the same cluster and on a timeline that
-- descends from the token's timeline at or beyond its LSN (check through the
-- failover controller after any promotion). A larger LSN on a diverged
-- timeline does not contain the write.
SELECT pg_last_wal_replay_lsn() >= '0/3000148'::pg_lsn AS has_my_write,
now() - pg_last_xact_replay_timestamp() AS time_since_last_replay;
-- Pay for "visible on the synchronous standby" only on the writes that need it.
BEGIN;
SET LOCAL synchronous_commit = remote_apply;
INSERT INTO posts (author_id, body) VALUES (42, 'hello');
COMMIT;
Replace '0/3000148' with the value your write returned. One caution on time_since_last_replay: on a quiet system it grows even when the replica is fully caught up, because nothing new has arrived to replay, so alert on bytes behind, not on that number alone. Likewise, replay_lag in pg_stat_replication can be NULL; that means no timing measurement is available, and an idle, caught-up standby is only one reason. Do not alert on NULL lag alone, and do not treat it as proof of health either: confirm the expected replica is connected (its row is present), its replay byte gap is within budget, and its status reports are recent (reply_time) before classifying it as healthy (PostgreSQL docs: pg_stat_replication).
Failure modes and edge cases
Lag spikes. A batch update, a schema change, a slow disk or a long transaction can push one replica minutes behind while the others are fine. Monitor bytes behind per replica (the primary's current LSN minus its replay_lsn) together with replay_lag, whether replay is still advancing between samples, and replicas missing from pg_stat_replication entirely (a disconnected walreceiver shows as an absent row, not a large number). Pull a replica from the read pool past a threshold. Our lab used fixed lag; real spikes make stale reads worse.
Failover loses recent commits. If the primary dies under asynchronous replication, confirmed commits not yet shipped are gone when a replica is promoted. Configuring synchronous standbys is necessary but not sufficient. Zero-loss failover requires promoting a surviving node that contains all acknowledged commits: under ANY 2 (a, b, c), a commit may be on a and b only, so promoting c loses it, and membership changes widen the gap. Enforce candidate eligibility and WAL-position checks through the failover controller (compare each candidate's received and replayed LSN, promote the most advanced eligible one), fence the old primary, and document the tolerated failure set. synchronous_commit = on alone is not a complete failover policy (PostgreSQL docs: Synchronous Replication).
Split brain. If the old primary returns still acting as primary after a replica was promoted, you have two primaries by accident and no conflict rule. Failover tooling must fence (cut off) the old primary first.
Query conflicts on the replica. In PostgreSQL, replay can conflict with running standby queries in several ways: vacuum on the primary removing row versions a standby snapshot still needs, exclusive locks (for example DROP TABLE or ALTER TABLE), dropped tablespaces or databases, and buffer pins. The standby delays replay (lag grows) up to max_standby_streaming_delay, then cancels the query. hot_standby_feedback addresses only the cleanup conflicts, by holding back vacuum on the primary, which can bloat tables; it does nothing for lock or tablespace conflicts. Count cancellations by cause in pg_stat_database_conflicts on each standby (PostgreSQL docs: Hot Standby conflicts).
A dead replica fills the primary's disk. A replication slot makes the primary keep every WAL segment its replica has not consumed. A replica gone for days can fill the disk and stop the primary. Set max_slot_wal_keep_size and alert on retained WAL per slot (pg_replication_slots.wal_status and safe_wal_size). The cap has a price: once a slot exceeds it, the primary can remove WAL the replica still needs, the slot becomes lost, and the replica must recover from a WAL archive or be rebuilt from a fresh base backup (PostgreSQL docs: max_slot_wal_keep_size). Size it from the budget above: 36 GB covers 30 minutes at 20 MB/s.
Replica reads inside a write flow. Writing a row, then reading it back through the default replica connection, fails only under load. Keep a write and its read-back on the primary. Likewise, a check-then-act race (is this seat free?) gets a window as wide as the lag if the check runs on a replica.
Trade-offs
| Option | Good for | Costs you | Use when |
|---|---|---|---|
| One primary, no replicas | Simplicity, no staleness | Reads and writes share one server; no standby for failover | Small apps, or before read load is the bottleneck |
| Primary plus async read replicas | Read-heavy workloads, cheap read scaling | Replication lag, stale reads, possible loss of recent commits on failover | Most reads tolerate being a second old |
| Read-your-writes routing | Users always see their own changes | Routing logic, some reads back on the primary | With any async replicas and user-generated content |
| Synchronous standby (on) | Confirmed commits durable on the required standbys | One network round trip per commit; stalls if the standby stalls; reads there can still be stale; zero loss on failover also needs a controller that promotes a node holding every acknowledged commit | Data you cannot lose, with a standby close by |
| Synchronous standby (remote_apply) | Reads on the standbys commit waited for see the write | Highest commit latency; waits only for the required selected standbys or quorum, other listed standbys may lag | Few, important writes that must be readable on specific replicas right after commit |
| Multi-primary with last-write-wins | Writes in several regions, writing through a split | Silent loss of concurrent writes; clock skew | Data where losing an overwritten value is acceptable |
| Multi-primary with conflict detection, or one home primary per key | Several write locations without silent loss | Rejected transactions to retry, or routing by key | Multi-region writes to data that matters |
Common mistakes
In an interview
- Clarify: what is the read to write ratio, and which reads must see the latest data (your own profile, balances, stock) versus which can be a second old (feeds, counts, search)?
- Assumptions: state numbers, for example 50,000 reads/s and 2,000 writes/s, a replica serving about 5,000 reads/s, lag usually under a second.
- Structure: one primary for writes, a pool of async read replicas behind a router, a cache in front for the hottest keys, a synchronous standby for failover.
- Deep dive: lag and stale reads: explain read-your-writes (primary for a user's own reads for a few seconds, or LSN tracking), monotonic reads (pin a session to a replica), and what each synchronous level really guarantees.
- Trade-offs: replicas do not scale writes and add failover and monitoring work; synchronous modes trade write latency for freshness or durability; multi-primary needs a conflict rule.
- Wrap up: monitoring of lag per replica, automatic removal of lagging replicas, fenced failover, and sharding as the next step when the primary's writes hit the ceiling.
Most apps are read-heavy, so I would keep one primary for all writes, add asynchronous read replicas that replay its log, and spread plain reads across them. Read capacity grows with the replica count; in a small simulation, ten replicas served about ten times the reads and p95 latency fell from 382 to 40 ms. Writes do not scale: they all still hit the primary, which also ships each change to every copy. The catch is lag: right after you post, a replica may not have it yet, and more replicas do not help. So I would send a user's reads to the primary for a short window after their writes, or to a replica whose replay position has passed their write. Synchronous replication only fixes this at remote_apply, at a cost in write latency. With two primaries I would pick a real conflict rule, because last-write-wins silently drops writes.
Practice
BeginnerWhy do replicas help reads but not writes?
Any replica can answer a read, so reads spread across copies. Every write must go through the one primary to keep a single order of changes, and the primary also has to ship each write to every replica.
BeginnerA user posts a comment, refreshes, and it is gone; a second refresh shows it. What happened?
The first refresh read from a replica that had not applied the comment yet (replication lag). The second hit a replica that had, or the same one after it caught up.
IntermediateEach replica serves 500 reads/s and you need 4,200 reads/s with one replica allowed to be down for maintenance. How many replicas?
4,200 / 500 = 8.4, so 9 replicas to carry the load, plus 1 spare for maintenance: 10. That ignores replay cost, so check how busy replicas are with the real write rate.
IntermediateReads come 0 to 2 seconds after the write, spread evenly, and lag is 300 ms. Roughly what share of read-backs are stale?
About 300 / 2000 = 15%, the chance the read arrives before the change is applied. The replica count does not change that number.
IntermediateWhy can round-robin across replicas show a comment and then hide it on the next refresh?
The replicas have different lag. The first read hit one that had applied the comment, the next one hit a slower one. Pinning the session to one replica, or tracking the highest position it has seen, gives monotonic reads.
AdvancedYour PostgreSQL primary uses synchronous_commit = on with one synchronous standby. Is a read on that standby right after commit guaranteed to see the write?
No. "on" waits for the standby to flush the change to disk, not to apply it. Only remote_apply waits until the change is visible to queries there.
AdvancedTwo primaries with last-write-wins report zero diverged keys after a test. Does that mean no writes were lost?
No. LWW makes both sides converge on the same winner, so they always agree. Writes that lost the comparison were dropped silently; in our simulation that was 9.5% at 50 ms lag and 37.6% at 500 ms. You have to track version chains to see the loss.
AdvancedA replica has been offline for three days and the primary's disk is nearly full. Why, and what prevents it?
A replication slot keeps every log segment that replica has not consumed. Drop the slot or bring the replica back, and set max_slot_wal_keep_size plus an alert on retained log size.
Cheat sheet
| Term or formula | Meaning |
|---|---|
| Primary (older name: master) | The one server that accepts writes and orders them |
| Replica or standby (older name: slave) | A copy that replays the primary's log and serves reads |
| WAL, binlog, LSN | The ordered change log and a position in it |
| Read ceiling ≈ R × 1000 / t_read | Reads scale with replica count (minus replay cost) |
| Write ceiling = 1000 / (t_write + R × t_ship) | Our model's write ceiling; real effect depends on the bottleneck |
| Little's law, closed system: R = N / X | Average response time = clients / throughput |
| P(stale) ≈ P(read gap < lag) | Staleness depends on lag and read timing, not replica count |
| Read-your-writes | Your own reads include your own writes |
| Monotonic reads | You never see an older value after a newer one |
| synchronous_commit = remote_apply | The only PostgreSQL level where the required synchronous standbys show the write before commit returns; other standbys may lag |
| Last-write-wins | Newer timestamp wins a conflict; the other write is dropped silently |
Further reading
- PostgreSQL docs, High Availability, Load Balancing, and Replication: the overview of every replication option PostgreSQL offers and what each one guarantees.
- PostgreSQL docs, Synchronous Replication: what the primary waits for at each acknowledgement level, and how
remote_applydiffers fromon. - PostgreSQL docs, Replication settings: the exact meaning of
synchronous_standby_names,max_slot_wal_keep_sizeand the hot standby settings. - PostgreSQL docs, WAL settings: the exact meaning of every
synchronous_commitlevel, fromofftoremote_apply. - MySQL Reference Manual, Replication: binary log formats, GTIDs, semisynchronous replication and Group Replication, for the MySQL side of the same ideas.
- Designing Data-Intensive Applications: the clearest book-length treatment of leaders and followers, replication lag, read-your-writes, monotonic reads and multi-leader conflicts.
- roadmap.sh System Design roadmap: where replication sits among the other data and storage topics, as a map for what to learn next.
This article describes general patterns, public database documentation and our own small simulation on one machine. It does not describe any company's private systems. The simulated numbers show the shape of the effect, not production measurements.
Frequently asked questions
What is a read replica?
A read replica is a copy of a database that receives every change from the primary and applies it, so it holds the same data a little later. Applications send read-only queries to replicas so the primary has its time free for writes. Replicas do not accept writes from the application.
Do read replicas make writes faster?
Not by themselves. Every write still goes to the one primary. Offloading reads can improve write throughput when reads were competing for the primary's CPU, cache or disk, while each direct replica adds WAL sender and network work that can reduce it once those resources saturate. Our simulation models only the second effect: its write ceiling fell from 2000 to 1667 writes/s from 0 to 10 replicas. To scale writes you need sharding or fewer, cheaper writes.
Why did I not see my own post right after I published it?
Because your read went to a replica that had not applied your write yet. That gap is replication lag. In our simulation, with 500 ms lag and a read 20 to 1000 ms after the write, the post was missing about half the time. Apps fix this with read-your-writes routing.
Does synchronous replication stop stale reads?
Only partly. In PostgreSQL, synchronous_commit = remote_apply waits for replay on the required selected standbys or quorum (for example one standby under FIRST 1, any two under ANY 2). Other listed standbys and async replicas may still lag, so check the target replica's replay LSN before a consistency-sensitive read. The 'on' and 'remote_write' levels wait only for flush or write, not replay. Every level adds write latency.
Is master-master replication better than primary and replica?
It gives you more than one place to write, but two primaries can accept conflicting writes to the same row. With last-write-wins conflict handling, our simulation silently lost 9.5% of writes at 50 ms lag and 37.6% at 500 ms. Multi-primary systems that detect conflicts, merge values, or give each key one home primary avoid that silent loss.
Were these numbers measured on a real database cluster?
No. They come from a small discrete-event simulation on one machine (virtual clock, seed 12, Node.js v22.13.1) with fixed service times: 2 ms per read, 0.5 ms per write, 0.01 ms per replica per write. They show the shape of the effect, not production numbers.
Sources
- PostgreSQL docs: High Availability, Load Balancing, and Replicationpostgresql.org
- PostgreSQL docs: Synchronous Replication (acknowledgement levels, remote_apply)postgresql.org
- PostgreSQL docs: Replication settings (synchronous_standby_names, max_slot_wal_keep_size)postgresql.org
- PostgreSQL docs: WAL settings (synchronous_commit)postgresql.org
- MySQL Reference Manual: Replicationdev.mysql.com
- Designing Data-Intensive Applications (book site)dataintensive.net
- roadmap.sh System Design roadmaproadmap.sh
Linked from
- Vertical vs horizontal scaling: one server or eight?Part 18 · Horizontal vs vertical scaling