The cable to Asia breaks. Your bank must choose: correct balance or an answer

When the network link between two copies of the same data breaks, a write on one side cannot reach the other, so each side must either refuse requests it cannot prove are up to date or answer from a copy that may be wrong. The CAP theorem says no design escapes that choice while the link is down and the data keeps changing. A common interview question asks it directly: what does CAP mean, and would you pick CP or AP for a bank?

Prerequisites

Replication. Keeping copies of the same data on more than one server, so it survives a server loss and can be read close to the user. Every write must travel to the other copies, and that takes time; we measured the effect in one database writes, ten read: replication lag. CAP is the extreme case: the trip never completes because the path is broken.

Two writers, one value. When two requests change the same thing at once, one place must decide the order, or both can win. We saw this with two fans racing for one seat, fixed by checking and claiming in one atomic step. Two withdrawals that each check "is there enough?" can both pass if they look at different copies.

Availability as a number. The share of requests a service answers. Copies side by side push it up, as in two servers in a row fail twice as often, and 99.9% is still 8 hours down a year turned the percentages into hours. CAP uses a stricter, yes-or-no meaning, defined below.

Definitions

How it works: what CAP actually says

Eric Brewer stated the idea as a conjecture in 2000, and Seth Gilbert and Nancy Lynch proved a precise version in 2002. Stated carefully: in an asynchronous network where messages can be lost, no system can guarantee both linearizable reads and an answer from every working node while a partition lasts and values keep changing.

The proof is short enough to follow in your head. Put one value on two nodes, G1 and G2, and cut every message between them. A client writes a new value to G1. Because the system is available, G1 must accept the write and reply without waiting for G2, since waiting could last forever. Now a second client reads from G2. Again, availability says G2 must answer, but G2 has never heard of the write. It returns the old value. The write finished before the read started, so linearizability required the new one. Both promises cannot hold at once. The only escapes are to have G1 refuse the write, or G2 refuse the read, and refusing is giving up availability.

The theorem only applies during a partition; when messages flow, a system can be both consistent and available. It also only applies when data changes, because if nobody writes during the cut, every copy is still current.

That is why "pick two of three" misleads. You cannot drop P, because you cannot drop network failures. A design that claims to be "CA" either runs on one node (no redundancy) or has not said what happens when the link breaks. Brewer's 2012 article makes the same point: the choice only arises during a partition, and good designs detect it, choose per operation, and plan the recovery.

For any replicated store, then, ask this: while the copies cannot talk, which operations refuse, and which answer from a copy that may be behind?

What does a CP system do during a partition?

A CP system gives up availability on the side that cannot prove it is current, and it usually decides which side that is by majority vote. Systems built on consensus protocols such as Raft (etcd) or ZAB (ZooKeeper) accept a write only after a majority of voters has stored it, and only the majority side can elect a leader and keep committing.

Reads need the same care. A minority side serving reads from its local copy would hand out values the majority has already replaced, exactly as in the proof. So a linearizable read must be confirmed by a majority, or served by a leader sure it still leads. etcd does this by default and offers a separate "serializable" read option that may return stale data from the local member. ZooKeeper takes the other default: plain reads are served by the server you are connected to and can be stale. Calling sync before a read helps bring that server's view up to date, but it does not strictly guarantee the latest value, because sync is not a quorum operation (ZooKeeper internals).

Our lab models this without a real consensus protocol. There are three votes: the Europe replica, the Asia replica, and a witness in Europe (a voter that stores little or no data and breaks ties). During the cut Europe holds 2 of 3 votes and keeps serving. Asia holds 1 of 3, so it refuses everything, reads included, because it cannot prove its balance is current.

The client sends each request to Europe or Asia by coin flip. During the 20 s cut, 224 of the 442 requests landed on Asia, so CP refused 224 / 442 = 50.7%. Europe served the other 218, and none of their reads differed from the ledger. At repair, Asia replayed the operations it was missing from Europe's log, and both sides agreed on 760.

Where the witness sits decides whose users see errors. In a third region, whichever side keeps contact with it wins. With no tiebreaker, neither half has a majority and CP must refuse writes on both sides. Placing the majority is a product decision.

What does an AP system do, and how does it repair?

An AP system keeps every working node answering from its own copy and lets the copies drift apart until the link returns. Each side records its own writes; when the partition heals, the sides exchange what they missed and reconcile conflicts.

Apache Cassandra lets you choose per request. At consistency level ONE, a read or write needs one replica, so it can keep working on either side of a split wherever a replica of that data is reachable. At QUORUM it needs a majority of the data's replicas and fails on the minority side. When replicas disagree, Cassandra keeps the write with the latest timestamp. These levels change how many replicas must respond, but QUORUM alone does not guarantee CAP consistency; Cassandra provides linearizable conditional operations through lightweight transactions that use Paxos (Cassandra guarantees).

In our lab AP served 100% of the 442 requests during the cut, and the answers paid for it. Once both sides had written, each copy lacked the other's changes, so almost every read showed a balance that differed from the ledger: 241 mismatched reads. Each side also checked funds against its own copy, so both scripted 700s passed, and so did smaller withdrawals while the combined balance was short. AP accepted 15 withdrawals while the ledger lacked the money, and the ledger fell to -135. CP's never dropped below 584.

A separate serial reference bank, one copy and no cut, ran the same stream in time order with its own decisions. It refused only 1 of the withdrawals AP accepted: Asia's 700 (we checked which one with an extra print in a local copy of the lab; the published run.mjs prints only the count). The other 14 were small ones the serial bank also paid, because it still had money after refusing that 700; AP accepted them while its combined books were short.

The repair step then decides what is lost. The lab runs AP twice with two repair rules:

• Log merge: both replica logs are combined by operation id, sorted by time, and replayed from the start balance, so every accepted operation counts once. The balance matched the ledger: error 0. The overdraft stays, because the money really left; merge only makes the books honest.
• Last-writer-wins on the balance: the replica with the later last write overwrites the other. Europe wrote last, so its 760 replaced Asia's copy and Asia's 94 cut-time writes vanished. The ledger said 92: off by 760 minus 92 = 668. The error stayed: 223 later reads differed, and the run ended at 803 against a ledger of 135.

The 668 falls straight out of Asia's writes. Both sides started the cut at 1336 (CP's frozen Asia still showed it at repair). Asia's writes took its copy to 668, a net change of 668 minus 1336 = minus 668, a decrease of 668. LWW threw that decrease away, which overstates the repaired balance by 668. LWW also ended at 803, the same as CP: its books look as if Asia had refused everything, but Asia paid out and the system forgot.

The ledger is the lab's own record of every operation any replica accepted, in time order. No replica reads it and no repair uses it; it is only the yardstick for money really handed out or taken in. Zero reads differing from it is a good sign, not a proof that CP is linearizable: the lab has instant replication and perfect clocks, and a linearizability checker, like the ones behind Jepsen's analyses, tests whether a recorded history admits at least one legal sequential order that respects real time. Passing checks one history; proving an implementation correct requires reasoning about all possible histories.

Does consistency stop overspending?

No, and this is the most common confusion about CAP. Linearizability says operations behave as if on one copy, in real-time order; it says nothing about which operations are allowed. A linearizable store will take the balance to -400 if the application writes "balance = balance - 700" twice without checking.

"Never below zero" is an invariant the application must enforce, usually by checking and changing in one atomic step, as with the seat in Part 2. Many stores offer a conditional write: in DynamoDB a write can carry a condition expression and fails if it is false. In SQL it is an UPDATE ... WHERE balance >= 700 that reports how many rows changed.

-- debit.sql: check and pay in one atomic step on the one up-to-date copy
UPDATE accounts
SET    balance = balance - 700
WHERE  id = 42 AND balance >= 700;
-- 1 row updated  = paid
-- 0 rows updated = not enough money, decline

CP makes that step meaningful during a partition, because the check runs on the copy that has every accepted write. In our lab both modes ran the same funds check; AP ran it on a copy missing the other side's withdrawals.

Brewer's 2012 article describes the other road: let the operation through during the partition, record it, detect the broken invariant at recovery, and compensate. His example is the automated teller machine, which can allow withdrawals up to a limit while disconnected and settle later. The pattern is to bound the risk, log every accepted operation, and define the fix for when the books go red. That needs merged logs; last-writer-wins erases the evidence.

What CAP does not say

CAP is a narrow theorem, often stretched beyond what it proves.

• It is not a label for a whole database. Many stores let you choose per request, so one system can be CP for balances and AP for profile pictures.
• Its C is not the C in ACID. ACID's consistency means transactions keep the application's rules, closer to the invariant above. CAP's C is linearizability.
• Its A is stricter than uptime. A CP system that refuses requests on the minority side for 20 seconds fails CAP availability but can still meet a 99.9% service target.
• It says nothing about latency. Consensus needs quorum communication, so when the required majority spans continents, every write waits for a cross-region round trip even with no partition. PACELC adds this: if Partitioned, A or C; Else, Latency or Consistency. More bandwidth does not shorten that wait, as ten lanes, still slow showed for latency versus throughput.
• Many systems are neither. A single leader with asynchronous followers is not linearizable (followers lag) and not fully available (the minority cannot write).

So the choice belongs to each piece of data. Part 8 of this series asks why your like count can be wrong but your balance never.

Worked example

The video uses one shared account and two withdrawals at the same moment. These numbers are a worked example, concept only: the lab's scripted 700s happen alongside regular traffic, so the lab's own numbers differ.

Assumed: one account, start balance 1000, copies in Europe and Asia, link cut.
One client in Europe and another in Asia each ask for 700 at the same moment.

CP (witness vote in Europe, so Europe has 2 of 3 votes)
  Europe: check 1000 >= 700, pay        Europe copy = 1000 - 700 = 300
  Asia:   1 of 3 votes, no majority     "try later"
  total paid 700, books 300, never negative

AP (each side trusts its own copy)
  Europe: check 1000 >= 700, pay        Europe copy = 300
  Asia:   check 1000 >= 700, pay        Asia copy   = 300
  total paid 700 + 700 = 1400
  combined ledger = 1000 - 1400 = -400  (each copy still shows 300)

And the measured version, from our lab (labs/sd-7, seed 7, one Apple M2 Pro machine, Node v22.13.1, simulated clock). Assumptions: 60 simulated seconds at about 20 requests per second, split evenly between the sides; 50% reads, 25% deposits, 25% withdrawals of 5 to 25; one scripted 700 on each side at 30 s; the cut runs from 20 s to 40 s. Every mode receives the identical request stream, including amounts.

requests            before 398   during 442   after 398    (total 1238)

during the cut                          CP             AP (merge or LWW, same until repair)
  served                                218 (49.3%)    442 (100%)
  refused                               224 (50.7%)    0
  mismatched reads                      0              241
  withdrawals accepted, ledger short    0              15
  withdrawals the serial bank refused   0              1
  lowest ledger balance                 584            -135

at repair (20 s to 40 s writes: Europe 105, Asia 94 under AP; Asia 0 under CP)
  CP        Europe 760, Asia 1336 -> replay -> 760, ledger 760, error 0
  AP-merge  Europe 760, Asia 668  -> merge  ->  92, ledger  92, error 0
  AP-LWW    Europe 760, Asia 668  -> Europe wins, 94 writes dropped
            balance 760, ledger 92, error 760 - 92 = 668
            after repair: 223 reads differ; end 803 vs ledger 135 (668)

check:      1336 + (760 - 1336) + (668 - 1336) = 92   (both sides' changes, once each)
reference:  serial bank, one copy, no cut, final balance 835

CP turns the minority side's traffic into errors and keeps the books right; AP keeps answering and pays with mismatched reads, overdrafts and a delicate repair. "About half refused" holds only because traffic was split evenly and the majority sat on one side.

Code

This Python script reproduces the 1000 / 700 / 700 example in both modes, then repairs the AP run both ways. Standard library only; run it with python3 cap_toy.py.

# cap_toy.py: one account, two copies, a cut link, two withdrawals of 700 (CP vs AP)
START = 1000
REQUESTS = [("Europe", 700), ("Asia", 700)]   # same moment, link already cut
VOTES = {"Europe": 2, "Asia": 1}               # the witness vote sits in Europe
MAJORITY = 2                                   # 2 of 3 votes

def run(mode):
    copy = {"Europe": START, "Asia": START}    # each side's own balance
    log = {"Europe": [], "Asia": []}           # each side's own accepted operations
    ledger = START                             # our yardstick: every accepted operation
    for t, (side, amount) in enumerate(REQUESTS):
        if mode == "CP" and VOTES[side] < MAJORITY:
            print(f"  {side:6} -{amount}: try later (cannot reach a majority)")
            continue
        if copy[side] < amount:                # check and pay in one step, on this copy
            print(f"  {side:6} -{amount}: declined, not enough money")
            continue
        copy[side] -= amount                   # the link is cut, so the other copy never hears
        log[side].append((t, -amount))
        ledger -= amount
        print(f"  {side:6} -{amount}: paid, this copy now shows {copy[side]}")
    print(f"  copies show {copy}, ledger of accepted operations {ledger}")
    return copy, log, ledger

def repair_merge(log):
    ops = sorted(log["Europe"] + log["Asia"])  # union of both logs, in time order
    return START + sum(delta for _, delta in ops)

def repair_lww(copy, log):
    last = {s: (log[s][-1][0] if log[s] else -1) for s in copy}
    winner = max(copy, key=lambda s: last[s])  # the side that wrote last keeps its balance
    return winner, copy[winner]

print("CP (refuse rather than guess):")
run("CP")
print("AP (answer from the local copy):")
copy, log, ledger = run("AP")
print(f"  repair by merging logs: balance {repair_merge(log)} (matches the ledger, still overdrawn)")
winner, bal = repair_lww(copy, log)
print(f"  repair by last-writer-wins: {winner} wins, balance {bal}, ledger {ledger}, off by {bal - ledger}")

What to observe when you run it:

CP (refuse rather than guess):
  Europe -700: paid, this copy now shows 300
  Asia   -700: try later (cannot reach a majority)
  copies show {'Europe': 300, 'Asia': 1000}, ledger of accepted operations 300
AP (answer from the local copy):
  Europe -700: paid, this copy now shows 300
  Asia   -700: paid, this copy now shows 300
  copies show {'Europe': 300, 'Asia': 300}, ledger of accepted operations -400
  repair by merging logs: balance -400 (matches the ledger, still overdrawn)
  repair by last-writer-wins: Asia wins, balance 300, ledger -400, off by 700

Try three changes. Set VOTES = {"Europe": 1, "Asia": 2} and Europe becomes the side that says try later. Set both to 1 and CP refuses both withdrawals. In AP, make the second request 200: both pay, the ledger ends at 100, and nothing goes red, which is why AP can be acceptable when amounts are bounded. The full lab (node labs/sd-7/run.mjs) runs the same idea over a whole stream with reads and deposits.

Failure modes and edge cases

You cannot tell a partition from a slow node. A node only sees that replies stopped; the other side may be dead, overloaded or cut off. So partitions are detected by timeouts, and a timeout is a guess. Too short and a busy moment triggers a needless leader election; too long and users wait. Watch election counts: a rising count often means timeouts are too tight for your real latency.

Split brain. Two nodes both believe they lead and both accept writes, typically a two-node failover pair where each side promotes itself when it loses sight of the other. The fix is a real quorum (an odd number of voters, or a witness) plus fencing: a leader term or token that storage checks, so an old leader's writes are rejected.

A deposed leader serving reads. A leader cut off in the minority may still think it leads for a while and serve values the new leader has already changed. Consensus systems confirm leadership with a majority before answering, or use leases that rely on bounded clock drift. Turning on a "fast local read" option chooses stale reads for that request.

Partial and one-way partitions. A can reach B and B can reach C, but A cannot reach C; or messages flow one way only. These expose bugs in timeouts and failure detectors, so test with fault injection that drops traffic between specific pairs, not only whole-node kills.

Clock skew under last-writer-wins. LWW trusts timestamps from different machines. If one site's clock runs ahead, its writes win even when they happened earlier. Our lab has perfect clocks, so real LWW can lose more. Prefer merges that keep every operation, or a single home copy per key.

Retries during a partition. A client told "try later" may retry elsewhere, and a timed-out request may have succeeded. Without a unique id per write, a retried withdrawal can be paid twice once logs merge. Give every write an id and deduplicate on it, as the lab's merge does by operation id.

Trade-offs

Common mistakes

In an interview

1. Clarify: which data is replicated across sites, and which operations must never be wrong (balances, stock) versus can be briefly stale (likes, views)?
2. Assumptions: state the sites, say a partition between them will happen, and assume a majority quorum for anything strongly consistent.
3. Structure: draw the replicas and voters, mark where the majority lives, and say which side refuses during a split.
4. Deep dive: define C as linearizability and A as an answer from every working node; walk through what a read and a write do on each side during the cut and at repair.
5. Trade-offs: CP costs errors on the minority side and cross-region latency when the quorum spans regions; AP costs stale reads, broken invariants and careful merging.
6. Wrap up: choose per operation: CP with an atomic funds check for the balance, AP with merge for counters and feeds, and a tested recovery plan.

Practice

Cheat sheet

Further reading

• Gilbert and Lynch (2002): the formal statement and proof of CAP, with the exact definitions of consistency, availability and partition used here.
• Brewer, CAP Twelve Years Later (2012): the author of the idea on why "two of three" misleads, and how to detect partitions, choose per operation and compensate afterwards.
• Herlihy and Wing (1990): the original definition of linearizability, including how operations that overlap in time may be ordered.
• Jepsen, Consistency Models: a clear map of consistency models, from linearizable down to eventual, and which ones can stay available during a partition.
• Designing Data-Intensive Applications: book-length treatment of replication, consensus and the cost of linearizability, for the step after this article.
• roadmap.sh System Design roadmap: where CAP sits among the other system design topics, as a guide for what to learn next.

This article describes general patterns and our own small simulation. It does not describe any bank's or company's private systems. The simulated numbers show the shape of the trade-off, not production measurements.