# A highway with 10 lanes can still be slow URL: https://systemdesignsimple.com/ten-lanes-still-slow/ Published: 2026-11-03 Category: System Design Series: System Design, simply, Part 6: Latency vs throughput, measured Publisher: System Design Simple (https://systemdesignsimple.com) ## Short answer Latency is how long one request takes; throughput is how many requests finish per second. In our simulation, 1 to 10 parallel workers raised throughput from 5 to 49.8 req/s while median latency stayed at 200 ms. One shared 50 ms lock capped throughput at 19.9 req/s, near 1 ÷ 50 ms = 20, and at 10 workers raised median latency to 500 ms, mostly time spent waiting for the lock. ## Key facts - **Latency:** Time for one request, here from the moment a worker picks it up to the moment it is done - **Throughput:** Requests finished per second: 1,000 requests ÷ batch time - **More lanes, no lock (simulated):** 1, 2, 5, 10 workers: 5, 10, 24.9, 49.8 req/s; p50 200 ms and p95 209 ms at every count - **Lock ceiling:** A serial step held 50 ms per request allows at most 1 ÷ 0.05 s = 20 req/s, whatever the worker count - **With the lock (simulated):** 1, 2, 5, 10 workers: 4, 8, 19.5, 19.9 req/s; p50 250, 251, 256, 500 ms - **Where the time went:** At 10 workers: 200 ms work + about 249 ms average lock wait + 50 ms lock hold ≈ 500 ms - **Little's law check:** L = λW: 49.8 req/s × 0.2 s ≈ 10 busy workers; 19.9 req/s × 0.5 s ≈ 10 Adding parallel workers lets a system finish more requests per second without making any single request faster, and one shared step that only one request can use at a time puts a hard ceiling on the whole path. This article separates latency from throughput with a small simulation, shows where the extra time goes when a 50 ms lock sits in the path, and explains how to find that lock in a real system. ## Prerequisites **A request spends its time in two ways.** It is either being worked on (CPU, a query running) or waiting for something busy (a free worker, a lock, a connection). Work time belongs to the request; waiting time depends on everything else happening at that moment. **Percentiles.** The p50 (median) is the latency half of all requests beat; the p95 is the latency 95% beat, the slow end one request in twenty sees. A mean hides that tail. **Throughput ceilings from part 5.** A resource with c parallel slots, each held S seconds per request, can finish at most c ÷ S requests per second. Part 5 used it to explain [why a fast server can lose under load](https://systemdesignsimple.com/fast-alone-slow-crowded/); here we apply it to a step with exactly one slot, a lock. ## Definitions > Definition: **Latency**: the time one request takes from a chosen start point until it is done. Always say where the clock starts: at the client, at arrival in the queue, or (in our simulation) at worker pickup. > Definition: **Throughput**: requests finished per second. In our simulation, 1,000 requests divided by the batch time. > Definition: **Worker (lane)**: a thread, process, container or server that handles one request at a time, in parallel with the others. > Definition: **Serial step**: a part of the request path that only one request can be inside at a time, such as a mutex, a row lock held by every write, or a single-threaded consumer. Its hold time per request sets a throughput ceiling of 1 ÷ hold time. > Definition: **Lock contention**: requests asking for a lock faster than it can be granted, so they wait. It shows up as lock wait time, not CPU. > Definition: **Bottleneck**: the resource with the highest demand relative to its capacity. It sets the maximum throughput of the whole path, however much capacity everything else has. > Definition: **Little's law**: in any stable system, the average number of items inside equals their throughput times the average time each spends inside: L = λ × W. The same split is how working engineers explain it. Alex Xu, author of the *System Design Interview* books, put it this way on X: latency is "what users feel when they click a button", throughput is "how many packets flow through the pipe".
Post on X by Alex Xu (@alexxubyte) titled Latency vs. Throughput: latency is the delay per request that users feel, throughput is the volume handled per second
Post on X by @alexxubyte (Alex Xu, ByteByteGo). Screenshot; open the link for the original post.
## How it works: work, waiting and lanes Write the latency of a request as a sum: ```text latency = own work + waiting = (time being processed) + (time in line for workers, locks, connections, disks) ``` Adding workers changes only the second term. A request that needs 200 ms of CPU still needs 200 ms when nine others run beside it on other cores. More workers change how many 200 ms jobs happen per second, and how long a new request waits for a free worker. With nothing shared between workers: ```text throughput ≈ workers / work time (c / S) latency ≈ work time (once a worker has picked the request up) ``` So 1 worker at 200 ms per request finishes 1 ÷ 0.2 s = 5 requests a second; 10 workers finish 10 ÷ 0.2 s = 50. Latency from pickup stays at 200 ms. Scaling out buys more output per second at the same speed per item. [Image: One highway lane carrying cars, with a readout of 5 requests per second and median 200 ms, above an empty four-row results table | One lane: 5 requests a second, each taking a median of 200 ms.] ### Where the clock starts matters Our simulation starts the latency clock at pickup. That isolates the work and any waiting after pickup (for the lock), and deliberately leaves out time spent in the backlog before a worker was free. In a real service that backlog wait is often most of what a user feels, and more workers do shrink it. Two teams can both be right about "latency" while measuring different intervals. ### Adding a shared step Now suppose every request, after its own 200 ms of work, must hold one shared lock for 50 ms. Think of one hot database row, a shared map under a mutex, or a counter every request increments. The lock is granted first come, first served. Two things change: - The work per request grows from 200 ms to 250 ms, so with a single worker latency becomes 250 ms and throughput 1 ÷ 0.25 s = 4 req/s. - A new ceiling appears: one holder at a time for 50 ms means at most 1 ÷ 0.05 s = 20 grants a second, whatever the number of workers. The throughput of the whole path is whichever limit is lower: ```text X = min( workers / (work + hold), 1 / hold ) = min( c / 0.25 s, 20 req/s ) ``` For 1, 2 and 5 workers the first term gives 4, 8 and 20 req/s; at 5 the limits meet. At 10 the lanes could do 40 req/s, the lock allows 20, and the surplus can only become waiting. ## What the simulation measured To check the arithmetic we wrote a discrete-event simulation: a virtual clock and a list of timed events (work finished, lock granted), jumping from one event to the next. No real threads, scheduler, garbage collector, network or disk, so machine speed does not affect the numbers (Node.js v22.13.1 on an Apple M2 Pro, seed 42). Each run is a backlog of 1,000 requests with 200 ms ± 10 ms of uniform work; the same 1,000 work times are reused in every run, so only the worker count changes. Latency is timed from pickup to done. | Workers | Batch time | Throughput | p50 | p95 | | --- | --- | --- | --- | --- | | 1 | 200.2 s | 5.0 req/s | 200 ms | 209 ms | | 2 | 100.2 s | 10.0 req/s | 200 ms | 209 ms | | 5 | 40.1 s | 24.9 req/s | 200 ms | 209 ms | | 10 | 20.1 s | 49.8 req/s | 200 ms | 209 ms | **Run 1, independent lanes.** Throughput rises in step with workers: 5, 10, 24.9, 49.8 req/s, within half a percent of c ÷ 0.2 s (the shortfall is lanes going idle at the end of the batch). Latency does not move: p50 200 ms and p95 209 ms at every count. [Image: All ten highway lanes full of cars, with a readout of 49.8 requests per second and the median still at 200 ms | Ten lanes: 49.8 requests a second, and the median is still 200 ms.] | Workers | Batch time | Throughput | p50 | p95 | Avg lock wait | | --- | --- | --- | --- | --- | --- | | 1 | 250.2 s | 4.0 req/s | 250 ms | 259 ms | 0 ms | | 2 | 125.3 s | 8.0 req/s | 251 ms | 259 ms | 0 ms | | 5 | 51.3 s | 19.5 req/s | 256 ms | 260 ms | 6 ms | | 10 | 50.2 s | 19.9 req/s | 500 ms | 500 ms | 249 ms | **Run 2, one shared 50 ms lock.** Each row is a different regime. - **1 worker: 250 ms, no waiting.** The only request in flight always finds the lock free: 200 ms of work plus 50 ms of holding the lock, average wait 0, throughput 1 ÷ 0.25 s = 4.0 req/s. - **2 workers: almost no waiting.** Throughput doubles to 8.0 req/s. The 251 ms p50 shows the two requests occasionally reaching the lock together; the average wait still rounds to 0. - **5 workers: at the knee.** Five lanes ask for the lock about 5 ÷ 0.25 s = 20 times a second, exactly its capacity. It is busy 19.5 × 0.05 = 97.5% of the time, waits appear (6 ms average) and throughput is 19.5 req/s. - **10 workers: past the ceiling.** 19.9 req/s, barely above 5 workers, and the median doubles to 500 ms. Five extra workers bought 0.4 req/s and about 249 ms of waiting per request. [Image: A toll booth labelled LOCK 50 ms placed across all the lanes, with a bracket labelled counted running from pickup to done | The shared lock sits across every lane; latency is counted from pickup to done.] ### Where the 500 ms comes from The 10-worker latency splits exactly into its parts: ```text 1 worker: 200 ms work + 0 ms lock wait + 50 ms lock hold = 250 ms 10 workers: 200 ms work + ~249 ms lock wait + 50 ms lock hold ≈ 500 ms ``` The work time is the same in both rows, so every extra millisecond is lock wait. The 10-worker p95 is also exactly 500 ms because a lock busy all the time grants 50 ms slots back to back. Ten lanes take turns, so each lane gets the lock once every 10 × 50 = 500 ms, and each request takes exactly one turn. Work jitter disappears, since a request that finishes early just waits longer, and the saturated lock sets the pace for every lane. ## Why one serial step caps everything The lock is a resource with one slot: c ÷ S = 1 ÷ 0.05 s = 20 req/s, and Run 2 converges on it. This is the bottleneck bound. Each request places a demand D on each resource (time per request ÷ parallel slots), and: ```text X ≤ 1 / D_max (the busiest resource sets the ceiling) X ≤ N / (D_1 + D_2 + ... + D_k) (with N requests in flight and no waiting at all) Run 2: D_workers = 0.25 s / c, D_lock = 0.05 s / 1 c = 10 → D_workers = 0.025 s, D_lock = 0.05 s → the lock is the bottleneck ``` These bounds come from operational analysis, as in Quantitative System Performance (see sources). More workers lower D_workers but cannot touch D_lock, so the ceiling stays at 20 req/s. To raise it you have to hold the lock for less time, split it into independent locks, or remove it from the path. ### Amdahl's law says the same thing about speedup Amdahl's law gives the speedup of a job whose fraction s must run serially: speedup(N) = 1 ÷ (s + (1 − s) ÷ N), never more than 1 ÷ s. Here s = 50 ÷ 250 = 0.2, so at most 5 times one worker: 5 × 4 = 20 req/s, the same ceiling. The curves differ before the ceiling. Amdahl assumes nothing else runs during the serial part, giving 1 ÷ (0.2 + 0.08) ≈ 3.6 times at 10 workers, about 14.3 req/s. Our other lanes keep working while one request holds the lock, so throughput reaches the ceiling much sooner (19.9 req/s). Treat Amdahl as the pessimistic curve and 1 ÷ D_max as the limit both agree on. Often the serial step exists for correctness: a seat must not be sold twice, as in [how an atomic update stops two fans buying one seat](https://systemdesignsimple.com/two-fans-one-seat/). Keep the lock; shrink what happens inside it. ### The real-world caveat: the backlog our metric does not count Run 2's 10-worker median of 500 ms looks worse than the 1-worker 250 ms, but for whoever is at the back of the line it is far better. The last of the 1,000 requests finished after 250.2 s with one worker and after 50.2 s with ten. Our metric starts at pickup, so it never sees the minutes the 1,000th request waited for the single worker. In a real system, more workers also shrink that pre-pickup queue, so end-to-end user wait can improve even when pickup-to-done p50 does not. The same data shows where that stops. From 5 to 10 workers the batch time went from 51.3 s to 50.2 s, about 2%. Past the lock's ceiling, extra workers only move waiting from the backlog into the lock's line; the worker count decides where the wait is spent and which dashboard shows it. ## Reading p50 and p95 honestly Run 1's narrow gap (200 against 209 ms) is just the ±10 ms work jitter. Run 2 at 1 worker shifts both by the 50 ms hold (250 and 259 ms). At 10 workers the gap closes (500 and 500 ms) because the saturated lock imposes one rhythm. Three lessons for real dashboards: - **Identical printed percentiles do not prove identical distributions.** Run 1's four rows really are the same distribution, but only because the code reuses the same 1,000 work times. On a dashboard you cannot know that. If 1% of requests take 2 s, the p95 does not move at all. Percentiles are a few rounded points on a curve. - **A narrow gap at a high value often means one saturated resource:** everyone waits behind the same line. A wide gap points at something hitting only some requests, such as slow queries or garbage collection pauses. - **Do not average percentiles.** The mean of ten servers' p95 values is not the fleet's p95. Merge histograms first, then compute percentiles. The Google SRE monitoring chapter (see sources) agrees: collect latency as a distribution, and track failed requests separately so fast errors do not flatter it. ## Little's law ties the two together Latency and throughput are not independent knobs. For any stable system (Little, 1961): ```text L = λ × W L average number of requests inside the system λ throughput (arrival rate = completion rate when stable) W average time each request spends inside ``` Apply it to the 10 workers. In both runs every lane is busy all the time, so about 10 requests are inside: ```text Run 1, 10 workers: L = 49.8 req/s × 0.2 s ≈ 10 (all ten lanes doing work) Run 2, 10 workers: L = 19.9 req/s × 0.5 s ≈ 10 (same ten lanes, half the throughput) ``` With L fixed at 10, cutting λ from about 50 to about 20 must stretch W from 0.2 to 0.5 s. The law is also the fastest sanity check on a load test: throughput × latency should equal the concurrency you configured; if not, something is counting a different interval. The law works on any box you draw. Three boxes for Run 2 at 10 workers: ```text doing its own work: 19.9 req/s × 0.200 s ≈ 3.98 requests holding the lock: 19.9 req/s × 0.050 s ≈ 1.00 request (lock busy 99.5% of the time) waiting for the lock: 19.9 req/s × 0.249 s ≈ 4.96 requests total ≈ 9.9 ≈ 10 workers ``` Half of the lanes are parked at the lock, which is why a lock-bound service looks calm on CPU graphs: waiting threads use no CPU. The lock's utilisation (throughput × hold time) is 99.5%. Part 1 walks through the curve behind this in [performance and scalability for a ticket rush](https://systemdesignsimple.com/50k-fans-5k-seats/): waiting grows slowly at moderate utilisation and very fast near 100%. The same law turns requirements into concurrency: 300 requests a second (assumed) at 200 ms each keeps 300 × 0.2 = 60 in flight, so at least 60 workers before headroom. Getting the 300 is the subject of [requirements, SLOs and capacity estimation](https://systemdesignsimple.com/ride-app-requirements/). ## Finding the toll booth in a real system In production the serial step rarely has "LOCK" painted on it. The signature is the same: throughput flattens while latency rises, CPU stays moderate, and more instances do not help. The USE method (see sources) gives a checklist: for every resource check Utilisation, Saturation and Errors. It includes software resources such as mutex locks, where utilisation is the time held and saturation is the number of threads queued for it. ### In PostgreSQL - **Who is waiting right now:** `pg_stat_activity` shows each session's `wait_event_type`; a steady set showing `Lock` means lock contention. The `pg_locks` view lists every lock, granted or not, and `pg_blocking_pids(pid)` names the sessions blocking a given one. - **Waits over time:** `log_lock_waits = on` logs any lock wait longer than `deadlock_timeout` (1 s by default). - **The usual hot spots:** an UPDATE on one counter row from every request, or `SELECT ... FOR UPDATE` on a shared parent row. The explicit locking documentation lists which lock modes conflict; a row lock taken by an UPDATE blocks other writers of that row until commit, so the hold time is the rest of the transaction, not just the statement. ### In application code - **Go:** `runtime.SetMutexProfileFraction` and the block profile in `pprof` show where goroutines wait on mutexes. - **Java:** thread dumps that repeatedly show many threads BLOCKED on the same monitor point straight at it. - **Linux:** off-CPU analysis (time threads spend blocked) finds waits a CPU profiler never sees. - **Your own metrics:** a histogram of time to acquire each lock or pool, recorded separately from hold time, tells you more about contention than any other lock metric. ### Confirm it with a step test Raise concurrency in steps (1, 2, 5, 10, 20) and record throughput, p50, p95 and wait time at each, like the tables above. When throughput stops rising, compute the suspect's ceiling (slots ÷ hold time). If it matches the plateau and that resource's wait time is what grew, you have found the bottleneck. Fix it and rerun: the plateau should move by the predicted amount. This is the "measure it" step of the design loop used in [designing Instagram's feed in five steps](https://systemdesignsimple.com/design-instagram-feed/): predict a number, measure it, and explain the gap. ## Worked example The video's picture: lanes are workers, cars are requests. Widening the road from 1 lane to 10 moves ten times as many cars, but each still drives the same 200 ms stretch. Then one toll booth taking 50 ms per car goes across all ten lanes. Inputs are simulation configuration; results come from its output. [Image: Cars piled up in front of the toll booth with a red waiting for lock pill, and the results table row reading 19.9 requests per second and 500 ms | With the booth in place, ten lanes give 19.9 requests a second at a 500 ms median.] ```text CONFIG 1,000 requests, work 200 ms ± 10 ms, lock 50 ms (run 2), seed 42 latency = pickup to done LANES c / S 1 / 0.2 s = 5 req/s measured 5.0 10 / 0.2 s = 50 req/s measured 49.8 latency 200 ms of work measured p50 200 ms at every count BOOTH ceiling 1 / 0.05 s = 20 req/s 1 worker 1 / 0.25 s = 4 req/s measured 4.0, p50 250 ms 10 workers min(40, 20) = 20 req/s measured 19.9 LITTLE W = L / X 10 / 19.9 req/s ≈ 0.50 s measured p50 500 ms split 200 work + 249 wait + 50 hold ≈ 500 ms BACKLOG last of 1,000 done: 1 worker 250.2 s, 5 workers 51.3 s, 10 workers 50.2 s ``` The decision follows from the ceiling. Up to about 5 workers, adding lanes is the cheap win. Past 5, each new worker costs memory and connections, adds no throughput and raises latency. More throughput must come from the booth: a 25 ms hold (assumed) gives a 40 req/s ceiling; splitting the lock into 4 independent ones (assumed, for partitionable data) lifts the lock's limit to 80. ## Code The simulation behind every measured number, trimmed for reading. Same model, seed and event ordering as our lab script, and the same numbers. Save it as `latency_lab.mjs` and run `node latency_lab.mjs` (Node.js 18 or newer, no packages, no network). ```js // latency_lab.mjs: more lanes, same work. Discrete-event simulation, virtual clock. // 1,000 requests, each 200 ms +/- 10 ms of its own work. Run 2 adds one shared // 50 ms lock (first come, first served) after the work. Latency = pickup to done. // Node.js 18+, no packages, no network. Same seed, same numbers, any machine. const SEED = 42, REQUESTS = 1000, WORK_MS = 200, JITTER_MS = 10, LOCK_MS = 50; const WORKER_COUNTS = [1, 2, 5, 10]; function mulberry32(seed) { // small seeded random number generator let a = seed >>> 0; return () => { a = (a + 0x6d2b79f5) >>> 0; let t = a; t = Math.imul(t ^ (t >>> 15), t | 1); t ^= t + Math.imul(t ^ (t >>> 7), t | 61); return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; } // The same 1,000 work times for every run: only the number of lanes changes. const rand = mulberry32(SEED); const work = Array.from({ length: REQUESTS }, () => WORK_MS - JITTER_MS + rand() * 2 * JITTER_MS); function simulate(workers, useLock) { const events = []; // "work done" events, sorted by (time, sequence) let seq = 0, next = 0, lockFreeAt = 0, lockWait = 0, last = 0; const latencies = []; const push = (t, start) => { const e = { t, s: seq++, start }; let lo = 0, hi = events.length; while (lo < hi) { const m = (lo + hi) >> 1; if (events[m].t < t || (events[m].t === t && events[m].s < e.s)) lo = m + 1; else hi = m; } events.splice(lo, 0, e); }; const pickUp = (t) => { if (next < REQUESTS) push(t + work[next++], t); }; for (let w = 0; w < workers; w++) pickUp(0); // every lane takes one request while (events.length) { const { t, start } = events.shift(); // this request's own work just ended let done = t; if (useLock) { // the one shared step const granted = Math.max(t, lockFreeAt); // wait if someone holds the lock lockWait += granted - t; lockFreeAt = granted + LOCK_MS; done = lockFreeAt; } latencies.push(done - start); // counted from pickup, not from arrival last = Math.max(last, done); pickUp(done); // lane is free, take the next request } latencies.sort((a, b) => a - b); const pct = (p) => Math.round(latencies[Math.ceil(p * REQUESTS) - 1]); return { workers, rps: (REQUESTS / (last / 1000)).toFixed(1), p50: pct(0.5), p95: pct(0.95), wait: Math.round(lockWait / REQUESTS) }; } for (const useLock of [false, true]) { console.log(useLock ? `\nshared ${LOCK_MS} ms lock` : "independent lanes"); console.log("workers req/s p50 ms p95 ms avg lock wait ms"); for (const w of WORKER_COUNTS) { const r = simulate(w, useLock); console.log(`${String(w).padStart(7)} ${r.rps.padStart(7)} ${String(r.p50).padStart(8)} ` + `${String(r.p95).padStart(8)} ${String(r.wait).padStart(18)}`); } } ``` Real output: ```text independent lanes workers req/s p50 ms p95 ms avg lock wait ms 1 5.0 200 209 0 2 10.0 200 209 0 5 24.9 200 209 0 10 49.8 200 209 0 shared 50 ms lock workers req/s p50 ms p95 ms avg lock wait ms 1 4.0 250 259 0 2 8.0 251 259 0 5 19.5 256 260 6 10 19.9 500 500 249 ``` The prediction side in plain Python 3, no simulation: ```python # predict.py: bottleneck law and Little's law, before you run anything. WORK_S, LOCK_S = 0.200, 0.050 # work per request, lock hold per request for workers in (1, 2, 5, 10): lanes = workers / WORK_S # Run 1 ceiling: c / S with_lock = min(workers / (WORK_S + LOCK_S), 1 / LOCK_S) # Run 2: lanes or lock, whichever is lower latency_ms = workers / with_lock * 1000 # Little's law: W = L / X print(f"{workers:>2} workers no lock {lanes:5.1f} req/s " f"lock {with_lock:5.1f} req/s, latency about {latency_ms:4.0f} ms") ``` ```text 1 workers no lock 5.0 req/s lock 4.0 req/s, latency about 250 ms 2 workers no lock 10.0 req/s lock 8.0 req/s, latency about 250 ms 5 workers no lock 25.0 req/s lock 20.0 req/s, latency about 250 ms 10 workers no lock 50.0 req/s lock 20.0 req/s, latency about 500 ms ``` The formula misses only at the knee: at 5 workers it predicts 20 req/s and 250 ms against a measured 19.5 req/s and 256 ms, because random work times make requests collide at the lock even when average demand equals capacity. Real systems lose more there, so plan a serial step well below 100% utilisation. What to observe when you change the simulation: - Set `LOCK_MS` to 25. The ceiling becomes 40 req/s; 10 workers reach 39.7 req/s and p50 falls to 250 ms. Halving the critical section doubles the ceiling, which no number of workers could do. - Set `WORKER_COUNTS` to `[10, 20, 50]`. With the lock, throughput stays at 19.9 req/s and p50 grows as workers × 50 ms: 1,000 ms at 20 workers, 2,500 ms at 50. ## Failure modes and edge cases **Autoscaling on latency into a lock.** An autoscaler that adds instances when p95 rises keeps adding them, because more instances make lock waits longer. Detect: flat throughput while instance count climbs. Fix: scale on the bottleneck's saturation and cap instances. **Lock held across slow work.** A transaction that takes a row lock and then calls an external API turns the hold time into that call's time. A 50 ms hold becomes 500 ms during a partner's slow minute, and the ceiling drops from 20 to 2 req/s. Detect with traces that show calls inside transactions; fix by doing slow work before taking the lock or after releasing it. **Measuring the wrong interval.** Server-side latency usually starts after the request leaves the accept or thread pool queue, so a saturated service can report a stable p50 while clients time out. Compare client-side and server-side latency, and record queue time as its own metric. **Retries against a saturated serial step.** Client timeouts shorter than the lock wait cause retries, which add demand to a resource already at 100%. Fix with backoff, jitter and early rejection, as covered for overload in [the ticket rush article](https://systemdesignsimple.com/50k-fans-5k-seats/). **Splitting a lock that guards an invariant.** Sharding one counter removes contention, but "never sell more than 5,000 seats" then needs per-shard quotas or a final check, or the faster version can oversell. ## Trade-offs | Option | Good for | Costs you | Use when | | --- | --- | --- | --- | | Add workers or instances | Throughput on independent work; drains backlogs faster | Memory, connections, money; no help past a serial ceiling | Utilisation of the workers is the highest demand on the path | | Shorten the critical section | Raises the 1 ÷ hold ceiling directly | Code changes; may need to restructure transactions | Lock wait dominates latency and work can move outside the lock | | Split the lock (sharding, striping) | Ceiling grows with the number of independent locks | Harder invariants across shards; uneven keys create new hot spots | The protected data partitions naturally (per user, per seat block) | | Batch inside the lock | One acquisition serves many requests | Added latency while a batch fills; larger failure blast radius | Hold time is mostly fixed overhead (commit, fsync) rather than per-item work | | Remove the lock (atomic operations, append-only log, single writer queue) | No waiting on the hot path | Different consistency model; work moves to a background consumer | The result can be applied asynchronously or with a single atomic instruction | | Limit concurrency in front of the bottleneck | Bounded waits, predictable latency for admitted requests | Some requests rejected or delayed at the edge | Arrivals can exceed the serial ceiling | ## Common mistakes - **Mistake:** Using "fast" for both latency and throughput. **Instead:** Say which one: "200 ms per request" or "50 requests per second", and at what concurrency. - **Mistake:** Expecting more servers to make each request faster. **Instead:** More servers add throughput and cut queue time; to cut work time, make the work itself smaller. - **Mistake:** Adding workers when throughput has flattened. **Instead:** Find the resource whose ceiling matches the plateau (slots ÷ hold time) and work on that. - **Mistake:** Reading low CPU as spare capacity. **Instead:** Check saturation of locks, pools and queues; waiting threads use no CPU. - **Mistake:** Quoting latency without saying where the clock starts. **Instead:** State the interval (client to client, arrival to done, pickup to done) and measure queue time separately. - **Mistake:** Treating equal p50 and p95 as proof that two systems behave the same. **Instead:** Compare full histograms, including p99 and the maximum. ## Explaining it in 60 seconds Explaining a flat throughput graph to your team or reviewing your friend's scaling plan, six steps keep it short. (They also work as a system design interview answer.) - **Clarify:** which latency interval, which percentile, what target throughput. - **Assumptions:** work time per request, and any one-at-a-time step. - **Structure:** throughput = min(workers ÷ work time, 1 ÷ serial hold time); latency = work + waiting. - **Deep dive:** Little's law to check numbers; wait metrics to find the serial step. - **Trade-offs:** more workers against a shorter, split or removed critical section. - **Wrap up:** the change and the step test that proves the plateau moved. > **A 60-second answer**Latency is how long one request takes; throughput is how many finish per second. In our simulation each request needs 200 ms of work, so going from 1 to 10 workers raised throughput from 5 to 49.8 requests a second while the median stayed at 200 ms: more lanes, same speed per car. Then I add one shared 50 ms lock. A lock serves one request at a time, so it caps throughput at 1 ÷ 50 ms = 20 a second. Ten workers reached only 19.9, and the median rose to 500 ms: 200 of work, about 249 waiting, 50 holding. Little's law agrees: 10 in flight ÷ 19.9 per second is about half a second. Up to the ceiling, more workers still drain a backlog faster; past it they only move the waiting to the lock. So I find the serial step, measure its wait, and shorten, split or remove it before adding servers. ## Practice **Q (Beginner): Each request takes 100 ms of work and you run 8 independent workers. What are the throughput and the pickup-to-done latency?** 8 ÷ 0.1 s = 80 requests a second, and each request still takes about 100 ms once picked up. **Q (Intermediate): Every request holds a shared lock for 20 ms. What is the maximum throughput with 100 workers?** 1 ÷ 0.02 s = 50 requests a second. The number of workers does not change a single-slot ceiling. **Q (Intermediate): In Run 2, one worker gives 250 ms and ten workers give 500 ms, yet the work did not change. Where did the extra 250 ms go?** Into waiting for the lock. 250 ms = 200 ms work + 50 ms hold with no wait; 500 ms ≈ 200 ms work + about 249 ms average lock wait + 50 ms hold. **Q (Advanced): Your load test shows throughput flat at 33 req/s from 8 to 32 workers, with low CPU. What would you suspect, and what hold time does it imply?** A single-slot serial step, such as a lock or a single-threaded consumer, held about 1 ÷ 33 ≈ 30 ms per request. Look for the wait metric that grows with workers. **Q (Advanced): With the 50 ms lock, why does 10 workers give a p95 identical to its p50, while 1 worker has a 9 ms gap?** The saturated lock grants back-to-back 50 ms slots, so each lane gets one every 500 ms and every request takes exactly one turn; lock waiting absorbs work jitter. At 1 worker nothing waits, so the ±10 ms jitter shows through. **Q (Advanced): In Run 2, pickup-to-done p50 went from 256 to 500 ms between 5 and 10 workers. Did users waiting in the backlog get worse service?** About the same. Pickup-to-done ignores time waiting for a worker, and the batch time barely moved (51.3 to 50.2 s): the waiting shifted from the backlog to the lock. From 1 to 10 workers, by contrast, the batch fell from 250.2 s to 50.2 s, a large real gain. **Q (Advanced): A serial step is 20% of each request's time. What speedup does Amdahl's law allow at most, and why did our simulation reach that limit sooner than Amdahl's curve predicts?** At most 1 ÷ 0.2 = 5 times one worker: 5 × 4 = 20 req/s. Amdahl assumes nothing else runs during the serial part; our other lanes keep working while one request holds the lock, so throughput hit the ceiling at about 5 workers. ## Cheat sheet | Term or formula | Meaning | | --- | --- | | Latency | Time for one request; always say where the clock starts | | Throughput | Requests finished per second | | Latency = work + waiting | More workers change only the waiting term | | X ≈ c ÷ S | Throughput of c independent workers with S seconds of work each | | X ≤ 1 ÷ hold | Ceiling of a single serial step such as a lock | | X ≤ 1 ÷ D_max | Bottleneck bound: the busiest resource sets the path's ceiling | | Amdahl: 1 ÷ (s + (1 − s) ÷ N) | Speedup with serial fraction s; never more than 1 ÷ s | | L = λ × W | Little's law: requests inside = throughput × time inside | | U = λ × S | Utilisation of a single-slot resource; waits grow fast near 1 | | p50, p95 | Latency that 50% and 95% of requests beat; not a full distribution | | Lock contention | Requests ask for a lock faster than it is granted; shows as wait time, not CPU | | USE method | For every resource check utilisation, saturation and errors, locks included | | Step test | Raise concurrency in steps; the plateau's level points at the bottleneck | ## Further reading - **Brendan Gregg, The USE Method:** a checklist for finding saturated resources, mutex locks included. - **Little (1961), Operations Research:** the original proof that L = λW holds for any stable queue. - **Quantitative System Performance (Lazowska et al.):** the operational laws and bottleneck bounds used in this article, free online. - **Google SRE book, Monitoring Distributed Systems:** the four golden signals (latency, traffic, errors, saturation) and why tails beat means. - **PostgreSQL, Explicit Locking:** table and row lock modes, which ones conflict, and how long they are held. - **Amdahl's law (Wikipedia):** the speedup formula for a serial fraction, with its assumptions. Next in the series, part 7: the cable to Asia breaks, and your bank must choose. This is a simplified, general explanation of common patterns, not a description of any company's private systems. All measured numbers come from our discrete-event simulation on a virtual clock (Node.js v22.13.1, seed 42, 1,000 requests per run, latency timed from pickup to done). It models queueing only, with no real threads, database or network, and does not represent the capacity of any production system. ## Frequently asked questions **What is the difference between latency and throughput?** Latency is the time one request takes from start to finish. Throughput is how many requests the system finishes per second. They are related but independent: you can raise throughput by doing more requests in parallel without making any single request faster, and a serial step can make latency worse while throughput stays flat. **Do more servers or threads reduce latency?** Not the work part. Each request still needs its own work time, 200 ms in our simulation, however many workers run. More workers reduce latency only by reducing waiting: the time a request spends in line before a worker picks it up. Past a serial bottleneck they stop doing even that. **How does a lock limit throughput?** Only one request can hold a lock at a time, so if each request holds it for 50 ms, at most 1 ÷ 0.05 s = 20 requests finish per second. Extra workers arrive at the lock faster than it can serve them, and the surplus becomes waiting time. In our simulation 10 workers reached 19.9 req/s with about 249 ms of average lock wait. **Why are p50 and p95 latency both reported?** The p50 (median) is what a typical request sees; the p95 is the latency that 95% of requests beat, so it shows the slow tail. Averages hide that tail. Two systems can print the same p50 and p95 and still differ at p99 or in the maximum, so percentiles describe a distribution only partly. **How does Little's law connect latency and throughput?** Little's law says the average number of requests in a system equals throughput times average time in the system: L = λW. With 10 workers always busy at 49.8 requests a second, W ≈ 10 ÷ 49.8 ≈ 0.2 s. With the lock, 10 ÷ 19.9 ≈ 0.5 s. If the work in progress is fixed, lower throughput means higher latency. **Were these numbers measured on real servers?** No. They come from a discrete-event simulation on a virtual clock (Node.js v22.13.1, seed 42, 1,000 requests per run), with no real threads, database or network. It shows the shape of the effect, not the capacity of any production system. ## Sources - [Brendan Gregg: The USE Method](https://www.brendangregg.com/usemethod.html) - [Little, J. D. C. (1961): A Proof for the Queuing Formula L = λW, Operations Research](https://pubsonline.informs.org/doi/10.1287/opre.9.3.383) - [Lazowska et al.: Quantitative System Performance (free online book)](https://homes.cs.washington.edu/~lazowska/qsp/) - [Google SRE book: Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/) - [PostgreSQL documentation: Explicit Locking](https://www.postgresql.org/docs/current/explicit-locking.html) - [Wikipedia: Amdahl's law](https://en.wikipedia.org/wiki/Amdahl%27s_law)