# Incremental config builds: rebuild only what changed, and prove it matches

URL: https://systemdesignsimple.com/rebuild-only-the-zone-that-changed/
Published: 2026-11-11
Updated: 2026-11-12
Category: System Design
Publisher: System Design Simple (https://systemdesignsimple.com)

## Summary

A full rebuild costs O(object size) per change. A targeted build maps each change to the smallest set of output keys it can affect, merges pending work per object, and falls back to a full build when it cannot prove the smaller one is safe. Cloudflare reports 92.95% of DNS zone builds now avoid a full rebuild and build memory fell 85% at peak.

## Key facts

- **Core idea:** Cost of a full rebuild is O(size of object); cost of a targeted build is O(size of change) plus a fixed overhead
- **Correctness rule:** Every key a targeted build writes must be byte-identical to what a full build would write from the same state
- **Fail-safe default:** Unknown, incomplete or timestamp-only events escalate to a full build, never to a smaller one
- **Reported build mix (Cloudflare, 24 h):** Resources only 86.81%, resources and metadata 2.54%, apex and metadata 2.30%, purge 1.30%, full zone 7.04%
- **Reported memory (Cloudflare):** Average working set 49 GiB to 7 GiB (86% less), peaks 79 GiB to 12 GiB (85% less)
- **Reported delay (Cloudflare):** Median build delay about 7.7 s to 3.3 s; largest p99 spike 14.6 min to 1.21 min
- **Offset safety:** Builds may finish out of order, but the committed Kafka offset only advances past a contiguous run of finished messages

When you push configuration to many servers, the simplest builder regenerates the whole object on every change and diffs the result, so the cost of a one-line edit grows with the size of the object instead of the size of the edit. Incremental builds fix that by tracking what changed, rebuilding only the outputs it can affect, and falling back to a full rebuild whenever the smaller build cannot be proven correct. Interview form: how would you propagate config changes to thousands of servers when some objects hold millions of entries?

## Prerequisites {#prerequisites}

**Authoritative DNS.** A zone is the set of DNS records one domain owner controls, such as every name under `example.com`. An authoritative server answers queries for that zone directly. The resolution path that ends at those servers is in [what happens when you type google.com](/you-type-google-com-100-ms/). For this article, all that matters is that every edge server needs a current, pre-computed copy of each zone's answers.

**Change logs and replicas.** Databases and brokers publish changes as an ordered stream: PostgreSQL's write-ahead log, MySQL's binlog, a Kafka topic. Consumers that replay the stream in order converge on the same state, a little later. We covered lag and stale reads in [one database writes, ten read](/one-database-writes-ten-read/), and both matter here because a builder that reads a lagging replica builds old state.

**Queues and backlog.** If events arrive faster than a consumer finishes them, the backlog grows without limit and delay grows with it. The queueing behaviour is in [fast alone, slow in a crowd](/fast-alone-slow-crowded/). Full rebuilds of big objects are exactly the kind of long, variable service time that turns a busy queue into a slow one.

## Definitions {#definitions}

> Definition: **Full rebuild (snapshot build)**: regenerate every output for an object from its complete current input, then diff against the stored output and write the difference. It does not need to know why the object changed.

> Definition: **Incremental (targeted) build**: use the change itself to choose which outputs can be affected, and regenerate only those from current input.

> Definition: **Delta propagation**: shipping only changed keys or entries to downstream servers instead of a whole new snapshot.

> Definition: **Dirty set**: the set of outputs (or input keys) marked as needing a rebuild since the last build. Merging events into a dirty set lets ten edits to one object cost one build.

> Definition: **Escalation**: replacing a smaller planned build with a larger one (up to a full rebuild) because the smaller one cannot be proven complete.

> Definition: **Equivalence invariant**: for the same input state, the incremental build must produce output byte-identical to a full build. This is the property you test, shadow and monitor.

> Definition: **Soft commit**: tracking finished messages in memory and committing a broker offset only past the longest contiguous run of finished messages, so out-of-order completion never skips an unfinished one.

## How it works: from change event to the smallest correct build {#how-it-works}

A config pipeline has four stages, and an incremental design changes what each one carries.

1. **Write.** A user edits a record or setting. The database commits it and a change event is produced, either by the application or by change data capture from the log.
2. **Transport.** Events travel through a log such as Kafka, ordered per partition. Partitioning by object key keeps all events for one object in order.
3. **Build.** A builder reads current state and converts it into the serving format: here, pre-computed answers keyed by zone and name, plus a small metadata key per zone.
4. **Distribute.** Changed output keys go into a replicated key-value store that every edge server reads locally.

In a full rebuild design the event only needs to say "zone 42 changed". The builder reads the zone, its settings, and every record, encodes all of them, lists all existing keys for that zone in the store, deletes keys that are no longer produced, and writes keys whose bytes differ. That is simple and self-healing: any drift in the store is corrected on the next build. It is also O(N) in the zone size for every single change, in CPU, database reads and memory, because the whole zone is materialised to diff it.

A targeted design needs three new pieces.

### 1. Events that say what changed

The event must carry which fields changed and, ideally, their old and new values. Without the old value you cannot clean up: if a record moves from `a.example.com` to `b.example.com`, you need to rebuild both names, and only the old value tells you `a` existed. A missing or vague event is not "probably small"; it is unknown, and unknown must escalate to full.

### 2. A dependency map from inputs to outputs

For each input field you need the set of output keys it can influence. Records are easy: a record under one name affects the output key for that name. Settings are the hard part. A setting can affect a single special name (the zone apex), the shared metadata key, or every name in the zone. Building this map means tracing each setting through the output code. Get it wrong in the "too small" direction and you silently serve stale data, which is why the default for an unclassified setting is a full build.

This is the same problem a build system like GNU Make solves with a dependency graph and file timestamps: a target is dirty if any prerequisite is newer ([GNU Make manual](https://www.gnu.org/software/make/manual/make.html#Rule-Introduction)). Make's classic failure, a missing header in the dependency list so an object file is not rebuilt, is the exact analogue of a missing setting in your classification table.

### 3. A build plan that merges

Represent work per object as a small plan: a set of names, plus flags for apex, metadata, purge and full. Merging two plans is a join on a simple lattice:

```text
merge(a, b):
  if a.purge or b.purge  -> purge          (object should not be published at all)
  if a.full  or b.full   -> full           (anything full makes the merged build full)
  else names = a.names ∪ b.names
       apex  = a.apex or b.apex
       meta  = a.meta or b.meta
```

The merge is commutative, associative and idempotent, which is what you want: the result does not depend on arrival order, and replaying an event after a crash does not change the plan. A queue of 50 events for one zone collapses into one plan and one build. That collapse is the dirty-set idea, and it is also the main defence against a hot object flooding the builder, the same batching trick behind [counting a million likes at once](/million-likes-at-once/).

### Building a targeted plan

For each name in the plan, read its current records, encode the key, and compare with the store. Deletion needs care. A targeted build cannot list the whole zone (that would be O(N) again), so it scans the store only under each rebuilt name and deletes keys there that the new build does not produce. Metadata, when flagged, is rebuilt as a whole key from a fresh read, never patched, so it cannot accumulate partial updates.

Write ordering matters when one key refers to another. If metadata points at records, write the records first and the metadata last. A reader that sees the new metadata then always finds what it points to. This is the same discipline as writing data pages before the commit record in a write-ahead log.

## What Cloudflare reports about targeted DNS zone builds {#source}

Cloudflare's write-up, [How we halved DNS build delay and cut memory use by 85%](https://blog.cloudflare.com/dns-targeted-zone-builds/), describes moving their DNS Zone Builder from full zone builds to targeted builds. Everything in this section is according to their write-up; we add nothing about their internals beyond it.

**The pipeline.** A record or zone setting change is stored in a database, which produces a Kafka event. A DNS Kafka Aggregator merges events from several topics and passes them to the Zone Builder, which converts zone data into keys and writes only changed keys to Quicksilver, their distributed key-value store (described in their [Quicksilver post](https://blog.cloudflare.com/introducing-quicksilver-configuration-distribution-at-internet-scale/)). A 2022 change had already made per-record builds possible; the write-up recalls those being 150 times faster than full builds on average and 4,250 times faster on a one-million-record test zone ([2022 post](https://blog.cloudflare.com/dns-build-improvement/)).

**Why full builds remained a problem.** Zone setting changes could not be mapped to specific record names, so they still triggered full builds. The write-up says the builder now sees about 700 record changes per second plus about 100 other events, and some zones hold more than 20 million records, which does not fit in memory for a full build at all.

**The design.** Events now carry the changed zone fields and settings, with previous and new values when available. Each setting was traced through the output code. There are five build types: resources only, resources and metadata, apex and metadata, purge, and full zone. Timestamp-only updates, missing details and unknown events still trigger a full build. The correctness rule is that every key a targeted build writes must be byte-identical to what a full build would produce, and old keys are found by scanning the store under each rebuilt name.

**Freshness and races.** Planning happens twice. The event produces a conservative work list, and just before running, the builder reads a fresh publication snapshot from read replicas and may escalate to full, convert to a purge, skip, or rebuild the metadata key from current candidates. A final database check rejects reads from replicas that are still behind, and the aggregator delays events to cover normal replication lag. Several internal zone objects can share one domain name during lifecycle changes, so metadata is always rebuilt from a fresh read of the current candidates, and the triggering zone's records are written before the metadata that refers to them.

**Scheduling and Kafka.** A planner decides what to build and a scheduler decides when. Work for the same zone is merged; any full requirement makes the merged build full, and a purge overrides rebuilds. Builds may finish out of order, but the committed offset only advances once all earlier messages have finished, so a restart replays work rather than losing it. Above a record-count threshold (the value is not given), the builder never escalates to full: it rebuilds the apex, metadata and queued names instead, because a full build of such a zone cannot finish and retries would hold back the partition.

**Rollout.** A mirror Zone Builder consumed production Kafka and production database state but wrote to a separate Quicksilver environment serving no traffic. A comparer checked a 20% sample of zones for missing keys, extra keys and value differences. The scheduler was enabled by hashing the zone name rather than the zone ID, so all zones sharing a name stayed on one scheduler and two schedulers never wrote the same metadata key.

**Reported results:**

| Metric | Before | After |
| --- | --- | --- |
| Builds that avoid a full rebuild | about 88% (derived) | 92.95% |
| Full builds as share of all builds | about 11.88% (derived) | 7.04% (about 40% fewer) |
| Aggregate working-set memory, average | 49 GiB | 7 GiB (86% less) |
| Aggregate working-set memory, peak | 79 GiB | 12 GiB (85% less) |
| Median build completion delay | about 7.7 s | about 3.3 s |
| Largest p99 delay spike | 14.6 min | 1.21 min |
| Kafka consumption vs production | 300 to 1,550 msg/s vs 600 to 800 | follows production at 600 to 800 |
| Mirror diff rate | 0.31% across 234,313 matched zones |  |

The 24-hour build mix was 86.81% resources only, 2.54% resources and metadata, 2.30% apex and metadata, 1.30% purge and 7.04% full zone. The write-up notes the median halved through the aggregator change plus shorter rebuild intervals, not targeted builds alone, and that it excludes Quicksilver distribution time from the delay figures.

## Why full rebuilds hurt: a cost model {#cost-model}

Let an object have N entries and an event change k of them. Let c be the cost to read, encode and compare one entry, and F a fixed overhead per build (connection, metadata reads, store round trips).

```text
cost_full     = F + c × N
cost_targeted = F + c × k + s × k          (s = cost of a prefix scan per rebuilt name)

speedup = (F + cN) / (F + (c + s)k)   →   roughly N / k when cN dominates
```

With a mix where a fraction p of builds must stay full, the expected cost per event is:

```text
E[cost] = p × (F + cN) + (1 − p) × (F + (c + s)k)
```

Two things follow. First, the full-build term dominates even at small p when N is large. For N = 1,000,000 and k = 1, ignoring F and s, a p of 1% already makes full builds 99.99% of the work; with the assumed F and s from the worked example below, it is still about 80%. Reducing p (the share Cloudflare cut by about 40%) matters more than making targeted builds faster. Second, the distribution of N matters more than its mean. Object sizes are usually heavy tailed: most zones have a handful of records and a few have millions. A builder's worst-case delay is set by the tail.

**Memory** follows the same shape. A full build holds the new output and the old key list in memory to diff them, so peak memory per worker is roughly `2 × N_max × bytes_per_entry`, and with W concurrent workers the fleet peak approaches `W × 2 × N_max × bytes_per_entry` whenever several large objects build at once. Targeted builds bound that by k, which is why a peak reduction (79 to 12 GiB) can be as large as the average one.

**Head-of-line blocking.** Kafka orders messages within a partition, and a consumer that commits offsets in order cannot move past an unfinished message. One 10-minute full build of a giant zone holds back every later event on that partition, even small ones for other zones. Service time variance is what kills queue latency: the Pollaczek-Khinchine formula for an M/G/1 queue gives mean wait

```text
W_q = λ × E[S²] / (2 × (1 − ρ))
```

so wait grows with the second moment of service time, not just the mean. A rare 600 s build among 0.05 s builds inflates E[S²] enormously. Removing the long builds shrinks the tail far more than the median, which matches the shape of the reported numbers (p99 spike down 12-fold, median roughly halved).

## Where the same pattern lives in Envoy, Redis, Kubernetes and Kafka {#elsewhere}

Almost every system that pushes state to many readers ends up with "deltas normally, snapshot as fallback". The details are worth knowing because each one solves a failure mode you will hit.

**Envoy xDS.** Envoy's control plane protocol has two variants ([xDS protocol docs](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol)). In state-of-the-world xDS, each response for a resource type such as clusters or listeners carries the full set, so one change to one cluster resends all of them. Incremental (delta) xDS sends only added or changed resources plus a list of removed names, and the client tracks versions per resource. The motivation in the docs is the same as here: with very large resource sets, full updates cost bandwidth and CPU on both sides. Note the explicit removal list: deltas must carry deletes, or stale entries live forever.

**Redis replication.** A Redis replica normally receives a stream of write commands. When it reconnects, it sends `PSYNC` with the replication ID and the offset it has processed. If the master's in-memory replication backlog still covers that offset, it sends only the missing part (partial resync). If not, the master produces an RDB snapshot and sends the whole dataset (full sync) ([Redis replication docs](https://redis.io/docs/latest/operate/oss_and_stack/management/replication/)). The backlog size, `repl-backlog-size`, is the knob: too small and short network blips cause full syncs of the entire dataset. Size it as at least `write rate in bytes/s × longest disconnect you want to survive`.

**Kubernetes watch.** Clients list a resource type once, then watch from that list's `resourceVersion` and receive add, modify and delete events. If the watch falls too far behind and the API server no longer holds history for that version, it returns `410 Gone` and the client must re-list everything ([Kubernetes API concepts: efficient detection of changes](https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes)). Informer libraries also re-list periodically as a safety net. That is the incremental build pattern exactly: deltas while the history is intact, snapshot on a gap, periodic reconciliation for drift.

**Kafka offsets.** A consumer's committed offset is the position of the next message it will read after a restart ([Kafka design: consumer position](https://kafka.apache.org/documentation/#design_consumerposition)). Committing after processing gives at-least-once delivery: a crash between processing and commit means the message is processed again. That is why builds must be idempotent. With parallel, out-of-order processing inside one partition, you must track completions and commit only the lowest unfinished offset, which is what the write-up's soft commit does.

**Nginx reload, the full snapshot model.** For contrast, `nginx -s reload` parses the whole configuration, starts new worker processes with it, and gracefully retires the old ones ([nginx: controlling nginx](https://nginx.org/en/docs/control.html)). It is a full snapshot every time. That is fine for files with hundreds of server blocks; it is why very large or very dynamic route tables move to an API-driven model instead of files.

## Worked example {#worked-example}

The only real figures below are the reported Kafka rates; everything else is labelled as assumed.

```text
Backlog when consumption falls behind (rates from the write-up's ranges, duration assumed)
  production  = 700 msg/s           consumption = 300 msg/s  (slow period)
  slow period = 10 min = 600 s      (assumed)
  backlog     = (700 − 300) × 600 = 240,000 messages

Drain time once the builder runs at its best reported rate
  net drain   = 1,550 − 700 = 850 msg/s
  drain time  = 240,000 / 850 ≈ 282 s ≈ 4.7 min
  worst delay for the last event in the backlog ≈ 600 + 282 = 882 s ≈ 14.7 min

Per-build cost for one record change in a 1,000,000-record zone (assumed costs)
  c = 2 µs per record, s = 50 µs per prefix scan, F = 5 ms
  full     = 5 ms + 2 µs × 1,000,000 = 2.005 s
  targeted = 5 ms + (2 + 50) µs × 1   ≈ 5.05 ms        speedup ≈ 397x

Expected cost per event at p = 7% full vs p = 11.88% full (same assumed zone)
  p = 0.1188:  0.1188 × 2.005 + 0.8812 × 0.00505 ≈ 0.243 s
  p = 0.0704:  0.0704 × 2.005 + 0.9296 × 0.00505 ≈ 0.146 s     (about 40% less work)
```

Notice that the 882 s worst-case delay in the backlog scenario lands close to the reported 14.6-minute p99 spike. That is a coincidence of the assumed 10-minute window, but it shows how a few minutes of under-consumption alone produces a double-digit-minute tail. Also notice the speedup is about 400x, not N/k = 1,000,000x: once full-build work is gone, the fixed overhead F dominates. The 2022 figures of 150x average and 4,250x on a large zone have the same shape.

## Code {#code}

A small, self-contained model: a 50,000-name zone, a full builder, a targeted builder with a merging planner, an equivalence check against a fresh full build, and a soft-commit offset tracker. Standard library only.

```python
# incremental_lab.py: full build vs targeted build, plan merging, soft commits
import json, random

random.seed(7)
ZONE = "example.com"

# Source of truth: zone settings plus name -> records
db = {ZONE: {"settings": {"ns": ["ns1", "ns2"], "caa": "ca.example"},
             "names": {f"h{i}.{ZONE}": [f"10.0.{i // 256}.{i % 256}"] for i in range(50_000)}}}

def encode(zone, name, recs):            # one output key per (zone, name)
    return f"{zone}|{name}", json.dumps(sorted(recs)).encode()

def apex_and_meta(zone):
    s = db[zone]["settings"]
    return {f"{zone}|@": json.dumps({"caa": s["caa"]}).encode(),
            f"{zone}|meta": json.dumps({"ns": s["ns"]}).encode()}

def full_build(zone):                    # O(N): read and encode every name
    out = dict(encode(zone, n, r) for n, r in db[zone]["names"].items())
    out.update(apex_and_meta(zone))
    return out

store = {}                               # stands in for the replicated key-value store

def apply(zone, new_keys, scope=None):   # scope=None: the whole zone is in scope
    prefix = zone + "|"
    old = {k for k in store if k.startswith(prefix)} if scope is None else {k for k in store if k in scope}
    for k in old - new_keys.keys():      # delete keys the new build no longer produces
        del store[k]
    writes = 0
    for k, v in new_keys.items():        # write only keys whose bytes changed
        if store.get(k) != v:
            store[k] = v
            writes += 1
    return writes

def plan(full=False, purge=False, meta=False, apex=False, names=()):
    return {"full": full, "purge": purge, "meta": meta, "apex": apex, "names": set(names)}

def merge(a, b):                         # purge beats full, full beats everything else
    if a["purge"] or b["purge"]:
        return plan(purge=True)
    if a["full"] or b["full"]:
        return plan(full=True)
    return plan(meta=a["meta"] or b["meta"], apex=a["apex"] or b["apex"], names=a["names"] | b["names"])

def plan_for(event):
    kind = event.get("kind")
    if kind == "record":
        return plan(names=[event["name"]])
    if kind == "ns":
        return plan(meta=True)
    if kind == "caa":
        return plan(apex=True, meta=True)
    return plan(full=True)               # unknown or incomplete event: fail safe

def targeted_build(zone, p):
    if p["purge"]:
        return apply(zone, {}), "purge"  # remove every key for the zone
    if p["full"]:
        return apply(zone, full_build(zone)), "full"
    keys, scope = {}, set()
    for n in p["names"]:
        k, v = encode(zone, n, db[zone]["names"].get(n, []))
        scope.add(k)
        if n in db[zone]["names"]:       # a deleted name stays in scope so its key is removed
            keys[k] = v
    if p["apex"] or p["meta"]:
        am = apex_and_meta(zone)
        keys.update(am)
        scope |= am.keys()
    return apply(zone, keys, scope), "targeted"

# 1) First publication is always a full build
print("initial writes:", apply(ZONE, full_build(ZONE)))

# 2) A burst of events for one zone, merged into one plan (the dirty set)
events = [{"kind": "record", "name": f"h{random.randrange(2, 50_000)}.{ZONE}"} for _ in range(5)]
for e in events:
    db[ZONE]["names"][e["name"]] = ["192.0.2.1"]
db[ZONE]["settings"]["caa"] = "other-ca.example"
events.append({"kind": "caa"})
del db[ZONE]["names"][f"h1.{ZONE}"]
events.append({"kind": "record", "name": f"h1.{ZONE}"})

p = plan_for(events[0])
for e in events[1:]:
    p = merge(p, plan_for(e))
writes, mode = targeted_build(ZONE, p)
print(f"merged {len(events)} events -> {mode} build, {len(p['names'])} names, {writes} key writes")

# 3) Equivalence check: the targeted result must be byte-identical to a fresh full build
expected = full_build(ZONE)
actual = {k: v for k, v in store.items() if k.startswith(ZONE + "|")}
print("byte-identical to full build:", actual == expected)

# 4) Soft commit: builds finish out of order; commit only the contiguous finished prefix
committed, done = 100, set()             # Kafka semantics: committed = next offset to read
for off in [101, 103, 100, 102, 105]:
    done.add(off)
    while committed in done:
        done.remove(committed)
        committed += 1
    print(f"finished {off}: commit offset -> {committed}")
```

What to observe when you run it with `python3 incremental_lab.py`:

- The first publication writes 50,002 keys (50,000 names plus apex and metadata).
- Seven events (five record edits, one CAA change, one delete) merge into one targeted build covering 6 names. It writes 6 keys: five records and the apex. The metadata key is regenerated but not written, because its bytes did not change, and `h1` is deleted, not written.
- The equivalence check prints `True`. Now break it: change `plan_for` so `"caa"` returns `plan()`, an empty plan, rerun, and it prints `False` because the apex key keeps the old CAA value. That is the classification bug a shadow comparer exists to catch.
- The commit offset stays at 100 until offset 100 finishes, then jumps to 102, then to 104, and stays at 104 when 105 finishes because 104 is still open.

## Failure modes and edge cases {#failure-modes}

**Under-classified dependency.** A setting that actually affects every name is mapped to "apex only". Output is silently stale for every other name. *Detect:* a shadow builder doing full builds from the same events, diffing against production; equivalence tests per setting. *Fix:* default unclassified settings to full, and add a test whenever a setting is added to the output code.

**Building from a lagging replica.** The event arrives before the replica has the write, the builder reads old state and writes old bytes, and nothing triggers another build. The store is wrong until the next unrelated change. *Detect:* compare the replica's replay position with the position of the change (in PostgreSQL, `pg_last_wal_replay_lsn()` on the standby against the commit LSN carried in the event). *Fix:* delay events by the normal lag, check freshness before reading, and requeue if behind, as the write-up describes.

**Orphaned keys.** A record is renamed or deleted, the build only regenerates the new name, and the old key keeps answering. *Detect:* periodic full scans counting keys with no source row. *Fix:* carry old values in events and always include the old name in the rebuild scope; deltas must carry deletes (Envoy's delta xDS has an explicit removal list for this reason).

**Torn reads across keys.** A reader sees new metadata that points at records not written yet. *Fix:* write dependents before the key that references them; for multi-key atomicity, version the metadata and write records under the new version first.

**Two writers for one shared key.** Two builder instances, each handling a different object that shares a name, both rewrite the shared metadata key and the last write wins with a partial view. *Fix:* partition work by the key that owns the shared output (the write-up hashes the zone name, not the zone ID), and always rebuild shared keys from a fresh read rather than patching.

**Gap in the change stream.** Events are dropped (retention expired, a consumer group reset, a bug in the producer). Incremental systems never notice on their own. *Detect:* sequence numbers per object, or periodic full reconciliation with a diff counter as a metric. *Fix:* fall back to snapshot, the way Redis full sync and the Kubernetes `410 Gone` re-list do.

**Head-of-line blocking.** One slow build holds the committed offset, so every later message on the partition waits and restarts replay all of it. *Detect:* consumer lag per partition (Kafka's `records-lag-max`) and age of the oldest unfinished offset. *Fix:* bound the build size (the write-up's large-zone policy), let old queued work run ahead of per-object rate limits, and split hot objects across more partitions where ordering allows.

**Poison event.** An event that fails every time blocks the partition forever. *Fix:* bounded retries, then a dead-letter topic plus an alert, and a full rebuild of the affected object once fixed.

## Trade-offs {#trade-offs}

| Option | Good for | Costs you | Use when |
| --- | --- | --- | --- |
| Full snapshot rebuild per change | Simplicity, self-healing, no dependency map | O(N) CPU, reads and memory per change; long-tail delay; head-of-line blocking | Objects are small (thousands of entries) or changes are rare |
| Targeted build per change | O(k) cost; flat memory; low tail latency | Rich events, a correct dependency map, equivalence testing, shadow infrastructure | Large or heavy-tailed objects with frequent small changes |
| Targeted plus merged dirty set | Collapses bursts into one build per object | Added delay up to the merge window; more scheduler state | Hot objects receive many edits per second |
| Ship raw deltas straight to servers | Lowest latency, no builder | Output format coupled to input schema; derived keys hard; no single place to compare | Output equals input (key-value copy), as in database replication |
| Periodic full reconciliation | Repairs drift from bugs and lost events | Background load proportional to total data | Always, alongside any incremental scheme, at low priority |

The number that drives the choice is N_max / k_typical together with event rate. If your largest object rebuilds in under 100 ms and events are tens per second, a full rebuild is the cheaper system to own. Once a single full build exceeds your latency target, or can no longer fit in memory, incremental is no longer an optimisation but the only option.

## Common mistakes {#common-mistakes}

- **Mistake:** Treating an event with no field details as "probably a small change". **Instead:** Missing information means unknown scope; escalate to a full build.

- **Mistake:** Patching shared keys (append this zone ID to the metadata list). **Instead:** Rebuild the whole shared key from a fresh read of its current inputs, so concurrent or replayed builds converge.

- **Mistake:** Shipping deltas without deletes. **Instead:** Carry old values in events, keep removed names in the rebuild scope, and delete keys the new build no longer produces.

- **Mistake:** Committing the Kafka offset of whichever build finished last. **Instead:** Commit only past the contiguous run of finished messages, and make builds idempotent so replays are harmless.

- **Mistake:** Removing full rebuilds entirely once incremental works. **Instead:** Keep them for first publication, broad-impact changes and scheduled reconciliation, and alert on reconciliation diffs.

- **Mistake:** Trusting unit tests alone for the dependency map. **Instead:** Run a shadow pipeline on production events that writes to a separate store and diffs a sample of objects, and watch for new, persistent difference types.

## In an interview {#in-an-interview}

**Clarify:** how many objects, the distribution of object sizes, change rate, freshness target, and whether servers read a key-value store or load files. **Assumptions:** state them, for example 1 million zones, a heavy tail up to 20 million records, 1,000 events per second, a 5-second p99 target. **Structure:** database, change events on Kafka partitioned by object, builder, replicated key-value store at the edge. **Deep dive:** event content (old and new values), dependency map, plan merging, fresh reads, write ordering, soft commits. **Trade-offs:** escalation rules, large-object policy, reconciliation cost. **Wrap up:** how you would validate (shadow builder and comparer) and what you would monitor (diff rate, consumer lag, oldest unfinished offset, share of full builds).

> **A 60-second answer**Changes land in the database and produce events on Kafka, partitioned by object so each object's events stay ordered. A builder turns current state into serving keys and writes only changed keys to a replicated store that every server reads locally. To avoid rebuilding a 20-million-entry object for a one-line edit, events carry old and new values, and a dependency map turns each change into the smallest set of output keys it can affect. Pending work per object merges into one plan; anything unknown or broad escalates to a full build. Before building I re-read fresh state and reject lagging replicas, rebuild shared keys whole from that read, and write dependents before the keys that point at them. Offsets commit only past contiguous finished messages, so a crash replays instead of losing work. I prove correctness with tests and a shadow builder that diffs against full builds, and I keep periodic full reconciliation to repair drift.

## Practice {#practice}

**Q (Beginner): Why does a full rebuild get slower as the object grows, even when the change is one line?**
It reads, encodes and compares every entry in the object to find the difference, so its cost is O(N) in the object size regardless of how small the change is.

**Q (Intermediate): An event says "zone 42 updated" with no field list. What should the planner do, and why?**
Escalate to a full build. Without knowing which fields changed, any output could be affected; a smaller build risks silently stale data, while a full build only costs time.

**Q (Intermediate): A record is renamed from a.example.com to b.example.com. Which names must the targeted build cover?**
Both. The new name needs its key written and the old name needs its key deleted. This is why events must carry the previous value.

**Q (Intermediate): Builds for offsets 7, 9 and 10 finish; 8 is still running. The last committed offset is 7. What do you commit?**
8. Kafka's committed offset is the next message to read; 7 is done so the commit can move to 8, but not past 8 while it is unfinished. If the process crashes, 8, 9 and 10 replay, so builds must be idempotent.

**Q (Advanced): Full builds are 7% of builds but objects are heavy tailed. Why can they still dominate CPU, and what do you change first?**
Expected cost is p × cN + (1 − p) × ck; with N in the millions, even a small p makes the first term most of the work. Reduce p by classifying more settings precisely, and cap N with a large-object policy, before optimising the targeted path.

**Q (Advanced): Your builder reads from a read replica. Describe the race and two defences.**
The event can arrive before the replica applied the write, so the builder writes old state and no later event fixes it. Delay events by typical lag, and check the replica's replay position against the change's commit position, requeuing if behind.

**Q (Advanced): Your Redis replicas keep doing full syncs after short network blips. What is happening and how do you size the fix?**
The replica's offset falls outside the master's replication backlog, so PSYNC cannot do a partial resync. Raise repl-backlog-size to at least write bytes per second times the longest disconnect you want to survive.

**Q (Advanced): How do you roll out an incremental builder without risking production output?**
Run a mirror builder on production events and state that writes to a separate store serving no traffic, diff a sample of objects against production for missing, extra and changed keys, track the diff rate, and enable the new path gradually by a hash of the key that owns shared outputs.

## Cheat sheet {#cheat-sheet}

| Term or formula | Meaning |
| --- | --- |
| Full rebuild | Regenerate all outputs from complete input, diff, write changes |
| Targeted build | Regenerate only outputs the change can affect |
| Dirty set | Outputs marked for rebuild since the last build; merges bursts |
| cost_full = F + cN | Grows with object size |
| cost_targeted = F + (c + s)k | Grows with change size; F dominates for tiny changes |
| E[cost] = p(F + cN) + (1 − p)(F + (c + s)k) | Share of full builds p drives total work |
| W_q = λE[S²] / (2(1 − ρ)) | Queue wait grows with service time variance |
| Equivalence invariant | Targeted output byte-identical to full output for the same state |
| Escalation | Unknown or broad change becomes a bigger build, never a smaller one |
| Soft commit | Commit offset only past contiguous finished messages |
| Delta xDS / PSYNC / watch + 410 | Deltas normally, snapshot when history is missing |
| Reconciliation | Periodic full compare that repairs drift |

## Further reading {#further-reading}

- **Cloudflare, How we halved DNS build delay and cut memory use by 85%:** the original write-up, with the five build types, two-phase planning, soft commits, mirror rollout and all the numbers quoted here.
- **Cloudflare, DNS record build speed (2022):** the earlier step that made per-record builds possible, useful for seeing why settings were the remaining hard case.
- **Cloudflare, Introducing Quicksilver:** how the replicated key-value store that receives the changed keys distributes configuration to edge servers.
- **Envoy xDS protocol:** the clearest public specification of state-of-the-world versus incremental updates, including removal lists and per-resource versions.
- **Redis replication:** partial resync, the replication backlog and when a full sync happens, the canonical delta-or-snapshot fallback.
- **Kafka documentation, consumer position:** what a committed offset means and why commit-after-processing gives at-least-once delivery.

This article explains general patterns for incremental configuration builds. Statements about Cloudflare's system are limited to what their public write-up states; the cost model, worked numbers and code are our own illustrations, not measurements of any company's private systems.

## Frequently asked questions

**What is the difference between an incremental build and a full rebuild?**
A full rebuild reads all the input for an object, regenerates every output, and diffs it against what is stored. An incremental build uses the change event to decide which outputs can be affected and regenerates only those. Both should produce identical bytes; the incremental one just touches far less data.

**When should an incremental system fall back to a full rebuild?**
Whenever it cannot prove the smaller build is complete: unknown event types, events missing their old and new values, changes to settings that can affect any output, first publication, and gaps in the change stream. Escalating costs time; under-building silently serves wrong data.

**Why not just send the diff from the database straight to the servers?**
Because the output format is usually not the input format. One setting can change many output keys, many rows can feed one output key, and shared metadata depends on several objects. A build step that reads current state and regenerates the affected keys keeps the output deterministic and lets you compare it with a full build.

**How do you know an incremental build is correct?**
Run both. Unit tests compare targeted and full builds from the same pinned state, and in production a shadow builder consumes the same events, writes to a separate store, and a comparer reports missing keys, extra keys and value differences for a sample of objects.

**How does Kafka fit into an incremental build pipeline?**
Kafka carries the change events in order per partition. The builder can merge and run work out of order, but it must only commit an offset once every earlier message on that partition is finished. After a crash, Kafka replays from the last commit, so builds must be idempotent.

**Do incremental builds remove the need for full rebuilds?**
No. You still need full builds for first publication, for changes that can touch any output, and as a periodic reconciliation to repair drift from bugs or lost events. In Cloudflare's report about 7% of zone builds were still full builds.

## Sources

- [Cloudflare blog: How we halved DNS build delay and cut memory use by 85%](https://blog.cloudflare.com/dns-targeted-zone-builds/)
- [Cloudflare blog: How we improved DNS record build speed by more than 4,000x (2022)](https://blog.cloudflare.com/dns-build-improvement/)
- [Cloudflare blog: Introducing Quicksilver, configuration distribution at internet scale](https://blog.cloudflare.com/introducing-quicksilver-configuration-distribution-at-internet-scale/)
- [Envoy docs: xDS protocol, state of the world and incremental (delta) xDS](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol)
- [Redis docs: Replication (partial resync, replication backlog, full sync)](https://redis.io/docs/latest/operate/oss_and_stack/management/replication/)
- [Apache Kafka documentation: consumer position and offset commits](https://kafka.apache.org/documentation/#design_consumerposition)