# Context window: why a long AI chat forgets your name
URL: https://systemdesignsimple.com/my-chat-forgot-my-name/
Published: 2026-11-05
Category: AI Systems
Publisher: System Design Simple (https://systemdesignsimple.com)
## Short answer
A context window is the fixed token budget one request must fit, shared by instructions, the chat history the app resends, and the reply. If the app keeps only the newest messages that fit, the oldest leave first. In our simulation (approximate tokens, no model run) a name from message 1 left at turn 19, 36 and 141 for 4k, 8k and 32k input budgets. A 12-token pinned note kept it on all 200 turns.
## Key facts
- **Name gone, keep-newest rule (simulated):** Turn 19 at a 4k input budget, turn 36 at 8k, turn 141 at 32k (approximate token units, 200-turn scripted chat)
- **Average turn size:** 229.6 approximate tokens per turn (one user message plus one reply, including 4 units of overhead per message)
- **Rule of thumb (heuristic):** Predicted turn = floor((budget - fixed parts) ÷ average turn size) + 1: gives 18, 35 and 140, each one turn before the measured 19, 36 and 141
- **Bigger window:** 8 times the budget (4k to 32k) moved the drop from turn 19 to 141, about 7.4 times later. It did not prevent it
- **Pinned memory note:** One line of 12 approximate tokens kept the name inside the input on 200 of 200 turns at all three budgets
- **Rolling summary:** 200 units reserved per turn, 0 to 87 actually used (the reserve never filled), name inside the input on 200 of 200 turns at all three budgets
- **What it does not show:** Simulation only: approximate tokens, no model run. It measures whether the name is in the input, not whether a model recalls it
A context window limits how many tokens a model can use in one request, including instructions, selected history and generated output. A chat app has to fit the instructions, the conversation so far and room for the reply inside it. When the conversation outgrows that budget, something has to leave, and under the common keep-newest rule the first thing you said is the first thing to go. The question for this part: when exactly does an early fact fall out, why does a bigger window only postpone it, and what does it cost to keep it?
## Prerequisites {#prerequisites}
**Tokens.** A model reads tokens rather than characters or words. Tokens are pieces of text from a fixed vocabulary: a common short word is usually one token, a long or rare word is split into several, and punctuation counts too. English prose tends to land at a little over one token per word. Every limit in this article is measured in tokens, so a "4k window" means about 4,000 tokens, not 4,000 words or characters. The Hugging Face tokenizer summary in the sources explains how byte-pair and similar tokenizers build those pieces.
**A chat request carries its own history.** This lab models independent requests containing the selected history. Serving systems may retain session state or reuse cached prefixes; cached state does not provide durable memory beyond the context retained. When you send message 20, the app builds a request that contains the instructions, as much of messages 1 to 19 as it chooses to include, and message 20. What feels like memory is the app re-sending the history every turn. This is the same idea as a stateless web server behind a load balancer: the server remembers nothing, so whatever state matters must travel with the request or live in a store the server can read.
**First in, first out.** A FIFO structure removes items in the order they arrived: the oldest leaves first. A queue at a ticket counter works this way. A cache that evicts by FIFO throws out its oldest entry when it is full, regardless of how often that entry is used. A long chat under a keep-newest rule behaves the same way.
## Definitions {#definitions}
> Definition: **Token**: one unit of text from a model's vocabulary, such as a word, part of a word, a number chunk or a punctuation mark. Context limits, prices and speeds are all counted in tokens.
> Definition: **Context window**: the maximum number of tokens a model can process in one request, counting everything it reads and everything it writes. Instructions, history, the new message and the reply all share it.
> Definition: **Input budget**: the part of the context window the app allows for what it sends (instructions, notes, history, new message). It equals the window minus the room reserved for the reply. Our lab uses input budgets only and reserves no reply room.
> Definition: **Truncation**: cutting text so it fits a token limit. **Keep-newest truncation** walks back from the latest message and keeps messages until the next one would not fit; everything older is left out of the request.
> Definition: **Eviction**: removing an item from a bounded store to make room for a new one. In a chat, a message is evicted when it is no longer included in the request, even though the app may still show it on your screen.
> Definition: **Pinned memory note**: a short piece of text, such as "user name: ...", that the app writes once and places at the top of every request, outside the truncation rule.
> Definition: **Rolling summary**: a note that condenses evicted messages and is updated each time more messages leave the window. Each update starts from the previous note plus the newly evicted messages, not from the full history.
> Definition: **Visible versus recalled**: a fact is visible if its text is inside the request. It is recalled if the model actually uses it in the reply. Visibility is necessary for recall but does not guarantee it.
## How it works: what one chat request carries {#how-it-works}
Every turn, the app assembles a request in roughly this shape:
```text
+---------------------------------------------- context window ----+
| instructions (system prompt) fixed |
| pinned notes or summary, if any fixed |
| history: oldest ... newest messages that fit variable |
| your new message variable |
|------------------------------------------------------------------|
| room for the reply reserved |
+------------------------------------------------------------------+
```
The constraint is one inequality:
```text
fixed + history + new message + reply room <= context window
```
The fixed parts barely change from turn to turn. The new message is small. History is the only term that keeps growing, by one user message and one reply every turn. So at some turn the inequality fails, and the app has to choose what to leave out.
The policy modeled here keeps the newest complete messages that fit. In code it is a loop that starts from the latest message, subtracts each message's token count from the remaining room, and stops at the first message that does not fit. Everything older than that point is not sent. The model does not see a gap or a marker. From its point of view, the conversation starts later than it really did.
In Hugging Face Transformers, passing `truncation=True` with a `max_length` cuts a tokenized input to that length, and a tokenizer's `truncation_side` setting decides which end is cut. The default is the right side, which keeps the beginning and cuts the end. For a chat that is the wrong end: you would lose the newest message, the one you are answering. A chat app wants left-side truncation, or better, truncation at message boundaries so it never sends half a message.
Two details make this rule matter for early facts:
- **The first message is always the oldest.** Introductions, goals, constraints and names usually come at the start. Under keep-newest, those are first in line for eviction.
- **The lab evicts individual messages, not turns.** Because the oldest messages leave first, turn 1's user message and its reply leave on the same step or within a step of each other. If the reply repeated the name, that copy goes too. Keeping a user message paired with its reply (or a tool call with its result) is an additional application constraint, not something keep-newest gives you for free.
The reply room is the other half of the window. If the app reserves 1,000 tokens for the reply in a 4,000-token window, the input budget is 3,000, and history has 25% less space. Platform documentation on managing a context window, such as Apple's TN3193 in the sources, treats the session as one budget for this reason. Our lab reserves nothing for the reply, which makes its eviction turns slightly optimistic.
## Inside a production request assembler {#request-assembler}
The lab counts each message as its text plus 4 units. A production context window assembler cannot be that loose, because the model never sees your message list. It sees one token sequence produced by the model's chat template.
**Count the serialized prompt, not the text.** In Hugging Face Transformers, `tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=True)` renders the list through the model's Jinja template and returns the exact token ids. The template adds role headers, separators and end-of-turn tokens, and `add_generation_prompt` appends the header that opens the assistant's reply (the lab's 3 "reply start" units model this). Templates are not always additive: some insert a default system message, drop earlier reasoning blocks or format the last turn differently. The safe rule is to render the final candidate and count that.
**Tool schemas are fixed cost on every request.** Tool definitions passed through `tools=` are rendered into the prompt by the template, so every JSON schema, description and enum list counts against the window on every turn, even when no tool is called. Measure it once by rendering with and without `tools` and treat the difference as part of the fixed parts.
**Reserve output space explicitly.** `input budget = window - max output tokens - safety margin`. The margin covers tokenizer drift when you count with a different tokenizer from the server's.
**Evict atomic groups, not messages.** An assistant message that requests a tool call and the tool result that answers it must stay or leave together. Sending a tool result whose call was evicted, or a call with no result, produces a request many templates and APIs reject or that the model misreads. Group the history first, then evict groups oldest first.
**Check the mandatory part before anything else.** If the system prompt, pinned notes, tool schemas and the current user message do not fit on their own, the keep-newest loop keeps zero history and can even drop the current message. Handle that case as an explicit error so it never passes silently.
```js
// Sketch of a production assembler. count() renders through the model's chat template.
function assemble({ system, pinned, tools, history, current, window, maxOutput, margin, count }) {
const budget = window - maxOutput - margin;
const mandatory = [system, ...pinned, current];
if (count(mandatory, tools) > budget) {
// Reject, or deliberately shrink: truncate a pasted file, summarise a tool result, drop tools.
throw new Error('context overflow: mandatory content does not fit');
}
const groups = groupAtomic(history); // [user] | [assistant + tool calls, tool results...]
const kept = [];
for (let i = groups.length - 1; i >= 0; i--) {
const trial = [system, ...pinned, ...groups[i], ...kept.flat(), current];
if (count(trial, tools) > budget) break;
kept.unshift(groups[i]);
}
const msgs = [system, ...pinned, ...kept.flat(), current];
if (msgs.at(-1) !== current) throw new Error('current user message lost in assembly');
return { msgs, droppedGroups: groups.length - kept.length, inputTokens: count(msgs, tools) };
}
```
Rendering the whole candidate on every step is quadratic in history length. In practice you cache a token count per group, sum them, and render once at the end to verify against the exact total, falling back to dropping one more group if the check fails. Log `droppedGroups`, `inputTokens` and the oldest turn kept on every request; those three fields are how you detect the failure modes later in this article.
### Cached attention state is not memory
During generation a transformer stores a key and a value vector per layer, per attention head, for every token so far, so each new token attends to earlier ones without recomputing them. The Hugging Face cache documentation describes these tensors. Their size is:
```text
KV bytes = 2 (K and V) x layers x KV heads x head dim x bytes per element x tokens x batch
```
With illustrative assumptions (batch 1, 32 layers, 8 KV heads with grouped-query attention, head dimension 128, 2 bytes per element for fp16 or bf16, 32,768 tokens), that is `2 x 32 x 8 x 128 x 2 = 131,072 bytes`, or 128 KiB per token, and `128 KiB x 32,768 = 4 GiB` for one full sequence. At a 4,000 token request the same model needs 4,000 x 128 KiB = 500 MiB. These are assumptions for a mid-sized model, not the figures of any particular product. A bigger window therefore costs memory on the serving side as well as tokens.
Serving engines can keep that state across requests. vLLM's automatic prefix caching stores KV blocks keyed by the token prefix that produced them, so a new request that starts with the same tokens reuses those blocks and computes only the new suffix. Two consequences matter for a chat app:
- **The cache never holds more than the request did.** It is keyed by exact token prefixes and evicted under memory pressure. Once the name leaves the request, no cached block containing it can match. Durable memory has to live in the application (a database row, a pinned note) and be sent again as text.
- **Eviction breaks prefix reuse.** Once keep-newest starts dropping messages, the history after the system prompt starts at a different message every turn, so only the system prompt and anything placed before history (tools, a stable pinned note) still match. A rolling summary that is rewritten on every eviction invalidates everything after it. If cache hit rate matters, put stable content first and update summaries in batches rather than every turn.
## At which turn does the first message fall out? {#when-does-it-drop}
If every turn adds about the same number of tokens, the turn at which the first message leaves is roughly predictable. History before reply t holds turns 1 to t - 1 plus the new user message. A heuristic for when that no longer fits:
```text
predicted turn = floor((budget - fixed parts) ÷ average turn size) + 1
```
Treat it as an estimate. The real eviction step also depends on the size of the current user message and on the cumulative sizes of the specific messages before it.
To check this, we built a small lab: a deterministic simulation of a 200-turn chat on one Mac. Message 1 contains a name. Every later message is filler about building a study planner app, generated from a fixed seed, and no filler message contains the name. Before each reply, the lab builds the input a model would receive under the keep-newest rule and checks whether the name's text is anywhere in it. It runs at three input budgets: 4,000, 8,000 and 32,000 approximate tokens.
Three limits apply to every number below:
- **It is a simulation.** No model is called. It measures the truncation mechanism, not any chat product, and real assistants may use larger windows, smarter summaries, retrieval or their own memory features.
- **Tokens are approximate.** The lab has no packages and no network, so it cannot ship a real vocabulary. It splits text the way byte-pair tokenizers pre-split it and charges long words about one unit per four letters, roughly 1.2 to 1.3 units per English word. All budgets and counts are approximate token units.
- **Visible is not recalled.** The lab checks whether the name is in the input. Whether a model would use it is a separate test the lab does not run.
The scripted chat averaged 229.6 approximate tokens per turn (one user message and one reply, including 4 units of overhead per message for role markers). The fixed parts are the instructions, 17 units including overhead, plus 3 units that start the reply: 20 in total.
| Input budget | Predicted drop turn | Measured: name gone at turn | Turns with name in input |
| --- | --- | --- | --- |
| 4,000 | floor(3,980 ÷ 229.6) + 1 = floor(17.33) + 1 = 18 | 19 | 18 of 200 |
| 8,000 | floor(7,980 ÷ 229.6) + 1 = floor(34.76) + 1 = 35 | 36 | 35 of 200 |
| 32,000 | floor(31,980 ÷ 229.6) + 1 = floor(139.29) + 1 = 140 | 141 | 140 of 200 |
The prediction lands one turn early at every budget. The reason is turn 1 itself: the introduction and its reply together cost 84 units, well under the 229.6 average, so the real history is a little shorter than the formula assumes and fits one turn longer. The floor matters too: 34.76 would round to 36, but the code floors it to 35. A reply reserve shifts eviction earlier for both the prediction and the measurement; it does not necessarily reverse the predictor's error, which depends on the sizes of the messages involved.
The per-turn log shows the name leaving in a single step. At the 4k budget, turn 18 sends 35 messages with turn 1 still the oldest, and nothing has been dropped. At turn 19, two messages are dropped, the oldest turn in the input is turn 2, and the name is gone. At 8k the step at turn 36 drops four messages at once, turns 1 and 2, because the newest messages needed more room than turn 1 alone freed. After that, eviction continues every turn, so by turn 200 at the 4k budget the oldest message in the input is from turn 183.
So under keep-newest an early fact has a deadline you can compute from two numbers you already have: the budget and the typical turn size.
## Does a bigger context window fix it? {#bigger-window}
Going from 4k to 32k multiplied the budget by 8 and moved the drop from turn 19 to turn 141, about 7.4 times later. That is close to proportional, which is what the formula predicts: the drop turn is the budget divided by the turn size. Doubling the window roughly doubles the number of turns before the first message leaves. It does not change the rule, so a long enough chat always reaches the deadline. In our run the 32k budget still had the name missing from the input on the last 60 of the 200 turns.
A bigger window also changes three other things.
**Every request carries a larger logical input.** Because the app resends history, input size per request grows until it hits the budget and then stays there. Summed over all 200 turns of the lab's log, the 4k run sent about 0.75 million approximate tokens of logical input, and the 32k run about 4.17 million, roughly 5.6 times as much. That is the volume of context the requests contain, not necessarily the work done: with prefix caching a server recomputes only the uncached suffix, so compute, time to first token and billing depend on cache hits and on the service's pricing for cached and uncached input. The relationship between work per request, waiting and throughput is the subject of latency vs throughput; a longer input is more work per request.
**Long inputs are not read evenly.** The Lost in the Middle paper (Liu et al., 2023) tested models on finding a relevant passage placed at different positions in a long input and found accuracy was usually highest when the passage was near the start or the end, and lower when it sat in the middle. A name that is technically inside a 32k window, 20,000 tokens back, is visible but not guaranteed to be used. This is the gap between visible and recalled, and it is why our lab does not claim recall.
**Serving memory grows with it.** Under the illustrative KV cache assumptions above, one full 32,768 token sequence holds 4 GiB of cached state against 500 MiB at 4,000 tokens, which limits how many long sessions one GPU can serve at once.
A bigger window buys time and pays for it with larger requests. To keep an important early fact, the app has to change what it sends.
## Why is this a FIFO cache? {#fifo-cache}
Look at the window as a store with a fixed capacity, and at each message as an entry. New entries arrive every turn. When the store is full, the oldest entry is removed to make room. Nothing about how useful an entry is affects the choice. That is a cache with first in, first out eviction, the simplest replacement policy in the cache replacement literature.
FIFO needs only an ordered list and a pointer to the oldest item. It evicts by age, so an entry that is read constantly leaves as quickly as one never read again. Other policies use more information:
- **LRU (least recently used)** evicts the entry that has gone longest without being read, so frequently used items stay.
- **LFU (least frequently used)** evicts the entry with the fewest reads.
- **TTL-based** policies evict entries whose time-to-live is closest to expiry.
Redis, for example, offers LRU, LFU, random and TTL-based eviction policies, plus `noeviction`, which refuses writes when memory is full. Plain FIFO is not on that list. A capped "newest N items" list is a different case: it is deliberately FIFO, as in the home feed design where each follower's list is trimmed to the newest entries in designing a photo feed. That works for a feed because readers want the newest posts. It fails for a chat because the oldest message often holds the facts the rest of the conversation depends on.
Why not just use LRU? Internal reliance on earlier messages is not directly exposed as cache-access telemetry: the model returns text, not a list of the earlier messages it relied on. Applications can instead track explicit signals, such as messages the user quotes or refers back to, retrieval hits, or relevance scores between old messages and the new question, and rank by those. Age remains the cheapest ordering, which is why keep-newest is the default. The fixes in the next section work around this by deciding up front which facts deserve to stay, the way a cache can pin hot keys so eviction never touches them.
Caching gets its own season in the main series, starting at Part 33 with cache-aside, which covers hit rates, eviction policies and invalidation in depth. For how a cache close to the user cuts latency in practice, see how a CDN keeps video close to viewers.
## How do apps keep a fact in a long chat? {#keeping-facts}
Both fixes below take a fact out of the FIFO and give it a reserved place in every request. They differ in how that place is filled and what it costs.
### Fix 1: a pinned memory note
When the user states a fact worth keeping, the app writes a short line and places it above the history in every request. In the lab the line is `Memory: user name: ...` and costs 12 approximate tokens including overhead. The history then gets the budget minus those 12 units.
Result: the name was inside the input on 200 of 200 turns at the 4k, 8k and 32k budgets. The cost is small and fixed. At the 4k budget, 12 units are 0.3% of the window. The note contributes 2,400 approximate units across 200 requests (12 × 200); the net input change also depends on which whole messages it displaces. In the saved trace the totals moved by -424 units at 4k (747,934 to 747,510), +1,384 at 8k (1,441,049 to 1,442,433) and +1,485 at 32k (4,167,129 to 4,168,614). The net can even be negative, as at 4k, because when the 12 units no longer fit the loop drops a whole message that is bigger than the note. Because the history loses 12 units of room, eviction can happen a little sooner, but in this run the keep-newest history under the note kept the same oldest turn as without it at turn 200 for all three budgets.
Storing the note is simple. The harder work is deciding what goes in it and keeping it correct:
- **Extraction.** Something must notice that "my name is ..." is a fact to keep. The lab uses a regular expression. A real app might use explicit user action ("remember this"), a classifier, or the model itself, and each can miss facts or store wrong ones.
- **Updates.** If the user later says "call me by my nickname", the note must change, or the app will keep sending a stale fact with full confidence.
- **Growth.** Every new fact makes the note longer, and a note that grows without limit becomes a second history with the same problem.
- **Privacy.** A pinned note is user data that outlives the visible conversation. Users should be able to see, edit and delete it.
### Fix 2: a rolling summary
Instead of a few hand-picked facts, the app condenses everything that leaves the window into a note that stays in it. In the lab, 200 approximate tokens are reserved for the note on every turn. Each time messages are evicted, the note is updated from its own previous content plus the newly evicted messages, never from the full history. Facts such as the name go first; then the note lists earlier topics while there is room, and the algorithm would skip any new topic that pushed it over its limit.
Result: the name was inside the input on 200 of 200 turns at all three budgets. The note used between 0 units (before anything was evicted, there is nothing to summarise) and 87 units, well under the 200 reserved.
The costs are different from the pinned note:
- **Reserved room is room lost.** The lab reserves the full 200 units every turn even when the note uses less, so history always has 200 fewer units. At the 4k budget that is 5% of the window, and the summary run kept 31.4 messages per request on average against 33.0 for plain keep-newest. At 32k it is under 1% and barely matters. A real app could reserve dynamically, at the price of more complex budgeting.
- **The summary is lossy.** The summary discards message wording and details while retaining distinct topic labels. This run did not fill its reserve: the generator draws from a fixed list of 18 filler topics plus the introduction, so the note peaked at 87 units. The algorithm would skip additional topics if they exceeded the limit, and in a chat with an open-ended topic list that saturation would silently hide later topics.
- **A model-written summary can drop facts.** The lab's summary is rule-based and is written to always keep the name. That shows what a good summary buys you and says nothing about summaries in general. When a model writes the summary, each rewrite is a chance to drop or change a fact, and because each new summary starts from the previous one, a fact lost once is lost for good. This is the main risk of the pattern.
- **It costs extra work.** A model-written summary is another request to a model each time it is updated, with its own latency and token cost.
In practice the two fixes combine: a small pinned note for facts that must never be lost (name, language, key constraints), and a summary for the general thread of the conversation. A third pattern stores old messages outside the window and looks up the relevant ones when a new question arrives. That is retrieval, and it is where this series goes next.
## What we saw when we ran it {#we-ran-it}
We ran `node labs/ai-3/run.mjs` on our own Mac. It plays a scripted chat of 200 turns with the name in the first message, then fits the history into 4k, 8k and 32k budgets with each strategy. The whole run took 103 ms.
*Our lab run in the terminal, with the turns around the drop at the 8k budget.*
With plain truncation the name left the window at turn 19 for 4k, turn 36 for 8k and turn 141 for 32k. In the 8k table you can watch it happen: at turn 36 the oldest message kept jumps from 1 to 3 and the name column flips to NO. What surprised us was how small the fix was. A pinned memory line of 12 approximate tokens kept the name visible on all 200 turns at every budget, and the rolling summary did the same while never using more than 87 of its 200 reserved approximate tokens. The lab counts what sits in the window, not what a model recalls. For your own app, a bigger window only moves the drop later, so pin or summarize any fact the whole chat depends on.
*Apple's TN3193 technote on managing a context window.*
Apple's official page treats the session as one context window budget, which is the general definition this article uses.
## Worked example {#worked-example}
The video's scenario: you open a chat, say your name in the first message, then keep asking questions about an app you are building. The lab plays that chat for 200 turns. All inputs are the lab's configuration; all results come from its output.
```text
CONFIG 200 turns, seed 20261105, name only in message 1
approximate token units, 4 units overhead per message
fixed parts: instructions 17 + reply start 3 = 20 units
no reply reserve (input budget only)
TURN SIZE average 229.6 units (user message + reply)
turn 1 (introduction + reply) = 35 + 49 = 84 units
PREDICT predicted turn = floor((budget - 20) / 229.6) + 1 (heuristic)
4k: floor(3,980 / 229.6) + 1 = floor(17.33) + 1 = 18
8k: floor(7,980 / 229.6) + 1 = floor(34.76) + 1 = 35
32k: floor(31,980 / 229.6) + 1 = floor(139.29) + 1 = 140
MEASURED keep newest: name gone at turn 19 / 36 / 141
name in input 18 / 35 / 140 of 200 turns
BIGGER 32k / 4k = 8x budget; 141 / 19 = about 7.4x later
the deadline moved, it did not disappear
FIX 1 pinned note 12 units = 0.3% of 4k
name in input 200 / 200 at every budget
total note tokens over the chat: 12 x 200 = 2,400 units
net input change vs plain: -424 / +1,384 / +1,485 (4k / 8k / 32k)
FIX 2 summary: 200 reserved = 5% of 4k, 0 to 87 used
name in input 200 / 200 at every budget
reserve never filled in this run (bounded topic list)
```
With the turn size and the window you know when early facts leave; anything that must outlive that turn needs a reserved slot.
## Code {#code}
The simulation behind every number in this article, trimmed for reading. It uses the same seed, tokenizer, chat generator and window rules as our lab script and prints the same results. The test name in message 1 is a placeholder with the same token cost as the lab's. Save it as `context_lab.mjs` and run `node context_lab.mjs` (Node.js 18 or newer, no packages, no network, runs in well under a second).
```js
// context_lab.mjs: a 200-turn chat squeezed into a fixed context window.
// A = keep the newest messages that fit. B = A plus a pinned memory line.
// C = A plus a rolling summary note with 200 units reserved every turn.
// Node.js 18+, no packages, no network. Same seed, same numbers, any machine.
const SEED = 20261105, TURNS = 200, BUDGETS = [4000, 8000, 32000];
const OVERHEAD = 4, PRIMING = 3, SUMMARY_RESERVE = 200;
const NAME = 'Your Name'; // placeholder; filler text never contains it
const SYSTEM = 'You are a helpful assistant. Answer clearly and kindly.';
function mulberry32(a) { // small seeded random number generator
return () => {
a |= 0; a = (a + 0x6d2b79f5) | 0;
let t = Math.imul(a ^ (a >>> 15), 1 | a);
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
const rand = mulberry32(SEED);
const pick = (arr) => arr[Math.floor(rand() * arr.length)];
const between = (lo, hi) => lo + Math.floor(rand() * (hi - lo + 1));
// Approximate tokens: byte-pair style pre-split; words over 7 letters cost about 1 per 4 letters.
const PIECE = / ?[A-Za-z]+| ?[0-9]{1,3}| ?[^\sA-Za-z0-9]+|\s+/g;
function countTokens(text) {
let n = 0;
for (const m of text.matchAll(PIECE)) {
const letters = m[0].replace(/[^A-Za-z]/g, '').length;
n += letters === 0 ? 1 : letters <= 7 ? 1 : Math.ceil(letters / 4);
}
return n;
}
const WORDS = ('the a an and or but so then if when because with for from into about over ' +
'after before while this that these those it its we you your our they them ' +
'is are was were be can could should would will may might need want try use ' +
'make build add keep save show send read write check test change move start ' +
'page screen button list table form field user users class student teacher ' +
'task tasks note notes date time week day reminder plan planner schedule ' +
'login password account email phone server database request response cache ' +
'query index error message simple small quick easy clear better faster first ' +
'next last each every some many more less one two three new old same other ' +
'data record update delete search filter sort order group share link file ' +
'image upload storage backup version release design layout color font mobile ' +
'desktop browser network offline online sync conflict friend group project').split(/\s+/);
const TOPICS = ['login screen', 'task list', 'reminder emails', 'database tables', 'search box',
'mobile layout', 'offline mode', 'file uploads', 'sharing with friends', 'weekly schedule',
'password reset', 'dark colors', 'backup plan', 'error messages', 'sorting tasks',
'group projects', 'release steps', 'caching pages'];
function sentence() {
const n = between(8, 16), w = [];
for (let i = 0; i < n; i++) w.push(pick(WORDS));
if (n > 10 && rand() < 0.5) w[between(3, n - 3)] += ',';
w[0] = w[0][0].toUpperCase() + w[0].slice(1);
return w.join(' ') + (rand() < 0.15 ? '?' : '.');
}
const sentences = (k) => Array.from({ length: k }, sentence).join(' ');
const chat = [
{ turn: 1, role: 'user', topic: 'introductions', text: `Hi! My name is ${NAME}. I am building a small study planner app for my class, and I would like your help step by step.` },
{ turn: 1, role: 'assistant', topic: 'introductions', text: 'Nice to meet you. A study planner is a great first project. Let us start with the screens you need, then the data you store, and then how people log in. Tell me what you want to build first.' },
];
for (let t = 2; t <= TURNS; t++) {
const topic = pick(TOPICS);
chat.push({ turn: t, role: 'user', topic, text: `Next question about the ${topic}. ${sentences(between(2, 4))}` });
chat.push({ turn: t, role: 'assistant', topic, text: `Good question about the ${topic}. ${sentences(between(9, 15))}` });
}
for (const m of chat) m.tokens = countTokens(m.text) + OVERHEAD;
const FACT = /\b[Mm]y name is ([A-Z][a-z]+(?: [A-Z][a-z]*)?)/;
const facts = (msgs) => [...new Set(msgs.filter((m) => m.role === 'user')
.map((m) => m.text.match(FACT)).filter(Boolean).map((h) => `user name: ${h[1]}.`))];
// Rolling summary: sees only its previous state plus messages evicted since last turn.
const render = (s) => `Summary of ${s.count} earlier messages. ${s.facts.join(' ')} Topics so far:` +
s.topics.map((x) => ` ${x},`).join('');
function updateSummary(prev, evicted) {
const limit = SUMMARY_RESERVE - OVERHEAD;
const s = { count: prev.count + evicted.length, facts: [...prev.facts], topics: [...prev.topics] };
for (const f of facts(evicted)) if (!s.facts.includes(f)) s.facts.push(f); // facts first
for (const m of evicted) { // topics while room
if (s.topics.includes(m.topic)) continue;
const next = { ...s, topics: [...s.topics, m.topic] };
if (countTokens(render(next)) <= limit) s.topics = next.topics;
}
return s;
}
const SYS = countTokens(SYSTEM) + OVERHEAD;
const pinText = `Memory: ${facts(chat.slice(0, 1)).join(' ')}`;
const PIN = countTokens(pinText) + OVERHEAD;
function buildWindow(t, budget, strategy, mem) { // what the model sees before reply t
const cands = chat.filter((m) => m.turn < t || (m.turn === t && m.role === 'user'));
let room = budget - SYS - PRIMING - (strategy === 'B' ? PIN : 0) - (strategy === 'C' ? SUMMARY_RESERVE : 0);
const kept = [];
for (let i = cands.length - 1; i >= 0 && cands[i].tokens <= room; i--) { // newest first
kept.unshift(cands[i]); room -= cands[i].tokens;
}
const dropped = cands.length - kept.length;
let header = strategy === 'B' ? pinText : '', headerTok = strategy === 'B' ? PIN : 0;
if (strategy === 'C') {
if (dropped > mem.evicted) mem.state = updateSummary(mem.state, cands.slice(mem.evicted, dropped));
mem.evicted = dropped;
if (mem.state.count) { header = render(mem.state); headerTok = countTokens(header) + OVERHEAD; }
}
const text = [SYSTEM, header, ...kept.map((m) => m.text)].join('\n');
return { visible: text.includes(NAME), headerTok };
}
const avg = chat.filter((m) => m.turn >= 2).reduce((s, m) => s + m.tokens, 0) / (TURNS - 1);
console.log(`average ${avg.toFixed(1)} approx tokens per turn, pinned line ${PIN} approx tokens`);
console.log('budget name gone (A) predicted visible A visible B visible C summary used');
for (const budget of BUDGETS) {
const mem = { evicted: 0, state: { count: 0, facts: [], topics: [] } };
const seen = { A: 0, B: 0, C: 0 }; let gone = null, lo = Infinity, hi = 0;
for (let t = 1; t <= TURNS; t++) {
for (const s of ['A', 'B', 'C']) {
const w = buildWindow(t, budget, s, mem);
if (w.visible) seen[s]++;
if (s === 'A' && !w.visible && gone === null) gone = t;
if (s === 'C') { lo = Math.min(lo, w.headerTok); hi = Math.max(hi, w.headerTok); }
}
}
const predicted = Math.floor((budget - SYS - PRIMING) / avg) + 1;
console.log(`${String(budget).padStart(6)} ${String(gone).padStart(14)} ${String(predicted).padStart(10)} ` +
`${String(seen.A).padStart(10)} ${String(seen.B).padStart(10)} ${String(seen.C).padStart(10)} ${lo} to ${hi}`);
}
```
Real output:
```text
average 229.6 approx tokens per turn, pinned line 12 approx tokens
budget name gone (A) predicted visible A visible B visible C summary used
4000 19 18 18 200 200 0 to 87
8000 36 35 35 200 200 0 to 87
32000 141 140 140 200 200 0 to 87
```
The full lab script also writes a per-turn log (tokens, oldest turn in the input, messages dropped, name visible) showing the exact step at which each message leaves.
What to observe when you change it:
- Set `BUDGETS` to `[16000]`. The predicted turn is floor(15,980 ÷ 229.6) + 1 = floor(69.60) + 1 = 70, and the measured drop lands near it. The heuristic scales because nothing else changes.
- Reserve room for the reply by subtracting 1,000 from `room` in `buildWindow`. The name now leaves at turns 15, 32 and 136 instead of 19, 36 and 141. The `predicted` column does not know about the reserve, so it stays at 18, 35 and 140.
- Delete the line marked `facts first` inside `updateSummary` so the summary keeps only topics. Strategy C's name count falls from 200 to 18, 34 and 139 of 200, the same as A or one turn worse, because its reserved room evicts sooner. That is the failure a model-written summary can produce when it decides a name is not worth keeping.
- Make the replies longer by changing `between(9, 15)` to `between(18, 30)`. Turns get bigger, and the name leaves at turns 12, 22 and 81. Turn size matters as much as window size.
## Failure modes and edge cases {#failure-modes}
**Silent loss of instructions.** The same rule that drops a name drops anything said early: "answer in Nepali", "use metric units", "the deadline is Friday". The model shows no error and stops following them. Detect it by logging, for every request, the oldest turn included and the number of messages dropped. Fix it by moving durable instructions into the pinned area or the system prompt, which are excluded from this application's history-eviction policy; mandatory content must still fit.
**Empty or headless request.** In the lab's `buildWindow`, if the newest message alone is larger than the room left after the fixed parts, the loop stops immediately and keeps no history at all, including the current user message. A pasted log or file can trigger this. The request then goes to the model without the question it is supposed to answer. Detect it by asserting, after assembly, that the mandatory parts fit and that the last message is the current user message. On failure, reject the request with a clear error or deliberately reduce the oversized content (truncate the paste, summarise the attachment) before inference, never after.
**Orphaned tool calls.** Evicting messages one at a time can keep a tool result whose tool call was dropped, or the reverse. Evict a tool call and its results as one group.
**Half a message.** Truncating at a token count instead of at message boundaries can cut a message in the middle, sometimes inside a code block or a table. The model then reads a broken fragment as part of the conversation. Always evict whole messages, and keep a user message and its reply together where possible.
**Tool output floods the window.** In apps that call tools, one large result (a file, a web page, a log) can be thousands of tokens and push out dozens of turns in one step. Detect it with per-message token counts in your logs. Fix it by truncating or summarising tool output before it enters the history, and by capping how many tokens any single message may take.
**Summary drift.** A model-written rolling summary is rewritten many times over a long session. Each rewrite can shorten, merge or alter facts, and errors compound because each summary is built from the previous one. Detect it by checking pinned facts against the summary, or by sampling long sessions and comparing the summary with the full history. Fix it by keeping critical facts in a pinned note that the summarizer cannot touch.
**Stale pinned facts.** A pinned note is sent with full authority on every turn. If the user corrects a fact and the note is not updated, the model keeps receiving the wrong one. Give the note an explicit update path and show it to the user.
**Budget miscount.** If the app estimates tokens with a different tokenizer from the model's, it can build a request it believes fits but the model rejects or cuts. Count with the model's own tokenizer where possible, and keep a safety margin when you cannot.
## Trade-offs {#trade-offs}
| Option | Good for | Costs you | Use when |
| --- | --- | --- | --- |
| Keep-newest truncation alone | Simplest, no extra state, recent context is always complete | Early facts leave at a predictable turn (19, 36, 141 in our lab) | Short sessions, or when only recent messages matter |
| Bigger context window | More turns before anything leaves | Larger logical input per request (compute and billing depend on cache hits and pricing), more KV cache memory, uneven use of long inputs | Sessions are long but bounded, and cost per request is acceptable |
| Pinned memory note | A few facts that must never leave; tiny fixed cost (12 units in our lab) | Needs extraction, updates and a privacy story; grows with each fact | Names, language, key constraints, user preferences |
| Rolling summary | Keeps the thread of a long conversation in bounded space | Reserved room (200 units in our lab), lossy, a model-written summary can drop facts | Long sessions where the general context matters more than exact wording |
| Retrieval from stored history | Exact old text brought back only when relevant | Storage, an index, a lookup per turn, and the chance of fetching the wrong passage | Very long histories or documents; covered in the next episode |
| Start a new session | Clean window, no stale context | Loses everything not carried over by a note or summary | The topic changes completely |
## Common mistakes {#common-mistakes}
- **Mistake:** Assuming the model remembers the conversation. **Instead:** Treat every request as carrying its own context. The model can use only what the app sends this turn; a server's prefix cache may speed that up but holds nothing beyond the context retained.
- **Mistake:** Treating a bigger window as the fix for forgetting. **Instead:** Estimate the drop turn as floor((budget - fixed) ÷ turn size) + 1. A bigger window moves the deadline and enlarges every request; it does not remove the deadline.
- **Mistake:** Assuming the keep-newest loop always keeps the current message. **Instead:** Assert that mandatory content fits and the current user message survives assembly, and evict tool calls together with their results.
- **Mistake:** Forgetting that the reply shares the window. **Instead:** Budget the input as window minus reply reserve, so a long answer does not get cut off or force extra truncation.
- **Mistake:** Trusting a model-written summary to keep every fact. **Instead:** Put facts that must survive into a pinned note outside the summary, and test long sessions for facts that disappear.
- **Mistake:** Reading "the fact is in the window" as "the model will use it". **Instead:** Visibility is necessary, not sufficient. Place critical facts near the start or end of the input, and test recall separately.
- **Mistake:** Testing only short conversations. **Instead:** Add a test that runs a chat past the computed drop turn and checks that pinned facts are still in the request.
## In an interview {#in-an-interview}
1. **Clarify:** how long sessions run, which facts must survive, the model's window size, and the cost limit per request.
2. **Assumptions:** for example, a 4k input budget, about 230 tokens per turn, a name and a language preference that must never be lost.
3. **Structure:** request = instructions + pinned note + optional summary + newest history that fits + new message, with room reserved for the reply.
4. **Deep dive:** the drop turn formula; keep-newest as FIFO eviction; why LRU needs explicit signals (references, retrieval hits, relevance scores) because the model's reliance on old messages is not exposed; the KV cache size and why prefix caching is not memory.
5. **Trade-offs:** bigger window against cost per request; pinned note against extraction and update complexity; summary against lossiness and reserved room.
6. **Wrap up:** log the oldest turn sent and messages dropped per request, alert when pinned facts are missing, and plan retrieval for very long histories.
> **A 60-second answer**A model reads a fixed number of tokens per request, and the instructions, the history the app resends and the reply all share it. When history outgrows the budget, a common policy keeps the newest complete messages, so the first message leaves first, at roughly budget ÷ tokens per turn. In a simulation with about 230 tokens per turn, a name from message 1 left at turn 19 with a 4k input budget, 36 at 8k and 141 at 32k, so a bigger window only delays it and makes every request larger. That rule is FIFO cache eviction. To keep facts, I give them a reserved slot: a pinned note of about a dozen tokens for things that must never leave, and a rolling summary for the general thread, knowing a model-written summary can drop facts. In the simulation both kept the name in the input on all 200 turns, which shows presence, not recall, so I would also test recall on long sessions.
## Practice {#practice}
**Q (Beginner): What shares a model's context window during one chat request?**
The instructions, any pinned notes or summary, the conversation history the app includes, the new message, and the reply the model writes. They all count against the same token limit.
**Q (Intermediate): A chat averages 300 tokens per turn and has 6,000 tokens of input budget after fixed parts. Around which turn does message 1 leave under keep-newest?**
6,000 ÷ 300 = 20 turns fit, so the first message leaves at about turn 21. Expect it within a turn or so either way, depending on how big turn 1 is compared with the average.
**Q (Intermediate): Going from a 4k to a 32k input budget moved the drop from turn 19 to 141. Why is this roughly proportional, and what does it cost?**
The drop turn is about budget ÷ tokens per turn, so 8 times the budget gives about 8 times the turns (7.4 times here, because the fixed parts and turn 1's size shift it slightly). Every request near the limit then carries about 8 times as much logical input; how much of it is recomputed and billed depends on prefix cache hits and pricing.
**Q (Intermediate): Why is keep-newest truncation a FIFO eviction policy, and why not use LRU instead?**
Messages are evicted strictly by age when the window is full, regardless of use, which is FIFO. LRU needs to know which entries were used recently. The model's internal reliance on earlier messages is not exposed as cache-access telemetry, so an app would have to track explicit signals instead: messages the user refers back to, retrieval hits or relevance scores.
**Q (Advanced): The lab's summary reserves 200 units but used at most 87. What does the unused reserve cost, and how could you reduce it?**
History always has 200 fewer units, 5% of a 4k window, so fewer messages fit: 31.4 per request on average against 33.0 without the summary. Reserving only the note's actual size each turn recovers most of it, at the price of more complex budgeting.
**Q (Advanced): Both fixes kept the name in the input on 200 of 200 turns. What would you still need to test before claiming the model remembers it?**
Recall: run a real model on long sessions and check whether replies actually use the name, with the note in different positions. Studies of long inputs show facts in the middle are used less reliably than facts near the start or end, so presence alone does not prove use.
**Q (Advanced): A model-written rolling summary works in tests but users report the assistant sometimes forgets their dietary restriction after an hour. What is happening and how do you fix it?**
Each rewrite of the summary can drop or soften a fact, and because each summary is built from the last one, a dropped fact never comes back. Move must-keep facts into a pinned note the summarizer cannot rewrite, and add a check that compares pinned facts with each new summary.
## Cheat sheet {#cheat-sheet}
| Term or formula | Meaning |
| --- | --- |
| Token | Unit of text from the model's vocabulary; all limits and prices count these |
| Context window | Max tokens per request, shared by everything sent and the reply |
| Input budget = window - reply reserve | Room for instructions, notes, history and the new message |
| Request carries its context | The app resends selected history; prefix caches may reuse computed state but are not durable memory |
| KV bytes = 2 x layers x KV heads x head dim x bytes x tokens | Cached attention state; 128 KiB per token under the article's illustrative assumptions |
| Keep-newest truncation | Keep the latest messages that fit, drop everything older |
| Predicted turn = floor((budget - fixed) ÷ avg turn) + 1 | Heuristic for when the first message leaves under keep-newest |
| FIFO eviction | Remove the oldest entry first, regardless of use |
| LRU / LFU | Evict least recently / least frequently used; needs a usage signal |
| Pinned memory note | Short fixed text at the top of every request, excluded from history eviction; must still fit |
| Rolling summary | Lossy note of evicted messages, updated from its previous version |
| Visible vs recalled | In the input vs actually used by the model; the first does not guarantee the second |
| Lost in the middle | Facts deep in a long input tend to be used less reliably than facts near its ends |
## Further reading {#further-reading}
- **Apple TN3193, Managing the on-device foundation model's context window:** platform documentation on what counts against a context window and what to do as a session fills it; used here for the general definition.
- **Hugging Face Transformers, Padding and truncation:** how `truncation`, `max_length` and the tokenizer's truncation side cut inputs to a limit.
- **Hugging Face Transformers, Summary of the tokenizers:** how byte-pair and related tokenizers split text, which explains why token counts differ from word counts.
- **Hugging Face Transformers, Chat templates:** how `apply_chat_template` serializes messages and tool schemas into the exact token sequence the model reads.
- **Hugging Face Transformers, Caching:** what the KV cache stores per layer and per token, the basis of the memory formula above.
- **vLLM, Automatic prefix caching:** how a serving engine reuses KV blocks across requests that share a token prefix.
- **Liu et al. (2023), Lost in the Middle:** measurements of how models use information at different positions in long inputs; the evidence behind "visible is not recalled".
- **Cache replacement policies (FIFO):** FIFO, LRU, LFU and other eviction policies compared, the cache view of keep-newest truncation.
- **Redis documentation, Key eviction:** the eviction policies a widely used cache actually offers, and how `noeviction` behaves when memory is full.
This builds on the first part of this series, where one AI writes and another reviews showed an AI app as a pipeline of plain system design parts. Next in the series, AI 4: a hundred pages, one question. How does AI find the one right line? That episode covers RAG retrieval: storing text outside the window and bringing back only the passages a question needs.
This is a simplified, general explanation of common patterns, not a description of any company's private systems or of any specific chat product. All lab numbers come from our deterministic simulation (labs/ai-3, Node.js v22.13.1, seed 20261105, 200 turns) using approximate token units from a heuristic splitter and input budgets with no reply reserve. It measures whether a fact is inside the input, not whether a model recalls it, and no model was run.
## Frequently asked questions
**What is a context window?**
It is the largest amount of text, counted in tokens, that a language model can handle in one request. The instructions, the conversation history the app sends, your new message and the reply the model writes all share that one budget. Anything that does not fit is simply not part of the request.
**Why does an AI chat forget things I said earlier?**
Each request can only use the history included in it. A serving system may retain session state or reuse cached prefixes, but that cache is not durable memory beyond the context retained. When the history no longer fits the window, the policy modeled here keeps the newest complete messages and drops the oldest. Your first message is the oldest, so it leaves first. In our simulation that happened at turn 19 with a 4k input budget.
**Does a bigger context window stop the forgetting?**
Under a keep-newest rule it only delays it. In our simulation 4k, 8k and 32k budgets dropped the name at turns 19, 36 and 141. The turn grows roughly in proportion to the budget, so a long enough chat always reaches it, and every request in a bigger window carries a larger logical input. How much of that is recomputed, how long it takes and what it is billed depends on prefix cache hits and the service's pricing.
**How do chat apps remember a name across a long conversation?**
Two common patterns are a pinned memory note, a short line such as the user's name that the app places at the top of every request, and a rolling summary that condenses evicted messages into a note kept in the window. Retrieval, which looks up relevant old text on demand, is a third pattern and the subject of the next episode.
**If my name is inside the context window, will the model always use it?**
Not necessarily. Being in the window makes recall possible but does not guarantee it. Research on long inputs, such as the Lost in the Middle paper, found that models use information near the start or end of the input more reliably than information buried in the middle. Our lab checks only whether the name is present.
**Were these numbers measured on a real AI model?**
No. They come from a deterministic simulation in Node.js (seed 20261105, 200 turns) that builds the input a model would receive and checks whether the name is in it. Token counts come from a heuristic splitter, not a real model vocabulary, and no room is reserved for the reply, so a real system would drop the name slightly earlier.
## Sources
- [Apple TN3193: Managing the on-device foundation model's context window](https://developer.apple.com/documentation/Technotes/tn3193-managing-the-on-device-foundation-model-s-context-window)
- [Hugging Face Transformers documentation: Padding and truncation](https://huggingface.co/docs/transformers/main/pad_truncation)
- [Hugging Face Transformers documentation: Summary of the tokenizers](https://huggingface.co/docs/transformers/tokenizer_summary)
- [Liu et al. (2023): Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172)
- [Cache replacement policies: First in, first out (FIFO)](https://en.wikipedia.org/wiki/Cache_replacement_policies)
- [Redis documentation: Key eviction](https://redis.io/docs/latest/develop/reference/eviction/)
- [Hugging Face Transformers documentation: Chat templates](https://huggingface.co/docs/transformers/main/chat_templating)
- [Hugging Face Transformers documentation: Caching (KV cache explanation)](https://huggingface.co/docs/transformers/main/cache_explanation)
- [vLLM documentation: Automatic prefix caching](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/)