AI agent tokens: why history makes total input grow

AI agent tokens pile up for a structural reason. In an append-only agent loop, each model invocation includes the retained conversation as logical context: the client can upload it, or the service can reconstruct it from stored conversation state. Either way the input of call k is everything before it, so with roughly constant increments total input across N calls grows close to N squared, and most of it is old tool output counted again. Interview version: "Your coding agent's input token count is 5,000 times a chat. Where did the tokens go, and how do you bring the bill down?"

Prerequisites

The model keeps no memory between invocations. Each invocation computes over the system prompt, tool definitions and messages that make up its context, and nothing else. A multi-turn conversation exists because earlier turns are put back into that context, either uploaded again by the client or rebuilt by the service from stored conversation state. The context window article (AI 3) shows the app rebuilding that list on every turn and what happens when it no longer fits.

Tool calling. A model cannot read a file or run a test. It writes a structured request naming a tool and its arguments, the app runs the tool, and the result goes back to the model as a new message. The tool calling article (AI 5) walks through that loop. An agent is that loop repeated until the model writes a final answer instead of a tool call.

Tokens and prices. APIs bill input tokens (what you send) and output tokens (what the model writes) at separate prices per million; output usually costs more. Counts depend on each model's tokenizer, so any other count is an estimate.

Definitions

How does an agent re-send its history?

The lab behind this article (labs/ai-7) builds a toy coding agent. It is a synthetic simulation with estimated tokens, not a billed run and not any product's real numbers. Every request is serialized as one JSON body with system, tools and messages, in the shape of common chat APIs, and counted as a whole, framing included. Every count in this article is therefore a synthetic JSON-size token estimate that includes JSON keys, quotes and escaping. Real APIs do not tokenize the HTTP body; they render messages and tool definitions into a model-specific input format with its own control tokens, so even a real tokenizer run over the raw request JSON would not give billable usage. For production accounting use the provider's token-counting endpoint before a call and the usage object it returns after. The estimates are fine for ratios and shape, which is what the lab claims.

request 1   [system][10 tool schemas][task]
request 2   [system][10 tool schemas][task][msg 1][result 1]
request 3   [system][10 tool schemas][task][msg 1][result 1][msg 2][result 2]
...
request k   [fixed part F][everything from calls 1 .. k-1]

Calls 1 to N-1 each write a short message and one tool call (read_file, grep, run_tests, edit_file, list_dir); the tool result joins the history. Call N writes the final answer. Nothing is ever removed: the loop is append-only, which is the condition for everything that follows.

Who holds the history is a separate question from what the model computes over. The OpenAI conversation state guide describes both modes: pass earlier messages again yourself, or chain to a stored response with previous_response_id, in which case the guide states all previous input tokens in the chain are still billed as input. Server-side state saves upload bytes; it does not shrink the logical context.

Why the logical context has to be there on every call comes from how a transformer serves a request, in two phases:

• Prefill. All input tokens go through every layer in parallel. Each layer projects every token into a key vector and a value vector, and each token attends to all earlier ones. The result is the KV cache: per layer, a key tensor and a value tensor with one row per token. Cost has a part linear in the input length L (projections and MLP, about 2 FLOPs per parameter per token) and a quadratic attention part (scores between every pair of tokens, about 2·layers·L²·d_model FLOPs with a causal mask).
• Decode. Output tokens are generated one at a time. Each new token computes its own query, key and value, attends over the stored KV cache and appends one row. Decode cost per output token is linear in L and is usually limited by memory bandwidth, because the whole KV cache is read for every generated token.

The KV cache is per request. When the request ends the server frees it (or keeps parts of it under prompt caching, below), so call 20 cannot use anything computed on call 4 unless the same prefix is still cached. Some worked numbers for an 8B-class open model with grouped-query attention (32 layers, 8 KV heads, head dimension 128, 16-bit values, about 8 billion parameters, d_model 4,096); this is a public open-weight shape used as an example, not any provider's served model:

KV bytes per token  = 2 (K and V) × 32 layers × 8 heads × 128 dims × 2 bytes = 131,072 B = 128 KiB
request 30 (51,686) = 51,686 × 131,072 B = 6,774,587,392 B ≈ 6.31 GiB of KV state

prefill FLOPs, linear part    ≈ 2 × 8e9 × L
prefill FLOPs, attention part ≈ 2 × 32 × L² × 4,096
attention / linear            ≈ 32 × 4,096 × L / 8e9 = L / 61,035
at L = 51,686                 ≈ 0.85, so attention is already close to half of prefill

So "the model reads every token" hides three different costs: upload bytes (zero for server-held history), prefill compute (superlinear in uncached L), and KV memory plus decode bandwidth (linear in L). A prefix cache removes the prefill for the cached part: only the new tail is projected, and each new token still attends over the full context. Open-source servers show how this works inside: vLLM's automatic prefix caching splits the KV cache into fixed-size blocks, keys each block by a hash of its tokens plus the hash of the block before it (so a block only matches if the whole prefix matches), and evicts unreferenced blocks in LRU order. Because the hash is chained, one changed byte early in a prompt invalidates everything after it.

Here is the lab's medium profile, request by request (full size / new tokens in it):

[1]  2,230 / 2,230    [10] 17,747 / 1,261    [20] 39,949 / 2,003    [30] 51,686 / 1,375
[2]  3,123 /   893    [11] 18,894 / 1,147    [21] 40,455 /   506
[3]  6,190 / 3,067    [12] 19,304 /   410    [22] 44,603 / 4,148

The new part is small and noisy (319 to 4,942 tokens in requests 2 to 30, by tool). The full size only goes up. Request 30 is 51,686 tokens, and 1,375 of them are new: the tool result and message from call 29 plus framing. The other 50,311 tokens, exactly the size of request 29, are copies.

Why do AI agent tokens grow close to N squared?

Let R_k be the size of request k. With a fixed part F, increments a₁ .. a_N-1 and nothing removed:

R_k  = F + a_1 + a_2 + ... + a_(k-1)

Exact (any increments):
T(N) = R_1 + R_2 + ... + R_N
     = N·F + sum over j=1..N-1 of a_j · (N - j)   (step j is re-sent in N - j later requests)

Constant-increment approximation (a_j = a for all j):
T(N) = N·F + a·N(N - 1)/2                         (an arithmetic series)
S(N) = 1 - [F + (N - 1)·a] / T(N)                 (re-sent share)

The exact sum is a weighted sum: increment j has weight N - j. Plugging an ordinary average of the a_j into the approximation is exact only if the increments are evenly spread; a large result early in the task (high weight) pushes the real total above the approximation, a large one late pushes it below. The first term is linear in N, the second quadratic: the fixed part dominates short tasks, the history term long ones. Early tool output is the most expensive: a large file read on call 2 of a 30-call task is sent in 28 later requests (requests 3 to 30).

Two identities follow from this.

New tokens telescope. The new tokens in request k are R_k minus R_k-1, so their sum over the whole task is R_N, the size of the last request. In the lab at 30 calls: total new input is 51,686, which is request 30. Everything else in the total was sent before.

Re-sent share approaches 1 - 2/N when the history term dominates. Re-sent share is exactly 1 - R_N / T(N). Under the constant-increment approximation, once (N-1)·a is much larger than F, R_N is about aN and T(N) about aN²/2, so S approaches 1 - 2/N. At N = 30 that is 1 - 2/30 = 93.3%. The lab measured:

re-sent share = 1 - R_30 / T(30) = 1 - 51,686 / 863,762 = 1 - 0.0598 = 94.0%

The share depends on N, on the ratio F / a and on how the increments are spread over the task. Scaling the increments alone can change S, because F is unchanged: smaller increments make the fixed part a larger share of every request. In the lab F is small next to the history by call 30, so the three trimming profiles below land within 0.2 points of each other (94.1%, 94.0%, 93.9%).

Measuring the exponent. If T(N) behaves like c·N^p, then T(50)/T(10) = 5^p, so p = log(T(50)/T(10)) / log 5. From the lab:

T(10) =    98,386        T(50) = 2,358,122
T(50) / T(10) = 23.97                       (5x more calls)
p = log(23.97) / log(5) = 3.177 / 1.609 = 1.97
pure square would give 5² = 25x; the linear N·F term pulls it below 25

With the cruder characters divided by 4 estimator the ratio is 21.31x, so the conclusion does not hinge on the counting rule. The exponent is a local fit between 10 and 50 calls, not a law; the fixed part pulls it toward 1 for short tasks.

In system terms: logical input tokens grow linearly per call and quadratically across this append-only task. Upload bytes, inference work and latency require separate accounting. Upload bytes follow the token curve only if the client sends the history; with server-held state they stay near the size of each increment. Uncached prefill work grows faster than linearly per call, because its attention part scales with L²; with a warm prefix cache, prefill per call is roughly proportional to the new tail times the context length. KV memory and per-token decode cost scale with L. Doubling an agent's step limit roughly quadruples the history part of its logical input, and input-token rate limits hit long tasks far harder than the step count suggests (check whether your provider counts cache reads toward them; rules differ by provider and model).

Why are there three different "times a chat" numbers?

The baseline is one normal chat in the lab: 158 input tokens (system prompt and one question, same JSON framing) and 601 output tokens, at an illustrative $0.0095. Compared with the medium 30-call task:

read multiple      = agent input / chat input
                   = 863,762 / 158                                   = 5,467x
combined multiple  = (agent in + out) / (chat in + out)              = 1,141x
cost multiple      = agent cost / chat cost = $2.6271 / $0.0095      =   277x   (unrounded chat cost $0.009489)

The three differ because the two workloads have opposite shapes. The prices are illustrative, USD 3 per 1M input tokens and 15 per 1M output tokens, the shape of a common public price list and not a quote for any product. Output is priced 5x input.

one chat:   input  158 × 3 / 1M  = $0.000474
            output 601 × 15 / 1M = $0.009015      output = 95.0% of the chat's cost
agent, 30:  input  863,762 × 3 / 1M = $2.5913     input  = 98.6% of the agent's cost
            output = $2.6271 - $2.5913 = $0.0358  → about 0.0358 / 15 × 1M ≈ 2,388 output tokens
check:      (863,762 + 2,388) / (158 + 601) = 866,150 / 759 = 1,141x

The chat is mostly expensive output; the agent is mostly cheap input. Dividing input by input (5,467x) compares the part where the chat is tiny. Dividing cost by cost (277x) compares what you pay. Neither is wrong, but "one task costs as much as 1,000 chats" is a cost claim, and in this lab the medium profile does not reach it even at 50 calls (752x). Only the heavy profile at 50 calls passes it, at 1,411x. Always say which multiple you mean.

How does trimming old tool results change the token curve?

The answer depends on when the trimming happens.

Smaller tool results entering history (simulated). The lab runs three synthetic profiles over the same steps and the same content. Each keeps a fixed fraction of every tool output, so light is an exact prefix of heavy and only size changes:

30 calls      keeps   total input   last request   re-sent   cost plain   cost cached   cost x chat
light          20%       417,788         24,801     94.1%       $1.29       $0.2467          136
medium         50%       863,762         51,686     94.0%       $2.63       $0.4733          277
heavy         100%     1,581,678         96,275     93.9%       $4.78       $0.8425          504

Read it with the formula T(N) = N·F + a·N(N-1)/2. Trimming at entry lowers a, the slope of the history term. It does not touch F, the model's own messages or N. So:

• Total input does not scale with the keep fraction. Light keeps 20% of each tool output but sends 417,788 / 1,581,678 = 26.4% of heavy's input, because the fixed part and model messages are not trimmed.
• The re-sent share hardly moves here (94.1%, 94.0%, 93.9%), because by 30 calls the history term dwarfs F in all three profiles. With a larger F or a shorter task, the same trimming would move S more. The curve keeps its shape; it is scaled down. At 50 calls light is still 1,115,728 tokens, 96.0% re-sent.
• The cost multiple changes a lot, 136x to 504x at 30 calls, because input is almost all of the agent's cost.

In practice: grep with a match cap, file reads by line range, test output with failures only, paginated listings. Since a token added at call j is paid N - j more times, a cap on tool output is worth most early in a long task.

Removing already-saved history (concept, not simulated). Compaction, summarization and tool-result clearing go back and delete or shorten history that is already in the message list. The lab does not simulate any of these, so the following is concept arithmetic only. If every request were capped at some ceiling W, then T(N) ≤ N·(F + W), which is linear in N once the cap is reached. Applying a cap at the size of request 10 to the lab's own 30 request sizes:

concept only, not a lab result: cap every request at R_10 = 17,747
T = (R_1 + ... + R_10) + 20 × 17,747 = 98,386 + 354,940 = 453,326   vs 863,762 uncapped
ignores: the tokens a summary costs to produce, cache misses after each rewrite, lost detail

Real APIs offer this. The Claude context editing docs describe clearing the oldest tool results past a threshold and replacing them with placeholder text, and note that clearing invalidates cached prefixes, so a clear should remove enough tokens to be worth it. An edit prevents reuse of cached prefixes that include it. An eligible unchanged prefix before the edit can still hit; the remainder incurs the applicable uncached or cache-write rate. So trimming at entry keeps the prefix stable and cache friendly; rewriting history caps growth but breaks the part of the prefix after the edit.

When does a clear pay off? Assume the lab's illustrative cache prices (read $0.30, write $3.75 per 1M) and a cache that would otherwise stay warm. You clear C = 20,000 tokens of old tool output, and R = 10,000 tokens that sit after the edit point and were cached must be written again:

saving per later call   = C × read          = 20,000 × 0.30 / 1M          = $0.0060
one-time rewrite penalty = R × (write - read) = 10,000 × (3.75 - 0.30) / 1M = $0.0345
break-even calls         = 0.0345 / 0.0060  = 5.75  → pays off from the 6th later call

In general, with s summary tokens left in place of the cleared ones and a summary call that costs C_sum, clearing pays off over k later calls when

k × (C - s) × read  >  R × (write - read) + s × write + C_sum

Clear in large batches, place the edit as late in the prefix as you can (R small), and do it right before a long tail of calls. Without caching, the saving per call is C × input price and the clear pays off almost at once. When editing, keep the history valid: every tool call still needs its matching result (Claude pairs tool_use with tool_result by ID, OpenAI pairs a function call with its output by call_id), so replace a result's content with a placeholder rather than deleting one half of the pair.

What does prompt caching change, and what not?

The series' Part 33, cache-aside, is "the fastest request is the one you never make": the app checks a cache before the database and fills it on a miss. Prompt caching has the same aim with a twist: the request is still made and the full context is still the model input, but the provider reuses the stored key/value tensors of a prefix it has seen recently and runs prefill only on the new tail. The OpenAI prompt caching guide states that the cache stores KV tensors, not the tokens themselves. It works the same way a pull CDN serves a stored copy instead of going back to the origin.

How it works, from the official docs:

• Exact prefix match. A hit needs the request to begin with exactly the same tokens as a cached entry. In the Claude docs the prefix is built in the order tools, system, messages, and a change at one level invalidates that level and everything after it. The OpenAI docs make the same point: put a breakpoint after the static content, so content that changes comes after the reusable prefix.
• Breakpoints, Claude. cache_control marks cache points, up to 4 per request, with a backward lookback of up to 20 blocks from each breakpoint.
• Claude limits and prices. Minimum cacheable prefix from 512 to 4,096 tokens depending on the model. Default entry lives 5 minutes, refreshed on each use, with a 1-hour option. A cache read is 0.1x the base input price on many models (lower on some), a 5-minute write 1.25x and a 1-hour write 2x. Usage reports cache_creation_input_tokens and cache_read_input_tokens.

OpenAI's guide separates model generations, so the rules below are not interchangeable:

• GPT-5.6 and later. Implicit caching by default, or explicit mode with prompt_cache_breakpoint on content blocks (up to four cache writes per request). Minimum cacheable prompt 1,024 tokens. Default minimum lifetime 30 minutes; in the Responses API a 12-hour lifetime can be requested at a higher write rate. Writes cost 1.25x the uncached input rate for 30 minutes and 2.5x for 12 hours; reads 0.1x on most models. Each input token is billed at exactly one of the uncached, cached or write rates. Cached tokens are reported at the exact eligible boundary; usage has cached_tokens (reads only) and cache_write_tokens. prompt_cache_key is optional and used for separate cache accounting.
• GPT-5.5 and earlier. No separate cache-write charge. Retention is set with prompt_cache_retention: in-memory entries typically last 5 to 10 minutes of inactivity, up to one hour, and the 24-hour option on listed models typically lasts around 30 minutes and up to 24 hours. Cached tokens are reported rounded down to a multiple of 128. A stable prompt_cache_key helps route requests that share a prefix to the same cache.

Check the current page for the model you use; these details change between generations.

An agent is close to the ideal workload: every request is the previous request plus a small tail. The lab prices it with write-premium caching (the Claude style, and OpenAI from GPT-5.6), assuming every re-sent token is a cache read at 10% of input and every new token is written once at 125%; minimum size and expiry are ignored:

medium, 30 calls, illustrative USD per 1M: read 0.30, write 3.75, output 15
re-sent  = 863,762 - 51,686 = 812,076 × 0.30 / 1M = $0.2436
new      =           51,686 ×  3.75 / 1M        = $0.1938
output                                           = $0.0358
cached total ≈ $0.4733 (lab)    plain total = $2.6271 (lab)
saving = 1 - 0.4733 / 2.6271 = 82.0%

The token count stays at 863,762, with 94.0% re-sent and growth close to N squared. Caching lowers the price of the history term but leaves its shape and the context window as they were: request 50 (medium) is 95,962 tokens, cached or not. At these prices a write costs 0.25x extra and one read saves 0.9x, so a prefix pays off once it is read again. Under caching the history term still costs prefill on each new tail against the full context, and KV memory still scales with request size. What disappears is the repeated prefill of the prefix.

Worked example

The video pictures one task as a paper stack beside a photocopier: each call copies all earlier pages and adds one thin sheet. The lab's medium profile in numbers:

Inputs (synthetic, chosen by us)
  fixed part F (system + 10 tool schemas + task)  = 2,230 estimated tokens = request 1
  tool mix: read_file, grep, run_tests, edit_file, list_dir; seeded, same every run
  profile medium: keeps 50% of each tool output
  tokens: synthetic JSON-size token estimates over the whole request body
          (keys and escaping included), word-piece rule cross-checked with chars/4;
          no model tokenizer, not billable usage
  prices: illustrative USD 3 / 15 per 1M input / output, cache read 10%, write 125%

30 calls
  request 30            = 51,686   of which new 1,375
  total input           = 863,762
  total new input       = 51,686   (= request 30, telescoping)
  re-sent share         = 1 - 51,686 / 863,762 = 94.0%
  read multiple         = 863,762 / 158 = 5,467x     (chars/4: 3,512x)
  cost multiple         = 277x plain;  cost $2.63 plain, $0.4733 cached (82% less)

growth
  10 calls 98,386 → 50 calls 2,358,122 = 23.97x  (chars/4: 21.31x),  exponent 1.97

profiles at 30 calls (keep 20 / 50 / 100% of tool output)
  cost multiple 136x / 277x / 504x

The read multiple moves from 5,467x to 3,512x under the other estimator, because the two rules count the chat's short prose differently; the growth ratio and re-sent share (94.0% vs 94.2%) barely move. Trust ratios within a run, not a multiple against a tiny baseline.

Code

This script reproduces the growth from the lab's 30 request sizes in a few lines: it sums them, checks the telescoping identity, fits the F + a model, prints the re-sent share against 1 - 2/N, prices the cache, and computes the concept-only cap. Node 22, no packages.

// history_growth.mjs: why an agent's total input grows close to N squared.
// Node 22, no packages. Numbers are the lab's medium profile (labs/ai-7/output.txt):
// estimated tokens from a synthetic simulation, not a billed run.

// Full size of requests 1..30 (request 1 = fixed part F: system + 10 tool schemas + task).
const R = [2230, 3123, 6190, 7193, 8087, 9039, 13981, 14310, 16486, 17747,
  18894, 19304, 23888, 24207, 28556, 30372, 33449, 33997, 37946, 39949,
  40455, 44603, 45108, 45655, 48336, 49070, 49633, 49957, 50311, 51686];

const sum = (xs) => xs.reduce((a, b) => a + b, 0);
const fmt = (n) => Math.round(n).toLocaleString('en-US');

// 1. Measured: total input is the sum of every request; new tokens telescope to R[N-1].
const N = R.length;
const total = sum(R);
const fresh = R.map((r, k) => (k === 0 ? r : r - R[k - 1]));
console.log('total input      ', fmt(total));                        // 863,762
console.log('new tokens       ', fmt(sum(fresh)), '= last request', fmt(R[N - 1]));
console.log('re-sent share    ', ((1 - R[N - 1] / total) * 100).toFixed(1) + '%');

// 2. Constant-increment approximation: fixed part F plus the same increment a per call.
//    R_k = F + (k - 1) a,  T(N) = N F + a N (N - 1) / 2
//    a is derived from the two endpoints (first and last request), an approximation,
//    not a fit: real increments are lumpy and the exact total is the weighted sum
//    N F + sum_j a_j (N - j), which is what `total` above measures.
const F = R[0];
const a = (R[N - 1] - F) / (N - 1);
const T = (n) => n * F + (a * n * (n - 1)) / 2;
console.log('avg increment a  ', fmt(a));
console.log('model T(30)      ', fmt(T(30)), 'vs measured', fmt(total));
for (const n of [5, 10, 20, 30, 50, 100]) {
  const share = 1 - (F + (n - 1) * a) / T(n);
  console.log(`  N=${String(n).padStart(3)}  T=${fmt(T(n)).padStart(10)}  re-sent ${(share * 100).toFixed(1)}%  (1 - 2/N = ${((1 - 2 / n) * 100).toFixed(1)}%)`);
}
console.log('model growth 10 to 50', (T(50) / T(10)).toFixed(2) + 'x', ' exponent', (Math.log(T(50) / T(10)) / Math.log(5)).toFixed(2));

// 3. Illustrative prices, USD per 1M tokens (assumed, not a quote): input 3, cache read 10%, write 125%.
const P = { input: 3, read: 0.3, write: 3.75 };
const plain = (total * P.input) / 1e6;
const cached = ((total - R[N - 1]) * P.read + R[N - 1] * P.write) / 1e6; // every new token written once, every old token read
console.log('input cost plain ', '$' + plain.toFixed(4), ' cached', '$' + cached.toFixed(4));

// 4. Concept only (the lab does NOT simulate this): cap every request at the size of request 10.
const cap = R[9];
const capped = sum(R.map((r) => Math.min(r, cap)));
console.log('capped at', fmt(cap), '->', fmt(capped), 'total input (ignores summary cost and cache misses)');

node history_growth.mjs
node labs/ai-7/run.mjs      # the full lab, from the repository root

What to observe in history_growth.mjs:

• Telescoping. New tokens sum to 51,686, exactly the last request, so the re-sent share is 94.0%.
• The approximation is close, not exact. With the endpoint-derived a = 1,705 the model gives 808,740 for 30 calls against the measured 863,762, 6.4% low, because real increments are lumpy and some large tool results land early. The modelled re-sent share follows 1 - 2/N closely from 20 calls up (90.6% vs 90.0% at 20, 98.0% at 100).
• The model's own exponent between 10 and 50 calls is 1.93 (22.22x); the lab measured 1.97 (23.97x). Both sit just under 2 because of the linear F term.
• Caching takes the input cost from $2.5913 to $0.4374. Add the $0.0358 of output and you get the lab's $2.6271 and roughly $0.4733.
• The cap gives 453,326 tokens. It is a concept, not a lab result.

What to observe in node labs/ai-7/run.mjs: it runs in a few seconds with no network and no API key, prints the three profile tables for N = 5, 10, 20, 30 and 50, all 50 per-request sizes for the medium profile, the chars/4 cross-check and a final RESULT JSON line. Same seeds, same numbers every run. Check that the "new input" column always equals the "last request" column: that is the telescoping identity again.

Failure modes and edge cases

Hitting the context window. Request size grows linearly, so a long task eventually exceeds the model's context window and the call fails. Detect it by logging request size per call and alerting at a fraction of the limit; fix it with tool output caps, sub-tasks that start with a fresh context, or compaction. The Claude context windows page also warns that accuracy and recall degrade as token count grows, so a request that fits is not automatically a good one.

Silent cache misses. A timestamp in the system prompt, tool definitions serialized in a different key order, a tool added mid-task or an edit to an early message changes the prefix. Cached prefixes that include the change can no longer be reused; an eligible unchanged prefix before it can still hit, and everything after it incurs the applicable uncached or cache-write rate. A change in the system prompt or tools is the worst case, because almost nothing precedes it. Detect it from the usage fields, reads and writes both: Claude reports cache_read_input_tokens and cache_creation_input_tokens; OpenAI reports cached_tokens and, on models with write pricing, cache_write_tokens. In a healthy append-only loop, reads rise with the call index and writes stay near the size of each new tail. Writes close to the whole request size on consecutive calls mean the prefix is unstable.

Cache expiry during slow steps. A tool that outlasts the TTL (a long test suite, a human approval) leaves no reusable prefix, so the next request bills the whole history at the uncached or cache-write rate, depending on provider and model. Use a longer TTL where offered, and price the longer TTL's higher write rate against how often your steps actually exceed the short one.

One huge early tool result. An unfiltered log on call 2 is re-sent on every later call. Rank tool results by size times remaining calls; fix it at the tool.

Budget checks per call instead of per task. Each call looks reasonable; the task does not. Check the running total against a per-task budget on every call.

Reasoning tokens. The lab does not simulate them. Where an API expects reasoning items passed back, they join the history and the bill.

Trade-offs

Common mistakes

In an interview

1. Clarify: how many calls does a typical task take, how big are tool outputs, and is the concern cost, latency or the context limit?
2. Assumptions: state F, the average increment a and N; say prices are assumed.
3. Structure: client agent loop, model with no memory between invocations, tools, the retained message list (client-side or server-stored) as the only state, usage logging.
4. Deep dive: request k = F + history; T(N) = N·F + a·N(N-1)/2; new tokens telescope to R_N; re-sent share near 1 - 2/N.
5. Trade-offs: tool output caps vs missed detail, caching vs prefix stability, compaction vs cache misses and lost facts.
6. Wrap up: metrics per task (total input, cached share, largest tool result, peak request size) and a per-task budget.

Practice

Cheat sheet

Further reading

• Claude API docs, Prompt caching: prefix order, breakpoints, minimum sizes, TTL, prices, invalidation.
• OpenAI API docs, Prompt caching: KV tensors, implicit and explicit breakpoints, per-generation lifetime and reporting, cached_tokens and cache_write_tokens.
• vLLM docs, Automatic prefix caching: block hashing chained over the prefix and LRU eviction, an open implementation of the same idea.
• OpenAI API docs, Conversation state: passing history yourself vs stored, chained responses; earlier input still billed.
• Claude API docs, Context windows: how the window fills, and why more context is not automatically better.
• Claude API docs, Context editing: clearing old tool results, and the effect on cached prefixes.
• Claude API docs, Token counting: real tokenizer counts instead of estimates like ours.

Earlier in the series: why text appears before the reply is complete in streaming AI replies (AI 2), and how retrieval keeps prompts small by sending only the chunks that matter in RAG retrieval (AI 4). Tool results are also untrusted input that stays in history, as the prompt injection article (AI 6) shows.

This article describes general patterns for agent loops, illustrated with our own synthetic simulation in labs/ai-7. Token counts are estimates from a word-piece rule cross-checked with characters divided by 4, not a model tokenizer; prices are illustrative and nothing was billed. The numbers show the shape of the effect, not any company's product, prices or private systems.