Why does AI say "too many requests" after one message?

An AI API can return "429 too many requests" after one message because that call exceeds a token budget or shares an exhausted limit with other traffic. In our simulation, the second message of a long chat was blocked for tokens while 5.9 of its 6 request credits per minute were still free.

Why does AI say too many requests after one message?

The status code says "requests", but an AI API often meters more than one thing. The public rate limit pages of the Anthropic API and the OpenAI API both describe limits on requests per minute and on tokens per minute, and a request is rejected when any one of them is exceeded. Limits usually apply to an organization or project, so other traffic on the same key can drain them before your message arrives. The same 429 comes back whichever budget ran out, so read the error body and the limit headers.

In a client-managed chat, the app includes retained conversation history in each model request. That history counts as input again, subject to the provider's cache accounting. The app may trim or summarise earlier turns; our lab resends all 45 earlier turns. We showed the same effect for agents in why an AI agent re-sends its history, and the reason a chat forgets in my chat forgot my name.

How a token bucket decides

The Anthropic page says its limits use the token bucket algorithm, with capacity refilled continuously rather than reset at the top of each minute. Our simulation uses one bucket per budget and charges known input plus output tokens before admission:

requests bucket: capacity 6,      refill 6 / 60 s      = 0.1 credit per second
tokens bucket:   capacity 10,000, refill 10,000 / 60 s ≈ 166.7 tokens per second

a request passes only if  requests >= 1  AND  tokens >= input + output

Real providers count differently. Claude documents separate input and output limits, including cache-aware input accounting; OpenAI documents admission estimates involving the requested output limit. Those differences decide whether a long history exhausts the budget: if cached input does not count, a long chat with a cached prefix costs far less against the limit than our lab charges.

A short question takes 1 request credit and a few hundred token credits. A long-chat message also takes 1 request credit, but thousands of token credits, so the tokens bucket can be empty while the requests bucket is almost full.

Credits refill as min(capacity, credits + elapsed × refill_rate), so a blocked message that fits the capacity can pass later with nothing else changed. A charge above capacity never fits: a 12,000-token request against our 10,000-token bucket fails no matter how long the client waits. The only fixes are a smaller request (trim the history or lower the output limit) or a higher limit on the account. Our lab flags this case separately instead of retrying it.

What our lab measured

Our lab is a Node 22 simulation with a simulated clock and seed 21. Run node labs/ai-21/run.mjs from the repository root; the script prints the same numbers every time, and our run is saved in results.json. We made the limits up so the effect is easy to see, and no provider uses them. Token counts come from a fixed counting rule, not a provider tokenizer, so treat them as approximate.

Message	Input	Output	Approx. tokens	Requests
Short question in a new chat	130	344	~474	1
Same question after 45 earlier turns	8,151	344	~8,495	1

The long-chat message cost 17.9 times the tokens of the short one (8,495 / 474). We sent it twice, 8.94 simulated seconds apart:

send #1  t=0.00s  needs 8,495  tokens left before: 10,000  -> 200, tokens left 1,505
         refill over 8.94 s: about +1,490 tokens            -> about 2,995 left
send #2  t=8.94s  needs 8,495  tokens left: 2,995           -> 429 (tokens)
         requests left at that moment: 5.89 of 6

The message needed about 8,500 tokens and the bucket held about 3,000, so it was rejected while the request budget was almost untouched. The token deficit was 5,500, so with no competing traffic the earliest admission was 33 seconds later: 5,500 / (10,000 / 60). A short question would have passed: 474 is far below 2,995. Trimming or summarising old turns would shrink that charge; we did not measure that here.

Why agents hit the request limit instead

An agent can turn one message into many model calls. In the lab, one agent message planned 10 model calls with tool calls between them. Each call resent a growing context, but each was small compared with the long chat, so this time the requests bucket ran out first. Without retry, call 7 got a 429 for requests: 7 requests sent, 6 succeeded. Either budget can be the one that runs out, so a client has to track both. Routing adds its own calls too, as in who decides which AI answers your message, and a semantic cache can remove some calls entirely.

How to retry without making it worse

Retry only temporary throttling. The OpenAI page separates rate limit errors, which clear with time, from quota and billing errors, which need account action; retrying those just repeats the failure. For throttling, retrying at once achieves nothing in our simulation because the bucket has not refilled yet. The usual fix is exponential backoff with jitter, described in the AWS Architecture Blog: wait a random time between 0 and an upper bound that doubles after each failure, up to a cap. The randomness spreads out clients that failed together.

RFC 9110 allows Retry-After as a number of seconds or as an HTTP date, so the client has to parse both:

// backoff.mjs: full jitter, base 1 s, cap 30 s (the lab's settings), plus a deadline
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

function retryAfterMs(value, now = Date.now()) {
  if (value == null) return null;
  const v = value.trim();
  if (/^\d+$/.test(v)) return Number(v) * 1000;      // delay-seconds
  const at = Date.parse(v);                           // HTTP-date
  return Number.isNaN(at) ? null : Math.max(0, at - now);
}

// call(signal): must abort its request when signal fires (pass it to fetch or the SDK)
// isTemporary(res): true for throttling, false for quota or billing errors (read the error body)
export async function withBackoff(call, isTemporary,
    { base = 1000, cap = 30000, retries = 8, deadlineMs = 60000 } = {}) {
  const deadline = Date.now() + deadlineMs;
  let res = null;
  for (let attempt = 0; ; attempt++) {
    const left = deadline - Date.now();
    if (left <= 0) return res;                          // no time for another attempt
    res = await call(AbortSignal.timeout(left));        // the attempt itself is bounded too
    if (res.status !== 429 || attempt === retries || !(await isTemporary(res))) return res;
    const hinted = retryAfterMs(res.headers.get('retry-after'));
    const wait = hinted !== null ? hinted : Math.random() * Math.min(cap, base * 2 ** attempt);
    if (Date.now() + wait >= deadline) return res;      // the wait alone would pass the deadline
    await sleep(wait);
  }
}

The deadline only holds if call honours the signal; a call that ignores it can still run past the deadline. Remaining time is checked before every attempt, so a late timer cannot start a new request after expiry. Many SDKs also retry 429s on their own. Either turn that off (for example maxRetries: 0) or count those inner retries against the same attempt and time budgets, or one outer attempt can hide several real ones. The OpenAI rate limit guide recommends the same pattern of backoff with a bounded number of retries.

In the lab, backoff took failed messages from 3 of 7 to 0 of 7. The cost was waiting: 135 simulated seconds across all messages, and 52.9 seconds for the slowest one.

Trade-offs

Option	Good for	Costs you	Use when
Resend the full history	The model sees everything	Token cost grows with every turn	Short chats
Trim or summarise old turns	Smaller token charge per message	The model may lose details	Long chats near the token limit
Backoff with jitter	Fewer user-visible errors from temporary throttling	Seconds of waiting	Any client that can be throttled
A client-side token budget	Reduces predictable rate-limit rejections	Needs provider-specific estimates and coordination across workers	Agents and batch jobs

Our take

Log which limit was exhausted, next to the input and output token counts. Keep the history you send under control, and retry only temporary throttling, with backoff and jitter. Our simulation shows that waiting lets messages recover; it does not give you a tested retry schedule for a production API. The free course covers limits, queues and retries step by step.

The lab is a simulation on one machine with made-up limits and an approximate token count; it shows the shape of the effect, not any provider's real limits or timings. Provider behaviour is described from their public documentation, not from any company's private systems.