Streaming AI replies: why text arrives before completion

Streaming AI replies exist because a language model produces its reply one token after another. After the request waits in a queue and the model reads the whole prompt (prefill), each step of ordinary decoding adds one token, so the first piece exists after queueing, prefill and one step, while the full reply exists only after the last step. An app can wait for the end and send everything together, or send each piece as soon as it is made. This part measures what each choice changes: when the first text arrives, and when the last one does. Interview version: "Why do chat apps stream their replies, and what does streaming actually improve?"

Prerequisites

Latency and throughput. Latency is how long one thing takes from start to finish; throughput is how many things finish per second. They are different questions with different fixes, and a system can be good at one and bad at the other. If those words are new, read why ten lanes can still be slow (Part 6, latency vs throughput) first. This article splits latency itself into two parts: how long until something starts arriving, and how long until everything has arrived.

Request and response over HTTP. A browser or app sends a request; the server sends back a response made of a status line, headers and a body. Nothing in HTTP says the body must be sent in one go. A server can start the body, keep the connection open, and keep writing more bytes until it is done. That small fact is all streaming needs.

Tokens. A model does not write characters or whole sentences. It writes tokens: pieces of text from a fixed vocabulary, often a whole short word and often part of a longer one. "streaming" might be one token in one vocabulary and "stream" plus "ing" in another. The context window article (AI 3) counts tokens in detail; here it matters only that a reply is produced as a sequence of them.

Definitions

How streaming AI replies work

A decoder-only transformer generates a reply in two phases, and the split between them is what makes streaming possible. This section describes ordinary autoregressive decoding without speculative decoding; the variation is covered after the decode loop.

Prefill. The whole prompt (instructions, conversation, your message) goes through the model in one forward pass. In every layer, each position is projected into a query, a key and a value vector. Attention is causal: position i may attend only to positions 0 to i, enforced by a mask that sets the scores of later positions to minus infinity before the softmax. Prefill processes all prompt positions at once as large matrix multiplications, so it is mostly compute bound, and its time grows with prompt length (the attention part grows with the square of it).

Logits and sampling. The final hidden state of the last position is multiplied by the output (unembedding) matrix, giving one score, a logit z, per vocabulary entry. The decoder turns logits into probabilities with a temperature T, p_i = exp(z_i / T) / Σ_j exp(z_j / T), optionally keeps only the top k entries or the smallest set whose probabilities sum to p (top-p, nucleus sampling), and then picks one token: the highest (greedy) or a random draw. The Hugging Face article on decoding methods walks through these strategies. That one token ID is the first model output.

Decode with a KV cache. Because the mask is causal, adding a token never changes the keys and values of earlier positions. So the server keeps every layer's keys and values in a KV cache and, on each decode step, runs only the newest token through the model: compute its query, key and value, append the key and value to the cache, attend over all cached keys, produce logits, sample. Without the cache, step t of a reply with a P-token prompt would recompute everything for all P + t positions. Two parts grow differently. Recomputing the per-position projections costs Σ O(P + t) over the reply, quadratic in output length. Recomputing full dense attention over each growing prefix costs Σ O((P + t)²), cubic in output length for a fixed prompt length. The cache removes the recomputation but not attention itself: each cached step still scores the new query against all P + t cached keys, so cached decoding still accumulates Σ O(P + t) attention work, quadratic as the context grows. In Hugging Face Transformers this cache is the past_key_values object (a DynamicCache by default) that generate() passes from step to step (see the KV cache documentation in the sources).

Speculative decoding changes the step count, not the dependency. A small draft model (or extra prediction heads) proposes several tokens; the target model scores all of them in one forward pass and accepts the longest prefix that matches what it would have produced, plus one token of its own. One verification step can therefore emit several tokens, so verification steps, output tokens and streamed events need not correspond one to one. Each accepted token still depends on every token before it. Hugging Face describes the variants (assisted decoding, prompt lookup) in its generation strategies guide.

Why decode is slow per token. A decode step does little arithmetic per byte it reads: for a dense model it must read every weight plus that sequence's cache from memory to produce one token (a mixture-of-experts model reads only the experts routed for that token). At small batch sizes it is bound by memory bandwidth, not compute. That is why servers batch many sequences into each step, and why time per token depends on how busy the server is.

How big the cache gets. For a model using grouped-query attention (fewer key/value heads than query heads, see Ainslie et al., GQA), assuming a dense model, full-context attention, an unquantized KV cache and no prefix sharing:

KV cache bytes = 2 (K and V) × layers × tokens × KV heads × head dimension × bytes per element

assumed: dense model, full-context attention, unquantized KV cache, no prefix sharing
         32 layers, 4,096 tokens, 8 KV heads, head dimension 128, 2 bytes (fp16 or bf16)
         2 × 32 × 4,096 × 8 × 128 × 2 = 536,870,912 bytes = 512 MiB per sequence
         per token: 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB

That excludes weights, activations and allocator overhead. A sliding-window layer caps its cache at the window size instead of growing across the whole context, and a quantized cache shrinks the bytes per element, so check the model's cache type before reusing this number. Now connect it to latency with more assumed numbers: if 64 GiB of accelerator memory is left for the cache, at most 64 GiB / 512 MiB = 128 full-length sequences fit. A further request must wait in the queue (its TTFT grows by the wait) or the server must preempt a running sequence and later swap its cache back or recompute it (the running reply stalls mid-stream). Each step also reads every active sequence's cache: with 16 GB of weights and 64 full-length sequences (32 GiB of cache, about 34.4 GB) on a device with 2 TB/s of memory bandwidth, one step needs at least (16 + 34.4) GB / 2 TB/s ≈ 25 ms just to move bytes, against 8 ms for the weights alone at batch size 1. Larger active batches can increase decode-step latency; the effect depends on bandwidth, compute utilization and scheduling. vLLM's PagedAttention (Kwon et al. in the sources) attacks the memory side by storing the cache in fixed-size blocks, much like operating system pages, so sequences of unknown final length do not reserve one large contiguous region each.

From token IDs to text. The serving loop hands each sampled token ID to a detokenizer. With byte-level tokenizers one token can be an incomplete UTF-8 sequence, so a server often holds it until the bytes form valid characters. A first model token therefore does not always mean first text.

Those phases give a timing model for one reply of n tokens. Keep server and client measurements apart:

server TTFT              = queueing + prefill (grows with prompt length) + first sample
server generation time   = server TTFT + Σ (decode intervals for tokens 2 to n)
                         ≈ server TTFT + (n - 1) × mean time per token   (ordinary decoding)

client first parsed text = request transit + server TTFT + response transit
                           + any proxy buffering + parsing of one complete event
client completion        = server generation time + request and response transit
                           + any transport delay not overlapped with generation

Network time belongs to the client numbers, not to server TTFT. Nor does transport add a fixed cost to every decode step: while token k travels, token k + 1 is being generated, so transmission overlaps generation. It adds to completion only when the path or the client is slower than the generator and backpressure holds output back.

The key point is what exists when. After the first step, one token exists. After step 60, sixty tokens exist. The full reply exists only after step n. A buffered app can show the reply only at the end of that sum. A streamed app can show the first token at the start of it and then grow the text step by step.

Nothing about streaming changes the loop itself. The model does the same work and produces the same tokens in the same order. Streaming is a decision about delivery: where the tokens wait between being made and being shown. In a buffered design they wait in a server buffer until the last one arrives. In a streamed design they go straight out.

Does streaming finish sooner?

Not inherently, and the measurement shows why.

Simulation in one Node.js process on our Mac (Apple M2 Pro), not a provider benchmark: no real model, no HTTP server and no network.

Our lab (labs/ai-2) runs one simulated generator that writes a fixed 120-token reply, one whole word per token, with a delay of 33 ± 8 ms per token from a seeded random source, so the full reply takes about 4 seconds. Two in-process endpoints (byte streams, no socket) use the same generator: one buffers the reply and sends it as one JSON body; the other sends one SSE event per token. A client calls each endpoint 20 times, all 40 calls at once on one event loop, and records when it has parsed the first text and the last text.

Two things stand out. The first text arrived 3,943 ms sooner when streaming: about one token's delay instead of the whole reply. And the total time barely moved: the completion-time medians differed by 5.9 ms, about 0.15% of the buffered median, within the lab's chosen 5% tolerance. These trials do not establish the cause or statistical significance of that difference. The p95 values come from only 20 concurrent calls per mode, so treat them as coarse. In this lab, streaming did not make the reply finish sooner; it made it start sooner. In general, streaming does not inherently accelerate generation; client completion time can change because transport overlaps generation or backpressure delays it.

So why does it matter? Because the wait a person feels is mostly the empty wait before anything appears. Usability research has long used rough limits of about 0.1 s for feeling instant, about 1 s for keeping the flow of thought, and about 10 s for keeping attention on the task (see the Nielsen Norman Group article on response times). A 4-second blank screen sits between the last two limits. A reply that starts after a few tens of milliseconds and keeps growing gives the reader something to read while the rest is generated, and reading speed is usually slower than generation speed, so the text stays ahead of the reader.

This is also why "it feels faster" and "it is faster" are different claims. Our lab measured when text was available to the client, not when a person read it. The benefit to a reader is practical, not measured here: you can start reading after the first line instead of after the last one.

How streaming travels: SSE, chunks and frames

Streaming needs a response body that can be delivered in pieces. HTTP has supported that for a long time.

HTTP/1.1 chunked transfer coding. When a server does not know the body length in advance, it can send Transfer-Encoding: chunked and write the body as a series of chunks, each prefixed with its size in hexadecimal, ending with a zero-size chunk (RFC 9112, section 7.1). Each chunk can be sent the moment it is ready.

HTTP/2 and HTTP/3 frames. These versions do not use chunked coding; the body is already a sequence of DATA frames on a stream, and a server can send a frame whenever it has bytes.

Server-Sent Events on top. Chunks and frames move bytes, but they do not say where one message ends. SSE adds that framing as plain text. The response has Content-Type: text/event-stream, and each event is one or more data: lines followed by an empty line:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache

data: "A "

data: "model "

data: "does "

data: [DONE]

The final [DONE] is an application convention that many APIs use, not SSE syntax; to the SSE format it is just another data payload. The client splits on blank lines, reads each data: payload and appends it to the screen. Chunk edges do not have to match event edges: one network read may contain half an event or three events, so a correct client buffers until it sees the blank line. The WHATWG parsing rules (in the sources) also say lines may end in CR, LF or CRLF, several data: lines in one event are joined with a line feed, one space after the colon is dropped, lines starting with a colon are comments, and an event still incomplete when the stream ends is discarded. The browser's built-in EventSource API does this for you (see MDN and the WHATWG specification), but it only makes GET requests and cannot send a request body, so chat apps that POST a conversation usually read the stream with fetch() and a ReadableStream and parse the events themselves.

One event per token is a choice. Our lab sends one event per token to make the effect easy to see. Grouping tokens cuts overhead and adds holding delay, and the delay is not small at real token rates. With batches of k emissions arriving every d ms, the first emission in a batch waits (k - 1) × d and the average emission in a full batch waits d × (k - 1) / 2:

k = 4, d = 33 ms
first emission in the batch waits   3 × 33 = 99 ms
average holding delay               33 × 3 / 2 = 49.5 ms

The overhead side, from the code sample below: the same 18-token reply is a 93-byte body buffered ({"text":"..."}) and a 276-byte body as SSE, because each event adds 10 bytes (data: , two JSON quotes and the blank line) and the end marker adds 14. Those counts exclude HTTP and TLS framing. Grouping 4 tokens per event divides the number of events, and so the framing bytes, by about 4 on a long reply. Keep three layers apart when counting: application events (what the parser dispatches), writes (each res.write() call), and transport units (TCP segments, TLS records, HTTP/1.1 chunks or HTTP/2 DATA frames). They do not map one to one: the kernel may coalesce several small writes into one segment or split one write across several, so count events at the application, never packets. A time window (flush every 20 to 50 ms) bounds the added delay better than a fixed token count when token speed varies. Either way, the first text arrives long before the last.

Alternatives exist: WebSockets give a two-way channel and suit apps that also stream input (voice, for example); long polling and plain chunked text without SSE framing also work. SSE stays popular for text replies because it is one ordinary HTTP response, passes through most infrastructure, and needs no protocol upgrade.

What quietly turns a stream back into one reply?

Streaming only works if every hop between the generator and the screen passes bytes on as they arrive. Any one buffering hop turns the stream back into a buffered reply, and the server's own logs will still say it streamed.

The Nginx row is the classic one. Its proxy module buffers upstream responses by default so a slow client does not hold the upstream busy; that is a good default for normal pages and the wrong one for a token stream. The reverse proxy article explains why proxies sit there in the first place, and the layer 4 vs layer 7 article explains why a layer 7 hop is the one that reads and may hold the body.

Worked example

The video's scenario: a demo chat app asks a question, and the reply is 120 tokens. Using the lab's setup (assumed delays, not a real model):

tokens per reply                 n  = 120
delay per token (requested)      d  = 33 ms (± 8 ms, seed 42)
time before the first token      t0 = 0 ms   (the lab skips prompt reading)

expected total   ≈ t0 + n × d = 0 + 120 × 33 = 3,960 ms
measured total   = 3,974.8 ms (wait) and 3,980.7 ms (stream), medians of 20 runs
                   (above 3,960 ms; timer lateness, the randomized ± 8 ms delays and
                    40 calls sharing one event loop can all contribute, and the lab
                    does not separate them)

wait:   first text = total        = 3,974.8 ms
stream: first text ≈ t0 + 1 × d   ≈ 33 ms     measured 31.8 ms (median)

first text sooner by streaming    = 3,974.8 - 31.8 = 3,943.0 ms
total time difference             = 3,980.7 - 3,974.8 = 5.9 ms

Now add a real-world first-token time. Suppose (assumed) a model spends 600 ms reading the prompt and queueing, and the network adds 50 ms each way. The buffered app shows text after about 600 + 50 + 3,960 + 50 = 4,660 ms. The streamed app shows its first text after about 600 + 50 + 33 + 50 = 733 ms. The finish is still about the same for both; the start moves by almost the whole generation time.

Code

A complete experiment in one file: a Node.js HTTP server with a buffered endpoint and an SSE endpoint, both fed by the same generator, and a client that measures first text and total time for each. It needs Node.js 18 or newer and no packages.

// stream_vs_wait.mjs: one HTTP server, two endpoints, the same token generator.
// Run: node stream_vs_wait.mjs   (Node 18+, no packages)
import http from 'node:http';
import { setTimeout as sleep } from 'node:timers/promises';

const WORDS = 'A model does not write the whole answer at once. It makes it one piece at a time.'.split(' ');
async function* generate(signal) {          // stand-in for a model: one token every 33 ms
  // timers/promises removes its own abort listener when the timer fires, so nothing piles up per token
  for (const w of WORDS) { await sleep(33, undefined, { signal }); yield w + ' '; }
}

// Resolve on 'drain' or 'close', whichever comes first, and remove BOTH listeners on every exit.
function drainOrClose(res) {
  return new Promise((resolve) => {
    const done = () => { res.off('drain', done); res.off('close', done); resolve(); };
    res.on('drain', done);
    res.on('close', done);
  });
}

// Write one chunk. If write() returns false the data is queued in memory:
// wait for 'drain' before writing more, and stop if the client leaves meanwhile.
async function send(res, chunk, signal) {
  if (res.write(chunk)) return;
  signal.throwIfAborted();
  if (!res.destroyed) await drainOrClose(res);  // 'close' may already have fired on a destroyed response
  signal.throwIfAborted();
}

const server = http.createServer(async (req, res) => {
  const ac = new AbortController();         // both modes: a client that leaves cancels generation
  res.on('close', () => { if (!res.writableFinished) ac.abort(new Error('client closed')); });
  try {
    if (req.url === '/wait') {              // buffered: collect everything, send once
      let text = '';
      for await (const t of generate(ac.signal)) text += t;
      res.writeHead(200, { 'content-type': 'application/json' });
      res.end(JSON.stringify({ text }));
    } else if (req.url === '/stream') {     // streamed: one SSE event per token
      res.writeHead(200, {
        'content-type': 'text/event-stream',
        'cache-control': 'no-cache',
        'x-accel-buffering': 'no',          // ask an Nginx proxy not to buffer this response
      });
      res.flushHeaders();                   // send headers now; Node otherwise waits for the first write
      for await (const t of generate(ac.signal)) await send(res, `data: ${JSON.stringify(t)}\n\n`, ac.signal);
      res.end('data: [DONE]\n\n');          // [DONE] is our app's end marker, not SSE syntax
    } else { res.writeHead(404).end(); }
  } catch (err) {
    if (ac.signal.aborted) return;          // a cancelled generation is not a server error
    console.error('generation failed:', err);
    if (!res.headersSent) {                 // nothing sent yet: a normal HTTP error
      res.writeHead(500, { 'content-type': 'application/json' }).end('{"error":"generation failed"}');
    } else if (!res.writableEnded && !res.destroyed) {   // mid-stream: tell the client in-band
      res.end(`event: error\ndata: ${JSON.stringify('generation failed')}\n\n`);
    } else {
      res.destroy();                        // cannot write any more: drop the connection
    }
  }
});

// Minimal SSE parser following the WHATWG rules this server needs: lines end in \n, \r\n or \r,
// data: lines are joined with \n, event: sets the type, a blank line dispatches the event
// (type 'message' if none was set) and resets both, comments (:) are ignored.
function sseParser(onEvent) {
  let buf = '', data = [], type = '';
  return (str) => {
    buf += str;
    for (;;) {
      const m = /\r\n|\r|\n/.exec(buf);
      if (!m) return;                                            // incomplete line: wait for more
      if (m[0] === '\r' && m.index === buf.length - 1) return;   // lone \r at the end may be half of \r\n
      const line = buf.slice(0, m.index);
      buf = buf.slice(m.index + m[0].length);
      if (line === '') { if (data.length) onEvent(type || 'message', data.join('\n')); data = []; type = ''; continue; }
      if (line.startsWith(':')) continue;                        // comment, e.g. a keep-alive
      const i = line.indexOf(':');
      const field = i < 0 ? line : line.slice(0, i);
      let value = i < 0 ? '' : line.slice(i + 1);
      if (value.startsWith(' ')) value = value.slice(1);
      if (field === 'data') data.push(value);
      else if (field === 'event') type = value;
    }
  };
}

async function measure(path) {
  const t0 = performance.now();
  const res = await fetch(`http://127.0.0.1:${server.address().port}${path}`);
  const headersMs = performance.now() - t0;           // headers arrived: says nothing about text
  const reader = res.body.getReader();
  const dec = new TextDecoder();
  let bytes = 0, firstText = null, lastText = null, raw = '';
  let complete = false, error = null;                 // success needs an explicit terminal marker
  const onText = (s) => {                             // called only with parsed reply text
    if (!s) return;
    const now = performance.now() - t0;
    if (firstText === null) firstText = now;
    lastText = now;
  };
  const feed = path === '/stream'
    ? sseParser((type, d) => {
        if (type === 'error') error = JSON.parse(d);  // the server reported a failure in-band
        else if (d === '[DONE]') complete = true;     // our app's end marker
        else onText(JSON.parse(d));
      })
    : (s) => { raw += s; };
  try {
    for (;;) {
      const { value, done } = await reader.read();
      if (done) break;
      bytes += value.byteLength;                      // count body bytes before decoding
      feed(dec.decode(value, { stream: true }));      // stream: true keeps split UTF-8 characters whole
    }
    feed(dec.decode());
    if (path === '/wait') {                           // text exists only once the whole JSON parses
      if (!res.ok) error = `HTTP ${res.status}`;
      else { onText(JSON.parse(raw).text); complete = true; }
    }
  } catch (e) {
    error = `read or parse failed: ${e.message}`;     // connection reset, truncated JSON, bad event
  }
  if (!complete && !error) error = 'stream ended without [DONE]';   // clean EOF is not success
  const r = (x) => (x === null ? null : Math.round(x));
  return { path, complete, error, headersMs: r(headersMs), firstTextMs: r(firstText),
           lastTextMs: r(lastText), closedMs: r(performance.now() - t0), bodyBytes: bytes };
}

server.listen(0, '127.0.0.1', async () => {
  console.table([await measure('/wait'), await measure('/stream')]);
  server.close();
});

What to observe. The reply has 18 tokens, so both lastTextMs values land a little above 18 × 33 = 594 ms (timers tend to fire late, not early). The columns separate the timestamps from the definitions: headersMs for /stream is a millisecond or two because flushHeaders() sends the headers before any token exists, which is exactly why first byte cannot stand in for first text. For /wait it is close to the total, because the headers leave with the body. firstTextMs for /stream is about one token's delay, and for /wait it equals lastTextMs, because the JSON only parses once the body is complete. closedMs is when the connection finished, tracked apart from the last text. bodyBytes counts raw bytes before decoding: 93 for /wait and 276 for /stream, the overhead worked out in the batching section above. complete is true only when the reply ended with its terminal marker ([DONE] for the stream, a parsed 2xx JSON body for the buffered call); a stream that reaches a clean end of file without [DONE] is reported as incomplete, and an event: error fills error, so a truncated reply never passes as a timing result.

Three production details are in the server, not just the client. Failures are handled, not thrown out of the request handler: a non-abort error is logged, then answered with HTTP 500 if no headers were sent, with an event: error if the stream is still writable, and otherwise by destroying the response. Both endpoints listen for close on the response and abort the generator through an AbortSignal, so a user who leaves stops the work in either mode. And the stream respects backpressure: Node's response.write() returns false when the data was queued in user memory instead of handed to the kernel, and the documented contract is to wait for drain before writing more (see the Node.js documentation in the sources). The send() helper waits for drain or close, whichever comes first, and removes both listeners on every exit, so a slow or vanished client cannot make the server pile up output, and a long reply with repeated backpressure does not pile up listeners either (a Promise.race() of two once() calls would leave the losing listener attached each time). The per-token delay uses setTimeout from node:timers/promises with { signal } for the same reason: it detaches its abort listener when the timer fires.

To watch the raw events, keep the server running (remove server.close() and print the port) and use curl with buffering off:

# -N turns off curl's own output buffering so each event prints as it arrives
curl -N http://127.0.0.1:<port>/stream

The numbers in this article come from our lab script (labs/ai-2/run.mjs in our public lab repository), an in-process version of this experiment with 120 tokens and 20 calls per mode; the code above lets you repeat the idea over real HTTP, and your times will differ. The lab prints the table, writes the word-by-word arrival times of one run to timeline.json (the video's phone replays exactly those times), and ends with a RESULT line holding every metric. Re-runs differ by a few milliseconds; a re-run on 2026-11-05 printed 32.2 ms and 3,984.8 ms for the first-text medians.

Failure modes and edge cases

A buffering hop. The most common bug: the endpoint streams in development and arrives all at once in production. Time to first byte alone will not catch it: headers or a keep-alive comment can arrive promptly while every text event waits in a proxy buffer. Detect it from the text events themselves. Record a client timestamp for each parsed text event and look at two numbers: first parsed text minus request start, and the inter-event gaps. A healthy stream shows gaps near the time per token (about 33 ms in our lab). A buffered one shows a long silence followed by many events within a millisecond or two of each other, often at buffer-sized boundaries (Nginx's default proxy buffers are one memory page each, 4 KiB or 8 KiB). Compare curl -N straight to the app with curl -N through the public URL, and alert on p95 first parsed text and on the share of streams whose median gap is below a few milliseconds. Fix the hop (table above).

Errors after the first byte. Once a 200 status and some tokens are sent, the server cannot change the status code. If generation fails halfway, the client must learn about it in-band: send an error event (event: error with a data line, as the code sample does) and have the client show the partial text as incomplete. The client must also treat a stream that ends without the app's terminal marker as incomplete, because a crashed process or a proxy timeout produces a clean-looking end of file with no error event at all. Buffered responses do not have this problem; they can return a 500 with no partial reply.

Moderation and post-processing. Some checks need the whole reply (a final format validation, a full-text filter). Bounded lookahead supports checks whose required context fits within that window. Checks requiring the complete reply must retain the complete reply before releasing it. Retraction cannot undo text already delivered. A hold-back of w tokens adds w × d to every token: 10 tokens at 33 ms is 330 ms. Streaming makes "show early" and "check everything first" pull in opposite directions, so decide per check which side it is on.

Client disconnects. If the user closes the tab, generation keeps running in either mode unless the server cancels it; streaming does not cancel anything by itself. Both buffered and streaming handlers can detect the disconnect (in Node, the response's close event fires before writableFinished is true), and both should propagate it to the generator: an AbortSignal in the code above, and on a model server a request abort that frees the sequence's batch slot and KV cache. Without it, abandoned replies keep consuming decode steps and cache memory that queued requests are waiting for.

Slow clients and backpressure. A phone on a weak network can read slower than the model writes. If the server ignores write() === false, Node keeps the unsent output in process memory, and many slow streams add up. Wait for drain (while also watching for close), monitor queued output bytes per response (res.writableLength), and set a slow-client limit: for example, end the stream with an error event and cancel generation when one response has more than 1 MiB queued or has not drained for 30 seconds.

Reconnects and resume. Reconnection is client behaviour, not something the format does on its own. The native EventSource reconnects automatically after a dropped connection (waiting the retry: interval if the server set one) and sends the last id: it recorded in a Last-Event-ID header. A fetch() reader gets none of that: it must implement retry, backoff and resume itself. IDs only help if the server stores the events of each in-flight reply (for example in Redis, keyed by reply ID, with a TTL) so it can replay everything after the given ID, and the client must deduplicate events it already showed. Without replay storage, many apps simply restart or show the partial text with a retry button. And because EventSource reconnects when the server closes normally too, the client must call close() itself when it sees the app's end marker such as [DONE], or it will reopen the request.

Rendering cost on the client. Re-rendering a long reply on every token (markdown parsing, syntax highlighting, layout) can make the page stutter. Batch DOM updates per animation frame, or append text nodes instead of re-parsing the whole reply each time. Our lab stops at "text parsed"; drawing it is extra time on top.

Tokens that are not characters. A token can end in the middle of a multi-byte character or a word. Decode bytes with a streaming decoder (TextDecoderStream or TextDecoder with stream: true) so a split character is not shown as garbage.

Trade-offs

Common mistakes

In an interview

1. Clarify: who reads the reply (a person in a chat, or another program)? How long are replies? Is there a check that needs the whole text?
2. Assumptions: state a first-token time and a per-token time, for example 500 ms and 30 ms, and a reply length, for example 300 tokens.
3. Structure: client, edge or CDN, reverse proxy, API server, model server; mark which hops must not buffer.
4. Deep dive: server generation time ≈ TTFT + (n - 1) × per-token time, with network time measured separately at the client; buffered shows text at the end of that sum, streamed at the start. SSE over one HTTP response, events with data: lines, client reads with a stream reader.
5. Trade-offs: per-event overhead vs batching delay ((k - 1) × d), errors after the first byte, what a hold-back window can and cannot check, cancellation, backpressure from slow clients, KV cache capacity vs concurrency.
6. Wrap up: metrics to watch (server TTFT p95, client first parsed text p95, inter-event gaps, per-token time, stream completion rate) and the buffering checklist.

Practice

Cheat sheet

Further reading

• MDN, Using server-sent events: practical guide to the event stream format and the EventSource API, with examples.
• WHATWG HTML, Server-sent events: the normative definition of the format, parsing rules, reconnection and Last-Event-ID.
• RFC 9112, section 7.1: how HTTP/1.1 chunked transfer coding lets a body be sent in pieces.
• Nginx proxy module, proxy_buffering: the default buffering behaviour and how to turn it off per location or per response.
• Hugging Face, How to generate text: the token-by-token generation loop and decoding strategies (greedy, beam search, sampling).
• Hugging Face Transformers, Caching: how the KV cache is stored and reused across decode steps, and the cache classes generate() uses.
• Kwon et al. (2023), PagedAttention: block-based KV cache allocation in vLLM, and why cache memory limits how many sequences a server can batch.
• Hugging Face Transformers, Generation strategies: decoding options including speculative (assisted) decoding, where one verification pass can accept several tokens.
• Ainslie et al. (2023), GQA: grouped-query attention, which shrinks the KV cache by sharing key/value heads across query heads.
• Node.js http, response.write(): the return value, the drain event and why ignoring them queues output in process memory.
• Sennrich et al. (2016): the subword (byte-pair encoding) paper behind why tokens are often parts of words.
• Nielsen Norman Group, Response Times: the 0.1 s, 1 s and 10 s limits used when reasoning about perceived waiting.

Next in the series: My chat got long. The AI forgot my name. Why? (AI 3), about the context window and what an app resends every turn. For the queueing side of latency, see why a server is fast alone and slow when crowded.

This article describes general patterns for streaming generated text, illustrated with our own small simulation. The numbers come from a simulated generator in one Node.js process (no real model, no network, no rendering) and are not measurements of any company's product or private system.