I interrupted the AI. Why did it keep talking? Voice AI barge-in explained

Voice AI barge-in detects an interruption during playback and stops the reply. A voice assistant can keep talking when speech detection or playback cancellation is missing, delayed or disabled. In our simulation the reply kept going 4,075 ms on average after you spoke; a 12 dB energy detector with stops at 100 ms chunk boundaries cut that to 155 ms, and it also stopped on 19 of 20 synthetic coughs.

Prerequisites

A voice reply may come from text-to-speech or direct audio generation. It often arrives in pieces, the same streaming idea as in why an AI starts answering before it is done, and the player keeps a small queue of audio and plays it in order.

Stopping is a separate job. Something has to watch the microphone while the speaker plays, decide that the sound is you, and tell the player to stop. That decision is an event and the player reacts to it, the same publish and react shape as in publish/subscribe with one slow service.

Definitions

How does voice AI barge-in detect speech?

The mic signal is cut into frames, 20 ms each in our lab (320 samples at 16 kHz). For each frame the detector computes 10 × log10(frame mean-square / calibrated noise mean-square), which gives how many decibels the frame sits above the quiet room. It triggers after three consecutive 20 ms frames exceed 12 dB. That rejects a single high-energy frame, though three loud frames in a row are still not proof of continuous speech.

We kept this rule crude on purpose. The WebRTC VAD source models noise and speech with two Gaussians in each of six frequency bands and updates them as it runs, so it looks at how energy spreads across frequencies instead of one total. Silero VAD uses a trained neural network; its README says one chunk of 30 ms or more takes under 1 ms on one CPU thread. Cloud voice APIs expose similar knobs as settings: the OpenAI Realtime docs list a speech threshold, prefix_padding_ms, silence_duration_ms and an interrupt_response flag.

What happens to the audio already queued?

Once the detector fires, the player has to stop what is playing and throw away everything not yet played. VoiceXML 2.0 wrote this rule for phone menus in 2004: when barge-in happens during a sequence of prompts, "all subsequent prompts are not played".

The W3C VoiceXML 2.0 Recommendation. Section 4.1.5 defines barge-in and lets authors turn it off for prompts users must hear in full.

Real players can stop inside a piece of audio. In the Web Audio API, stop() on an audio source ends it at a given time, including partway through a buffer. Our simulation allows stopping only at 100 ms chunk boundaries. Actual cancellation latency depends on the player and output buffers. In the lab, talk-over is the detector's delay plus the rest of the current chunk.

The model side needs cleanup too, and the transport decides who does it. With Realtime over WebSocket, the client stops playback, discards queued audio and sends conversation.item.truncate with the interrupted item's played duration as audio_end_ms. With WebRTC and SIP, the server manages output buffering and automatically truncates unplayed audio.

What we saw when we ran it

We ran node labs/ai-15/run.mjs on 2026-11-06. The script generates 60 synthetic microphone signals: 20 speech-like signals, 20 cough-like bursts and 20 clatter-like bursts, each starting at a random moment between 1.5 s and 6.5 s into an 8 s reply. It runs four setups on the same signals. The signals only test this energy rule and tell us nothing about recorded speech or coughs.

Our terminal after the lab run: talk-over, detector delay and false stops for each setup.

Without barge-in handling the reply played to its end: 4,075 ms of talk-over on average and 5,809 ms at p90, set only by how much reply was left. With the 12 dB detector, talk-over fell to 155 ms and the detector caught all 20 speech runs. It also stopped on 19 of 20 coughs, because a cough is short and loud and this detector sees only energy. At 6 dB it reacted fastest, in 81 ms, and stopped on all 20 coughs.

Worked example

One run from the lab: you start talking 2,587 ms into the reply.

reply: 0 to 8,000 ms, you start at 2,587 ms
no barge-in         plays to 8,000 ms           talk-over 8,000 - 2,587 = 5,413 ms
12 dB detector      fires at 2,740 ms (153 ms)  stop at the next chunk edge, 2,800 ms
                                                talk-over 2,800 - 2,587 = 213 ms
averages over 20 speech runs (12 dB): delay 123 ms + rest of chunk = talk-over 155 ms
detection is aligned to 20 ms frames, so the rest of a 100 ms chunk is 0, 20, 40, 60 or 80 ms;
equally represented detection phases would leave 40 ms on average; these 20 runs left 32 ms

Code

The detector and the stop rule from labs/ai-15/run.mjs (Node 22, standard library only):

// labs/ai-15/run.mjs (trimmed)
const FRAME = 320, FRAME_MS = 20, CALIB_FRAMES = 25, MIN_FRAMES = 3, CHUNK_MS = 100;

function frameDbOverFloor(buf) {               // energy of each 20 ms frame, in dB over the quiet room
  const n = Math.floor(buf.length / FRAME), ms = new Float64Array(n);
  for (let f = 0; f < n; f++) { let e = 0; for (let i = f * FRAME; i < (f + 1) * FRAME; i++) e += buf[i] * buf[i]; ms[f] = e / FRAME; }
  let floor = 0; for (let f = 0; f < CALIB_FRAMES; f++) floor += ms[f]; floor /= CALIB_FRAMES;   // first 500 ms
  return Array.from(ms, (e) => 10 * Math.log10(e / floor));
}
function detect(db, thr) {                     // first moment with 3 frames in a row above the threshold
  let run = 0;
  for (let f = CALIB_FRAMES; f < 8000 / FRAME_MS; f++) {
    run = db[f] > thr ? run + 1 : 0;
    if (run >= MIN_FRAMES) return (f + 1) * FRAME_MS;
  }
  return null;
}
// the queue is cleared, the chunk at the speaker plays out
const streamingStop = (detectMs) => Math.min(8000, Math.ceil(detectMs / CHUNK_MS) * CHUNK_MS);

Run node run.mjs in labs/ai-15. It prints the table, a text timeline of one run (one character per 50 ms) and writes timeline.json. Change THRESH or CHUNK_MS to move the trade-off.

Failure modes and edge cases

• The mic hears the assistant's own voice. Uncancelled playback can trigger a false stop when its microphone energy exceeds the threshold for three consecutive frames. Our lab assumes perfect cancellation, and leftover echo in real rooms causes false stops.
• Coughs, laughs and doors pass an energy rule. A minimum speech length helps, and so does waiting for a recognised word before a full stop.
• "Mm-hm" is speech, yet the listener did not mean "stop". Turn-taking logic that looks at words and pauses has to decide that.
• Network and output buffering can add latency; this simulation does not predict a product's total talk-over.
• Stopping playback does not stop generation. Unless the request is cancelled too, the model may keep producing a reply nobody hears. We did not measure this.

Trade-offs

Threshold	Talk-over (mean)	Coughs that stopped it	Clatter that stopped it	Observed in this simulation
6 dB	125 ms	20 of 20	19 of 20	Fastest reaction (81 ms delay), stopped on almost every non-speech sound
12 dB	155 ms	19 of 20	6 of 20	Caught all 20 speech runs, still stopped on most coughs
18 dB	220 ms	3 of 20	0 of 20	Fewest false stops, 65 ms more talk-over than 12 dB

The threshold trades reaction time against false stops. Chunk size is the other dial: smaller chunks cut the leftover in our simulation but mean more scheduling work and more audio boundaries.

Common mistakes

Mistake	Do this instead
Failing to implement playback cancellation	Stop the active source and discard queued audio
Stopping playback but not the request	Track the interrupted response and played audio duration. Cancel generation, discard subsequent audio for that response, and truncate unplayed conversation content using the API's supported mechanism
Tuning the threshold in a silent office	Test with coughs, clatter, music and echo, like our 20 cough and 20 clatter runs
Treating every loud sound as "stop"	Require a minimum speech length or a short confirmation window before a hard stop

Our take

Barge-in needs detection and cancellation to both work. Detection decides how soon the system knows you spoke, and cancellation decides how much audio still reaches you after that. In our simulation an energy rule was fast enough, with 123 ms of delay at 12 dB, and its weak point was telling speech from a cough. We would spend tuning effort on that classification and on clean cancellation before shaving more milliseconds off detection. For how the trimmed reply then feeds the next turn, see why an AI agent re-sends its history.

Our lab is a small simulation with synthetic sounds and a crude energy detector, echo assumed cancelled, stops only at chunk boundaries, and network and hardware delay left out. Its numbers describe this simulation only and are not measurements of any real assistant or product.