How should a voice agent handle barge-in when a customer talks over its spoken reply?

## Why turn-taking is the hard part of voice Text chat has an explicit floor: the user presses send. Voice has none. The system must continuously decide two things — has the user finished speaking, and is the user trying to take the floor while I am speaking. Get either wrong and the product feels broken in a way no amount of answer quality repairs. In a noisy drive-thru or a multilingual ordering lane, this is the dominant source of complaints. ## The three mechanisms of barge-in ### 1. Fast cancellation When incoming speech is detected during playback, output must stop within a couple of hundred milliseconds. That means cancelling at every layer: the audio device buffer, any queued synthesized chunks, the synthesis request itself, and the language-model generation still streaming. A common bug is stopping the speaker but letting buffered audio drain, so the assistant keeps talking for a second after the customer interrupts — which reads as rudeness. ### 2. Always-open capture The recognizer must be running during playback, not started when playback ends. If capture begins after cancellation, the first 200–400 ms of the interruption — often the whole word that carries the intent, like "no" or "wait" — is gone. Always-open capture is what makes the difference between "it stopped" and "it heard me". This only works with **acoustic echo cancellation**. Without it, the microphone picks up the agent's own voice from the speaker and the system barges in on itself, producing an agent that cannot finish a sentence. In a car lane or on a speakerphone this is the default condition, not an edge case. ### 3. Context repair — the part most teams miss Suppose the agent generated "Your total is fourteen fifty, and would you like to add a drink to that?" and was cut off after "fourteen fifty". The customer never heard the drink offer. If you append the full generated text to the conversation history, the model now believes it made an offer, and its next turn may say "as I mentioned, the drink" — a hallucination from the user's point of view. The fix is to track how much audio actually played, map that back to the corresponding text, and store *that* as the assistant turn, typically with a marker that it was interrupted. Realtime provider stacks expose the played duration for this reason; in a cascaded stack you track it yourself. ## Not every sound is an interruption Humans use back-channels — "mm-hm", "yeah", "right" — to signal listening, not to take the floor. An agent that halts on every one of them is exhausting. Practical policy: require a minimum duration and energy, ignore known short affirmatives, and optionally let a small classifier decide whether the incoming speech is a floor grab or acknowledgement. In genuinely noisy environments, raise the threshold rather than disabling barge-in. ## The mirror problem: deciding the user has finished The other half of turn-taking is endpointing. The old approach is acoustic voice-activity detection with a fixed silence timeout — after N milliseconds of silence, assume the turn ended. It cannot win: a short timeout cuts off a customer who pauses to check a menu or to think mid-order, and a long one makes every exchange feel sluggish. Practice as of mid-2026 has moved to **semantic (model-based) turn detection**: a small model looks at the audio and the partial transcript and predicts whether the utterance is complete. "I'll have a large coffee and…" is syntactically and prosodically unfinished, so the agent waits even through a long pause; "that's everything, thanks" is complete, so it can respond after a very short one. This is an adaptive timeout rather than a fixed one, and it is the single change that most improves how natural a voice agent feels. It is not free — it adds a model call in the hot path, and it degrades when the transcript is noisy or the language is one the detector handles poorly, so keep a generous fixed timeout as a backstop. ## What to measure - **Barge-in stop latency**: speech onset to silence at the speaker. - **False barge-in rate**: interruptions triggered by echo, noise, or back-channels. - **Missed-onset rate**: interruptions where the first words were not captured. - **False endpoint rate**: turns cut off mid-thought, and its opposite, **endpoint delay**. - **Context-repair correctness**: does the stored assistant turn match what was actually heard? Tune stop latency and endpointing thresholds per deployment; the right values in a quiet headset call and in a drive-thru lane are not the same.