For a tool-heavy voice agent, would you use a cascaded STT-LLM-TTS stack or speech-to-speech?

## The two architectures **Cascaded**: audio → speech recognition → text → language model (with tool calls) → text → speech synthesis → audio. Three or more independent components, each with its own vendor, price and failure mode. **Speech-to-speech (end-to-end)**: audio in, audio out, one model that never materializes a full text intermediate. Turn-taking, interruption and prosody are handled inside the model rather than by your orchestration code. ## What the cascade actually buys **An inspectable artifact at every hop.** The transcript is the single most valuable thing in a regulated voice product: it is what you log, redact, retain, hand to compliance, replay in a dispute and mine for eval sets. Speech-to-speech does not give you one for free. **Independent evolution.** Recognizers improve on a different cadence than reasoning models. In a cascade you can adopt a better recognizer on Monday and a better reasoning model on Friday, and A/B each in isolation. End-to-end couples all three to one vendor's release schedule. **Reuse of the text stack.** Tool definitions, retrieval, prompt templates, guardrails, judge-based evals, prompt caching — everything built for the text product applies unchanged. For an agent whose value is in calling twelve internal tools correctly, that reuse is most of the system. **Precise control.** You can rewrite the user's text before it reaches the model, refuse on a classifier, force a deterministic response for known intents, and pin exactly what gets spoken. ## What the cascade costs **Serial latency.** Recognition must finalize, the model must generate at least a first sentence, synthesis must start. Every hop adds queueing and network time, and the floor is noticeably above human conversational turnaround. Mitigations — streaming recognition, generating and synthesizing sentence-by-sentence, speculative starts on likely intents — narrow the gap but do not close it. **Lost paralinguistics.** Text is a lossy projection of speech. Sarcasm, hesitation, distress, urgency, emphasis on "I said *no* onions" — all gone before the model sees anything. For an empathy-sensitive or emotionally loaded interaction that loss is the product. **Flat output.** Synthesis re-invents prosody from text alone, with no memory of how the user sounded, so replies can be well-worded but tonally mismatched. **Error compounding.** A recognition error becomes the model's premise; the model cannot hear that the word was ambiguous. ## What speech-to-speech buys and costs Buys: sub-second response, natural interruption handling, prosody that carries through the reasoning, and a much simpler client loop. Costs: weaker observability (you must run a parallel transcription pass to get logs, which reintroduces cost and drift), less precise steering, immature tool-calling relative to the text stacks, single-vendor coupling, and evaluation methods that are far less mature than text evals. ## How to decide Ask four questions. 1. **How many tools, and how consequential?** A dozen tools that move money push hard toward cascaded, where you can validate arguments against a text trace. 2. **What is the audit obligation?** If a regulator, a dispute process or an internal review needs a record of what was said and why, you need text at the decision point — which the cascade gives you natively. 3. **Does tone carry information?** Distress lines, sales calls and companionship products lose real signal to text. Order-taking and account lookups mostly do not. 4. **What is the latency bar?** Long-tail agentic work already spends seconds in tools, so a few hundred extra milliseconds are invisible. Rapid back-and-forth chat is where end-to-end feels categorically better. ## The hybrid, and where practice sits in mid-2026 The common compromise is to keep the cascade for the reasoning and tool path but adopt end-to-end characteristics at the edges: a realtime streaming recognizer, model-based turn detection, sentence-level synthesis so the first words play while the rest is still generating, and a separate lightweight audio classifier that extracts sentiment or urgency and passes it to the text model as a tag — recovering some paralinguistic signal without giving up the transcript. Another pattern routes by intent: end-to-end for chit-chat and acknowledgements, cascaded whenever a tool must be called. As of mid-2026, cascaded remains the production default for tool-heavy agents while end-to-end realtime models have matured considerably; the gap is narrowing, and the decision is a judgment call about observability and control versus naturalness, not a settled best practice. State that plainly in an interview rather than claiming one architecture has won.