Speech-to-Speech Voice AI: Why 2026 Voicebots Are Dropping the Transcript

Speech-to-Speech Voice AI: Why 2026 Voicebots Are Dropping the Transcript

Short answer: Speech-to-speech voice AI processes audio directly into audio, replacing the three-step speech-to-text → language model → text-to-speech pipeline. It responds faster, handles accents and interruptions better, and conveys tone — but it removes the text layer that teams rely on for exact disclosures, logging, and pronunciation control.

Key takeaways

• The classic voice agent stack has three hops: speech recognition, a language model, then speech synthesis. Each hop adds delay and discards information.

• Speech-to-speech (also called native audio) models collapse those hops into one, so tone, hesitation and accent reach the model instead of being flattened into text.

• The cost is determinism. Compliance lines, reference-code readbacks and currency pronunciation are easy to guarantee when you control the text going into synthesis, and hard when you don't.

• Most production teams end up hybrid: speech-to-speech for conversation, a text path for anything that must be word-exact, and parallel transcription for logs and evals.

What is speech-to-speech voice AI?

Speech-to-speech voice AI is a single model that takes spoken audio as input and produces spoken audio as output, without converting the conversation into text in between. It is also marketed as native audio or realtime voice. OpenAI's Realtime API and Google's Gemini Live API are the two most widely used commercial examples.

The architecture it replaces is the cascade: a speech-to-text (STT) engine transcribes the caller, a large language model reads that transcript and writes a reply, and a text-to-speech (TTS) engine speaks the reply aloud. Nearly every voice agent deployed between 2023 and 2025 was built this way.

Why is the pipeline being replaced now?

Two reasons, and only one of them is latency.

The three-hop delay is structural, not a tuning problem. Speech recognition has to decide the caller has genuinely stopped speaking before it commits a final transcript. The language model has to generate at least a first clause. Synthesis needs enough text to produce natural prosody. Even a well-tuned cascade on good infrastructure typically lands in the high hundreds of milliseconds before the caller hears anything — before you add a CRM lookup.

That matters because researchers studying turn-taking in natural conversation consistently find the gap between speakers sits in the low hundreds of milliseconds, and stays roughly constant across very different languages. Callers do not consciously register "this is slow." They talk over the agent, repeat themselves, and treat it like the IVR they already dislike.

A transcript is a lossy summary of speech. The words survive. Almost nothing else does: hesitation, sarcasm, rising frustration, the difference between an enthusiastic "yeah" and a reluctant one, the accent that would have told the agent which language to switch into. In a cascade, the model reasoning about your caller has never actually heard them.

Speech-to-speech vs the STT-LLM-TTS cascade

What do you actually gain?

Turn-taking that feels human. When one model owns the whole loop, barge-in, backchannels like "mm-hmm" while the caller is still explaining, and recovery when both parties speak at once become behaviours of the model rather than heuristics layered on top.

Paralinguistics in both directions. The model hears tone, so it can respond to tone — and produce it. Emphasis lands in the right place. An apology sounds apologetic instead of a synthesis engine reading the word "sorry."

Resilience to messy input. Heavy accents, background noise, mumbled street names and mid-sentence language switching all survive much better in a model that keeps the audio. For markets where callers routinely mix two or three languages in one sentence, this alone can decide whether an agent is usable.

What do you give up?

This is the part the demos skip.

Determinism. Recording disclosure, AI disclosure and do-not-call confirmations need to be said reliably, in an approved form. A cascade lets you inject exact text into TTS and know precisely what the caller heard. A speech-to-speech model improvises — which is exactly what makes it sound good, and exactly what makes a compliance team nervous.

Auditability. No transcript means no free log. In a dispute, "we have the audio" is a much weaker position than "we have the audio plus a synchronised, searchable transcript." Which is why most teams run recognition in parallel anyway — the transcript returns as a side effect rather than the substrate.

Pronunciation control. The rules that make spoken data intelligible — a reference code read digit by digit, an amount read as a natural quantity — are trivial to enforce when you own the text going into synthesis. Without it, they become prompt engineering and hope.

Instruction adherence. Long structured prompts and hard "never do X" rules generally hold better in a text-native path today. If your voice agent is really a state machine wearing a persona, that state machine is easier to enforce with text in the loop.

How do you migrate without breaking compliance?

The shape that keeps recurring in production is hybrid, not either-or.

1. Route the conversational surface to speech-to-speech — greeting, discovery, objection handling, the parts where naturalness converts.

2. Keep a text path for anything word-exact — disclosures, confirmations, and any script legal signed off on verbatim.

3. Run parallel transcription continuously, feeding logs, evals and QA rather than driving the conversation.

4. Set an explicit latency budget for tool calls, with a natural holding line the agent can say while a backend query resolves.

5. Build a code and currency pronunciation test set before you migrate, so you can measure regressions instead of discovering them on live calls.

6. Pilot one low-risk flow first and compare against your existing stack on the same call types, same hours, same caller mix.

Our data: [Insert your own measured median and p95 response latency, cascade vs speech-to-speech, on a matched set of live calls. Original numbers from your own deployment are the single most citable thing on this page.]

What should you do this quarter?

You do not need to rewrite anything yet. You do need your own numbers.

Measure real end-to-end response latency on live calls — median and tail, not a test harness. Pull twenty recordings where the caller talked over the agent and find out why. Take your ten most compliance-sensitive lines and ask honestly whether an improvised version is acceptable. Then run one narrow pilot and compare.

The cascade is not dead this quarter. But the assumption that a transcript belongs in the middle of every phone conversation is going — and the teams that notice early will already have the pronunciation lexicons, eval sets and disclosure patterns ready.

Frequently asked questions

Is speech-to-speech voice AI cheaper than the STT-LLM-TTS pipeline? Usually not. Per-minute audio pricing on realtime models tends to run higher than the combined cost of three cheaper components, and you often still pay for parallel transcription for logging. The gain is conversational quality and latency, not unit cost.

Does speech-to-speech work for regulated industries? Yes, but rarely on its own. Sectors with scripted disclosure requirements — finance, healthcare, insurance, collections — typically keep a text path for the exact lines and use native audio for the surrounding conversation.

Can a speech-to-speech agent still call APIs and book appointments? Yes. Realtime audio models support tool calling, so the agent can query a CRM, check availability and write a booking. Budget for the latency the call adds, and give the agent a natural holding phrase to cover it.

How do you log calls without a transcript? Run a speech-to-text engine in parallel with the live conversation, purely for logging, analytics, quality assurance and evals. The transcript becomes a downstream artefact instead of a dependency in the response path.

Do you still need to disclose that the caller is speaking to an AI? Yes. Disclosure obligations attach to the interaction, not the architecture, and several jurisdictions require a truthful answer when a caller asks whether they are speaking to a machine. Never prompt an agent to claim it is human. Confirm current requirements for your markets with counsel.