How to build a voice AI interface: STT, LLM and TTS

Most voice assistants people have used are good at one thing (“set a timer for ten minutes”) and fall apart on anything with two clauses. The parts needed to do better are now off the shelf: streaming speech-to-text, language models that hold a multi-turn conversation, and text-to-speech that starts talking almost as soon as it gets a sentence. What is still hard is wiring them together so the result feels like a conversation and not a phone tree. This guide covers that wiring, with TypeScript you can adapt, and then the design patterns that decide whether anyone wants to use it.

There is another route worth knowing about: speech-to-speech models that take audio in and give audio out with no text step in the middle. They cut out two hops, but you give up control over each stage. Our look at GPT-Live-1 and the OpenAI Realtime API covers that option. Everything below is the cascaded pipeline, which is still the one you can inspect, test and swap piece by piece.

A voice interface is a speech-to-text, LLM and text-to-speech pipeline with streaming at every step

How a voice AI interface works

A voice AI interface is a pipeline of three streaming stages. Speech-to-text turns the user’s audio into words, a language model decides what to say, and text-to-speech turns the reply back into audio. The difficult parts are latency and turn-taking, so every stage has to stream its output instead of waiting for the stage before it to finish.

┌────────────────────────────────────────────────────┐
│                  Client (Browser/App)              │
│                                                    │
│  ┌───────────┐   WebSocket    ┌──────────────────┐ │
│  │ Mic Input │──────────────► │ Audio Streamer   │ │
│  │ (MediaAPI)│                │ (sends chunks)   │ │
│  └───────────┘                └────────┬─────────┘ │
│                                        │           │
│  ┌───────────┐   WebSocket    ┌────────┴─────────┐ │
│  │ Speaker   │◄────────────── │ Audio Player     │ │
│  │ Output    │                │ (plays chunks)   │ │
│  └───────────┘                └──────────────────┘ │
└────────────────────┬───────────────────────────────┘
                     │ WebSocket (bidirectional)
                     │
┌────────────────────▼───────────────────────────────┐
│                Voice AI Server                     │
│                                                    │
│  ┌─────────────┐   ┌─────────────┐   ┌───────────┐ │
│  │ STT Engine  │──►│ LLM Engine  │──►│ TTS Engine│ │
│  │ (streaming) │   │ (streaming) │   │(streaming)│ │
│  │ Deepgram    │   │ Claude      │   │ ElevenLabs│ │
│  └─────────────┘   └──────┬──────┘   └───────────┘ │
│                           │                        │
│  ┌────────────────────────▼─────────────────────┐  │
│  │           Conversation Manager               │  │
│  │  - Session state and history                 │  │
│  │  - Turn-taking                               │  │
│  │  - Interrupt handling                        │  │
│  └──────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────┘

Audio chunks flow from the microphone to STT as they are recorded. Final transcript segments accumulate until the STT service signals the end of the user’s turn. The LLM reply streams token by token, gets cut into sentences, and each sentence goes to TTS while the model is still writing the next one. TTS audio flows back to the client as it is synthesized.

Set a latency budget before you write code

People answer each other fast. Stivers and colleagues measured turn-taking in 10 languages for a 2009 PNAS study and found a median gap of about 100ms between one speaker finishing and the next starting, with a mean of 208ms. A cascaded pipeline will not beat that. Your job is to keep the gap short enough that it reads as “thinking” and not “broken”, and that means knowing where the time goes.

Stage	What sets the delay	How to cut it
End-of-turn detection	How long you wait in silence before deciding the user is done	Use an STT model with built-in turn detection, or tune the silence threshold
Speech-to-text	Streaming versus batch transcription	Always stream; act on final segments
LLM first token	Model size, prompt length, thinking	Small fast model, low effort, thinking off, short system prompt
TTS first audio	Model choice, text chunk size	Low-latency TTS model; start on the first sentence
Network	Distance between your server and each provider	Run the server in the same region as your providers

The row that surprises people is the first one. Deepgram’s documentation on end of speech detection advises setting utterance_end_ms to 1000 or higher, because interim results arrive about once a second. So if you rely on that signal alone, you have spent about a second before the LLM even sees the question. That is why Deepgram now offers Flux, which its models overview describes as having “model-integrated end-of-turn detection” for voice agents. I would test Flux, or a framework with semantic turn detection, before tuning anything else.

On the output side, ElevenLabs quotes about 75ms for eleven_flash_v2_5 on its models page, with the caveat that this excludes application and network latency. Treat vendor numbers like that as a floor, then measure your own pipeline end to end.

Step 1: stream speech-to-text

The server accepts a WebSocket from the browser, opens a streaming connection to Deepgram, and forwards audio. This uses version 5 of the Deepgram JavaScript SDK, which replaced createClient and listen.live with new DeepgramClient() and listen.v1.connect. The v5 README passes query options as strings, so the code does the same.

import { WebSocketServer, WebSocket } from "ws";
import { DeepgramClient } from "@deepgram/sdk";
import Anthropic from "@anthropic-ai/sdk";

const deepgram = new DeepgramClient(); // reads DEEPGRAM_API_KEY
const anthropic = new Anthropic(); // reads ANTHROPIC_API_KEY

type Turn = { role: "user" | "assistant"; content: string };

interface Session {
  ws: WebSocket;
  history: Turn[];
  pending: string; // final transcript segments for the current user turn
  speaking: boolean; // assistant audio is being generated or sent
  abort: AbortController | null;
}

const wss = new WebSocketServer({ port: 8080 });

wss.on("connection", async (ws: WebSocket) => {
  const session: Session = {
    ws,
    history: [],
    pending: "",
    speaking: false,
    abort: null,
  };

  const stt = await deepgram.listen.v1.connect({
    model: "nova-3",
    language: "en",
    smart_format: "true",
    interim_results: "true", // required for utterance_end_ms
    utterance_end_ms: "1000", // Deepgram advises 1000 or higher
  });

  stt.on("message", (msg) => {
    if (msg.type === "Results") {
      const text = msg.channel.alternatives[0]?.transcript ?? "";
      // The user started talking over the assistant: stop the reply
      if (text && session.speaking) interrupt(session);
      if (text && msg.is_final) session.pending += " " + text;
    }

    if (msg.type === "UtteranceEnd") {
      const utterance = session.pending.trim();
      session.pending = "";
      if (utterance) void respond(session, utterance);
    }
  });

  stt.connect();
  await stt.waitForOpen();

  // Browser sends WebM/Opus chunks from MediaRecorder
  ws.on("message", (audio: Buffer) => stt.sendMedia(audio));

  ws.on("close", () => {
    interrupt(session);
    stt.close();
  });
});

Two details matter here. The interrupt check runs on any non-empty interim result, so the user can cut in after a word or two. And UtteranceEnd arrives as its own message type with type: "UtteranceEnd", which is the trigger to hand the turn to the model. If you switch to Flux, the connection moves to listen.v2.connect and the end-of-turn event comes from the model instead.

Step 2: stream the LLM reply in sentences

The earlier version of this post used claude-sonnet-4-20250514. Anthropic retired that model on June 15, 2026, so requests to it now fail. For voice, the models overview lists Claude Haiku 5.5 as the fastest current model, and the effort docs recommend low effort for chat. Haiku 5.5 thinks by default, and thinking costs time before the first word, so this request turns it off.

const VOICE_SYSTEM_PROMPT = `You are a voice assistant on a phone call.
Answer in 1 to 3 short sentences. No lists, no markdown, no emoji.
Use contractions and plain words. If the answer is long, give the
headline and ask whether the caller wants the details.`;

async function respond(session: Session, utterance: string): Promise<void> {
  session.history.push({ role: "user", content: utterance });
  send(session, { type: "transcript", text: utterance });

  const abort = new AbortController();
  session.abort = abort;
  session.speaking = true;

  const stream = anthropic.messages.stream({
    model: "claude-haiku-5-5",
    max_tokens: 300,
    output_config: { effort: "low" },
    thinking: { type: "disabled" },
    system: VOICE_SYSTEM_PROMPT,
    messages: session.history,
  });
  abort.signal.addEventListener("abort", () => stream.abort());

  let reply = "";
  let sentence = "";

  try {
    for await (const event of stream) {
      if (
        event.type === "content_block_delta" &&
        event.delta.type === "text_delta"
      ) {
        reply += event.delta.text;
        sentence += event.delta.text;

        // Hand complete sentences to TTS while the model keeps writing
        if (/[.!?]\s/.test(sentence) || sentence.length > 150) {
          await speak(session, sentence.trim(), abort.signal);
          sentence = "";
        }
      }
    }
    if (sentence.trim()) await speak(session, sentence.trim(), abort.signal);
  } catch (err) {
    if (!abort.signal.aborted) throw err;
  } finally {
    session.speaking = false;
    if (reply) {
      const cut = abort.signal.aborted ? " [interrupted by the user]" : "";
      session.history.push({ role: "assistant", content: reply + cut });
    }
  }
}

function send(session: Session, message: object): void {
  if (session.ws.readyState === WebSocket.OPEN) {
    session.ws.send(JSON.stringify(message));
  }
}

The max_tokens: 300 cap is a guardrail, not the main control. The system prompt does most of the work of keeping replies short, and short replies matter more in voice than anywhere else (more on that below). Marking interrupted replies in the history is a small thing that helps: the model then knows the user did not hear the end of its last answer.

Step 3: stream text-to-speech back to the client

Each sentence goes to the ElevenLabs streaming endpoint. Two fixes from the original code: output_format is a query parameter, not a body field, according to the API reference, and eleven_turbo_v2_5 is now in the deprecated list, with ElevenLabs recommending Flash models “in all use cases”.

async function speak(
  session: Session,
  text: string,
  signal: AbortSignal
): Promise<void> {
  const voiceId = process.env.ELEVENLABS_VOICE_ID!;
  const res = await fetch(
    `https://api.elevenlabs.io/v1/text-to-speech/${voiceId}/stream?output_format=pcm_16000`,
    {
      method: "POST",
      headers: {
        "xi-api-key": process.env.ELEVENLABS_API_KEY!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ text, model_id: "eleven_flash_v2_5" }),
      signal, // aborting cancels the TTS request mid-stream
    }
  );
  if (!res.ok || !res.body) throw new Error(`TTS failed: ${res.status}`);

  const reader = res.body.getReader();
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    if (session.ws.readyState === WebSocket.OPEN) session.ws.send(value);
  }
}

function interrupt(session: Session): void {
  session.abort?.abort(); // stops the LLM stream and any TTS request
  session.abort = null;
  session.speaking = false;
  send(session, { type: "interrupt" }); // tells the client to flush audio
}

This version waits for one sentence to finish streaming before it requests the next. It is simple and the gaps between sentences are usually small, but if you hear them, run TTS requests for the next sentence in parallel and queue the audio in order.

Step 4: play audio and handle interruptions in the browser

pcm_16000 is raw 16-bit little-endian samples with no header, so the browser cannot use decodeAudioData on it. The player below converts samples to floats, schedules each chunk right after the previous one, and drops everything when the server says the user interrupted. Network chunks can split a 16-bit sample in half, which is why it keeps a leftover byte.

class VoicePlayer {
  private ctx = new AudioContext(); // create after a user click
  private playHead = 0;
  private sources = new Set<AudioBufferSourceNode>();
  private leftover: Uint8Array | null = null;

  handleMessage(event: MessageEvent) {
    if (typeof event.data === "string") {
      const msg = JSON.parse(event.data);
      if (msg.type === "interrupt") this.stop();
      return;
    }
    this.enqueue(new Uint8Array(event.data as ArrayBuffer));
  }

  private enqueue(chunk: Uint8Array) {
    let bytes = chunk;
    if (this.leftover) {
      bytes = new Uint8Array(this.leftover.length + chunk.length);
      bytes.set(this.leftover);
      bytes.set(chunk, this.leftover.length);
      this.leftover = null;
    }
    if (bytes.length % 2 !== 0) {
      this.leftover = bytes.slice(-1);
      bytes = bytes.slice(0, -1);
    }
    if (bytes.length === 0) return;

    const samples = new Int16Array(bytes.buffer, bytes.byteOffset, bytes.length / 2);
    const buffer = this.ctx.createBuffer(1, samples.length, 16000);
    const channel = buffer.getChannelData(0);
    for (let i = 0; i < samples.length; i++) channel[i] = samples[i] / 32768;

    const source = this.ctx.createBufferSource();
    source.buffer = buffer;
    source.connect(this.ctx.destination);
    const startAt = Math.max(this.ctx.currentTime, this.playHead);
    source.start(startAt);
    this.playHead = startAt + buffer.duration;
    this.sources.add(source);
    source.onended = () => this.sources.delete(source);
  }

  stop() {
    for (const source of this.sources) source.stop();
    this.sources.clear();
    this.playHead = 0;
    this.leftover = null;
  }
}

// Wiring: microphone up, audio down
const ws = new WebSocket("wss://your-server.example.com/voice");
ws.binaryType = "arraybuffer";
const player = new VoicePlayer();
ws.onmessage = (e) => player.handleMessage(e);

const mic = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: true },
});
const recorder = new MediaRecorder(mic, { mimeType: "audio/webm;codecs=opus" });
recorder.ondataavailable = (e) => {
  if (e.data.size > 0 && ws.readyState === WebSocket.OPEN) ws.send(e.data);
};
recorder.start(250); // send a chunk every 250ms

Do not skip echoCancellation. Without it the microphone hears the assistant through the speakers, STT transcribes the assistant’s own voice, and the server interrupts itself. Also check MediaRecorder.isTypeSupported before assuming WebM, since browser support for recording formats differs.

Turn-taking without a built-in signal

If your STT service gives you raw voice activity events but no end-of-turn signal, you need a small state machine. The thresholds below are starting values to tune against real recordings, not settled numbers.

class TurnManager {
  private vadState: "silent" | "speaking" | "pausing" = "silent";
  private pauseStart = 0;
  private readonly PAUSE_THRESHOLD_MS = 700; // gap that ends the turn
  private readonly SHORT_PAUSE_MS = 300; // natural mid-sentence pause

  handleVADEvent(event: { isSpeech: boolean; timestamp: number }) {
    if (event.isSpeech && this.vadState !== "speaking") {
      this.vadState = "speaking";
    } else if (!event.isSpeech && this.vadState === "speaking") {
      this.vadState = "pausing";
      this.pauseStart = event.timestamp;
    }
  }

  isEndOfTurn(now: number): boolean {
    return this.vadState === "pausing" && now - this.pauseStart >= this.PAUSE_THRESHOLD_MS;
  }

  isThinkingPause(now: number): boolean {
    if (this.vadState !== "pausing") return false;
    const pause = now - this.pauseStart;
    return pause >= this.SHORT_PAUSE_MS && pause < this.PAUSE_THRESHOLD_MS;
  }
}

Silence alone is a blunt signal. “I’d like to book a table for… um…” has a long pause in the middle of an unfinished sentence. That is the case model-based turn detection is built for, and it is the main reason to reach for a framework once the prototype works.

Design it voice-first, not voice-added

Most voice interfaces that feel bad have fine technology underneath. The problem is that someone bolted a microphone onto a screen flow.

The voice-added version reads a menu aloud and walks the user through a form:

User: "I want to order a medium pepperoni pizza"
System: "I found 3 pizza options. Please select from the following:
         1. Classic Pepperoni - $12.99
         2. Double Pepperoni - $14.99
         3. Pepperoni Supreme - $16.99"
User: "...uh, the first one"
System: "What size? Small, Medium, or Large?"
User: "I already said medium!"

The voice-first version takes everything from the first sentence and asks only for what is missing:

User: "I want to order a medium pepperoni pizza"
System: "Got it. Medium pepperoni, classic at $12.99.
         Regular crust okay, or would you prefer thin?"
User: "Regular is fine"
System: "Medium pepperoni, regular crust, $12.99.
         Delivering to your usual address on Oak Street?"
User: "Yes"
System: "Order placed. Anything else?"

A few patterns follow from that.

Disclose progressively. Audio is serial, and listeners cannot skim it. Say “Your total is $37.75 for 4 items. Want me to read them back?” instead of reading every line item and the tax.

Confirm once per action. “I’ll book a table for 4 at Mario’s, Friday at 7, outdoor seating. Should I go ahead, or did I get anything wrong?” beats three separate “Is that correct?” prompts.

Recover without dead ends. “I didn’t understand, please try again” leaves the user stuck. “I caught ‘book appointment’ but missed the time. What time works for you?” keeps the part that worked.

Hand off to a screen when the content is visual. Long lists, comparison tables, addresses and confirmation codes are painful to hear. “I found 8 flights. I’ll send the top 3 to your screen. The cheapest leaves at 6 AM for $342. Want that one?” uses each channel for what it is good at.

Common mistakes

Treating voice like typed text is the most common one. People say “um”, repeat themselves and change direction mid-sentence, so a regular expression that expects clean input will fail constantly. Let the model do the extraction:

// Bad: expects clean input
function parseOrder(transcript: string) {
  const match = transcript.match(/order (\d+) of (.+) at (\$[\d.]+)/);
  if (!match) throw new Error("Could not parse order");
}

// Better: let the LLM pull entities out of messy speech
async function parseOrder(transcript: string) {
  return llm.extract({
    instructions: `Extract order details from this spoken input.
                   Ignore filler words, repetitions and corrections.
                   Return JSON: { items: [{ name, quantity, size }] }`,
    input: transcript,
  });
}

Long replies are the second. Text on a screen can be skimmed; speech has to be sat through. Put explicit limits in the system prompt (one sentence for simple answers, two or three for harder ones, offer details instead of volunteering them) and ban lists and markdown, which sound strange read aloud.

The third is testing only in a quiet room. Real users are in cars, kitchens and open offices. Build noise into the eval suite so you see how accuracy degrades:

const noiseScenarios = [
  { name: "quiet", snr: 40 }, // signal-to-noise ratio in dB
  { name: "office", snr: 20 },
  { name: "car", snr: 15 },
  { name: "street", snr: 10 },
  { name: "crowded", snr: 5 },
];

for (const scenario of noiseScenarios) {
  test(`handles ${scenario.name} audio (SNR ${scenario.snr}dB)`, async () => {
    const noisyAudio = addNoise(cleanAudio, scenario.snr);
    const transcript = await stt.transcribe(noisyAudio);
    const wer = computeWordErrorRate(transcript, groundTruth);
    // Pick thresholds from your own baseline, then hold the line
    expect(wer).toBeLessThan(scenario.snr > 15 ? 0.05 : 0.15);
  });
}

The fourth is ignoring the latency budget until the end. Log a timestamp at end of user speech, first LLM token, first TTS byte and first audio played. Without those four numbers you will guess wrong about where the delay is.

One backend for text and voice

If you already have a text chatbot, voice does not need a second brain. Put the conversation logic behind a channel adapter, and voice becomes STT on the way in and TTS on the way out.

┌──────────────┐    ┌───────────────┐    ┌──────────────┐
│  Web Chat    │    │ Voice (Phone) │    │ Voice (App)  │
│  (text)      │    │ (Twilio)      │    │ (WebSocket)  │
└──────┬───────┘    └──────┬────────┘    └──────┬───────┘
       ▼                   ▼                    ▼
┌──────────────────────────────────────────────────────┐
│              Channel Adapter Layer                   │
│  - Text passthrough for chat                         │
│  - STT for voice input, TTS for voice output         │
│  - Channel-specific formatting (shorter for voice)   │
└───────────────────────┬──────────────────────────────┘
                        ▼
┌──────────────────────────────────────────────────────┐
│           Unified Conversation Engine                │
│  - LLM response generation                           │
│  - Context and history                               │
│  - Tool execution (order lookup, booking, etc.)      │
│  - Safety guardrails                                 │
└──────────────────────────────────────────────────────┘

The one thing that must differ per channel is formatting. A reply that works in chat (a bulleted list with links) is wrong for voice, so the adapter or the system prompt has to change the shape of the answer, not only the medium. If you are still deciding whether to build that engine at all, our chatbot build-vs-buy cost breakdown goes through the tradeoffs.

Where voice fits

Voice earns its place when hands or eyes are busy (driving, cooking, warehouse work), for people who find screens hard to use because of low vision, motor impairments or limited literacy, and for anyone more comfortable speaking a language than typing it. Phone support and appointment booking are the obvious first targets because the channel is already voice. It is a poor fit for anything where the user needs to compare options side by side. Voice is an extra channel, and it does not replace accessible visual design; the checklist in our WCAG compliance guide still applies to the screen half of the product.

Frameworks to consider

Writing the pipeline by hand, as above, is the fastest way to understand where the time and the bugs live. For production, two open-source frameworks are worth a look. LiveKit Agents is Apache-2.0, Python with a separate Node.js library, and includes semantic turn detection that uses a transformer model to decide when the user is done. Pipecat is a BSD-2-Clause Python framework built around composable pipelines and a long list of STT, LLM and TTS integrations. Both let you keep the providers above and swap any one of them later, which is the main thing I would protect. Provider pricing and quality shift quickly, and being locked to one vendor for all three stages is the decision people regret.

---

Building a voice interface or adding voice to an existing product? CODERCOPS builds chatbots and conversational AI across text and voice channels. Get in touch to talk through your project.