Build a Real-Time Speech-to-Text Pipeline with Whisper and WebSockets

What you'll build

A browser page that streams your microphone to a self-hosted Whisper model over a WebSocket and shows live captions: gray interim text that updates about once a second, committed black text every 15 seconds. Everything runs on your machine. No audio leaves it.

Prerequisites

Verified against faster-whisper 1.2.1, websockets 17.1, and ctranslate2 4.8.2 on Python 3.13 (macOS 15). Linux and Windows work the same way.

• Python 3.11 or newer (websockets 17.x requires 3.11+)
• Chrome, Edge, or Safari. Firefox can't resample mic input to a 16 kHz AudioContext; see Troubleshooting
• ~150 MB of disk for the base model, downloaded from Hugging Face on first run
• A CPU is enough for the base model. GPU needs CUDA 12 and cuDNN 9

Step 1: Install the server dependencies

faster-whisper runs Whisper on CTranslate2, about 4x faster than the reference implementation, which is what makes once-a-second re-transcription workable on a laptop CPU. websockets handles the socket side.

mkdir whisper-live && cd whisper-live
python3 -m venv venv && source venv/bin/activate
pip install "faster-whisper==1.2.1" "websockets==17.1"

Step 2: Write the transcription server

The protocol is deliberately dumb: the client sends raw 16-bit little-endian PCM at 16 kHz as binary frames, the server replies with JSON. The server appends every frame to a buffer and, once a second, re-transcribes the whole buffer and pushes the result as a partial. When the buffer hits 15 seconds it commits the text as a final and starts fresh. Re-transcribing the growing buffer costs more than a true incremental decoder, but it self-corrects earlier words as context arrives and needs no alignment logic.

Save this as server.py:

import asyncio
import json

import numpy as np
from faster_whisper import WhisperModel
from websockets.asyncio.server import serve

SAMPLE_RATE = 16000
BYTES_PER_SECOND = SAMPLE_RATE * 2  # 16-bit mono PCM
MAX_WINDOW_SECONDS = 15

model = WhisperModel("base", device="cpu", compute_type="int8")


def transcribe(pcm: bytes) -> str:
    audio = np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0
    segments, _ = model.transcribe(
        audio,
        language="en",
        beam_size=1,
        vad_filter=True,
        condition_on_previous_text=False,
    )
    return " ".join(segment.text.strip() for segment in segments)


async def handler(websocket):
    buffer = bytearray()

    async def worker():
        last_len = 0
        while True:
            await asyncio.sleep(1.0)
            if len(buffer) == last_len or len(buffer) < BYTES_PER_SECOND:
                continue
            snapshot = bytes(buffer)
            last_len = len(snapshot)
            text = await asyncio.to_thread(transcribe, snapshot)
            if len(snapshot) >= MAX_WINDOW_SECONDS * BYTES_PER_SECOND:
                # Drop only what was transcribed, keeping audio
                # that arrived while the model was busy.
                del buffer[: len(snapshot)]
                last_len = 0
                await websocket.send(json.dumps({"type": "final", "text": text}))
            else:
                await websocket.send(json.dumps({"type": "partial", "text": text}))

    task = asyncio.create_task(worker())
    try:
        async for message in websocket:
            if isinstance(message, bytes):
                buffer.extend(message)
    finally:
        task.cancel()


async def main():
    async with serve(handler, "localhost", 8765) as server:
        print("Listening on ws://localhost:8765")
        await server.serve_forever()


if __name__ == "__main__":
    asyncio.run(main())

Four choices matter here. model.transcribe accepts a float32 numpy array at 16 kHz directly, so there's no temp-file dance. asyncio.to_thread keeps the CPU-bound decode off the event loop, so incoming frames keep landing in the buffer while the model runs. beam_size=1 with condition_on_previous_text=False trades a little accuracy for latency and stops the model hallucinating repeats on partial audio. vad_filter=True runs Silero VAD first, so silence and keyboard noise don't get decoded into phantom words.

Step 3: Build the browser client

Creating the AudioContext at 16 kHz makes the browser resample the mic for you, so the worklet only has to convert float32 samples to int16. Save this as index.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Live transcription</title>
</head>
<body>
  <button id="start">Start listening</button>
  <p id="status">Idle</p>
  <p id="committed"></p>
  <p id="partial" style="color: #888"></p>
  <script>
    const status = document.getElementById("status");
    const committed = document.getElementById("committed");
    const partial = document.getElementById("partial");

    document.getElementById("start").onclick = async () => {
      const ws = new WebSocket("ws://localhost:8765");
      ws.onmessage = (event) => {
        const msg = JSON.parse(event.data);
        if (msg.type === "final") {
          committed.textContent += msg.text + " ";
          partial.textContent = "";
        } else {
          partial.textContent = msg.text;
        }
      };
      await new Promise((resolve) => (ws.onopen = resolve));

      const stream = await navigator.mediaDevices.getUserMedia({
        audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
      });
      // 16 kHz context: the browser resamples the mic to Whisper's input rate.
      const context = new AudioContext({ sampleRate: 16000 });
      await context.audioWorklet.addModule("pcm-processor.js");
      const source = context.createMediaStreamSource(stream);
      const node = new AudioWorkletNode(context, "pcm-processor");
      node.port.onmessage = (event) => {
        if (ws.readyState === WebSocket.OPEN) ws.send(event.data);
      };
      source.connect(node);
      node.connect(context.destination); // keeps the graph pulled; the node outputs silence
      status.textContent = "Listening...";
    };
  </script>
</body>
</html>

Step 4: Add the AudioWorklet processor

An AudioWorklet hands you audio in 128-frame chunks on the audio thread. Sending a WebSocket message every 8 ms would be wasteful, so the processor batches 4096 samples (~256 ms) per message. Save this as pcm-processor.js next to index.html:

class PCMProcessor extends AudioWorkletProcessor {
  constructor() {
    super();
    this.samples = [];
    this.chunkSize = 4096; // ~256 ms at 16 kHz per WebSocket message
  }

  process(inputs) {
    const input = inputs[0][0];
    if (!input) return true;
    for (let i = 0; i < input.length; i++) this.samples.push(input[i]);
    if (this.samples.length >= this.chunkSize) {
      const pcm = new Int16Array(this.samples.length);
      for (let i = 0; i < this.samples.length; i++) {
        const s = Math.max(-1, Math.min(1, this.samples[i]));
        pcm[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
      }
      this.port.postMessage(pcm.buffer, [pcm.buffer]);
      this.samples = [];
    }
    return true;
  }
}

registerProcessor("pcm-processor", PCMProcessor);

Verify it works

Start the server. The first run downloads the model, then you should see:

Listening on ws://localhost:8765

Serve the client from a second terminal in the same directory. getUserMedia needs a secure context and file:// doesn't qualify, but http://localhost does:

python3 -m http.server 8000

Open http://localhost:8000, click Start listening, allow the microphone, and talk. Within a couple of seconds the gray line starts updating and grows as you speak. Streaming a test sentence through this exact pipeline produced this sequence of messages on the wire:

{"type": "partial", "text": "Hello World!"}
{"type": "partial", "text": "Hello World, this is a live transcription test streaming"}
{"type": "partial", "text": "Hello World, this is a live transcription test streaming audio over a web socket."}

Notice the second partial extends and corrects the first. That's the re-transcription window doing its job.

Troubleshooting

• TypeError: handler() missing 1 required positional argument: 'path' — you copied a handler signature from an old tutorial. The legacy API called handler(websocket, path); the current websockets.asyncio API passes only the connection. Delete the path parameter.

• NotSupportedError: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. — Firefox. It won't resample a mic stream into a 16 kHz context (Mozilla bug 1725336). Use Chrome, Edge, or Safari, or create the context at the default rate and downsample inside the worklet before converting to int16.

• Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so} — you switched to device="cuda" without cuDNN 9. Run pip install nvidia-cublas-cu12 nvidia-cudnn-cu12 and add both packages' lib directories to LD_LIBRARY_PATH, or stay on CPU with compute_type="int8".

• OSError: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8765) — a previous server instance is still holding the port (errno 98 on Linux). Find it with lsof -nP -iTCP:8765 and kill it.

Next steps

The 15-second commit window is the crudest part of this design: text can flicker until it's committed. Production streaming systems solve that with the LocalAgreement policy, where a prefix is committed once two consecutive transcriptions agree on it. whisper_streaming implements it on top of faster-whisper and is a natural next read. On the model side, swap base for distil-large-v3 or large-v3 with device="cuda", compute_type="float16" for much better accuracy at similar latency. And before exposing this beyond localhost, put the socket behind TLS (wss://), because browsers block mixed-content WebSockets and mic capture on insecure origins anyway.