OpenAI's Realtime API Now Has a Local Open-Source Twin

When Hugging Face first published speech-to-speech in late 2024, it was a proof of concept with a chip on its shoulder: an attempt to approximate GPT-4o's voice mode by chaining open models — Silero VAD, Whisper, an instruct LLM, Parler-TTS — into one Python pipeline. Fun demo, rough edges, and it mostly faded from view. The repo that's back on GitHub's trending page is a different animal. It's now a pip-installable server that speaks OpenAI's Realtime WebSocket protocol, ships sane model defaults, and — the part that actually matters — runs in production as the conversation backend for thousands of Reachy Mini robots, the desktop bot Hugging Face inherited when it bought Pollen Robotics.

That production detail changes how you should read this project. Open-source voice pipelines are easy to demo and brutal to ship: turn-taking, barge-in, streaming synthesis, and thread coordination are where they all die. A pipeline that survives daily use on a consumer robot sitting in people's homes has been through that grinder.

The architecture bet: cascade, not end-to-end

The design is still a cascade — VAD → STT → LLM → TTS, each stage in its own thread, streaming into the next. That's worth dwelling on, because two years ago the smart money said cascades were a dead end. Speech-native models like Kyutai's Moshi and OpenAI's own Realtime models process audio tokens directly, preserving prosody and cutting the transcription round-trip. They were supposed to make the pipeline approach obsolete.

It hasn't worked out that way, at least not in open source. Speech-native open models remain scarce, hard to fine-tune, and locked to whatever LLM they were fused with. A cascade lets you ride every improvement in every layer independently — and the layers have improved a lot. The current defaults tell the story: NVIDIA's Parakeet-TDT 0.6B v3 for transcription (streaming-friendly, 25 languages, absurdly fast for its size), Qwen3-TTS for synthesis, Silero VAD v5 for turn detection, and any LLM you like — because the LLM slot is just an OpenAI-compatible endpoint. You lose some prosodic nuance versus speech-native models; the user's tone gets flattened to text before the LLM sees it. In exchange you get model choice, tool calling that actually works, inspectable transcripts, and the ability to swap one weak link without retraining anything.

The protocol is the real headline

The most consequential line in the README isn't about models at all: the server exposes ws://localhost:8765/v1/realtime, compatible with OpenAI's Realtime API protocol. This is the /v1/chat/completions playbook repeating one layer up the stack. OpenAI's text API schema became the lingua franca that vLLM, llama.cpp, Ollama, and every router adopted — to the point where "OpenAI-compatible" stopped meaning OpenAI. Hugging Face adopting the Realtime protocol for voice is a bet that the same consolidation is coming for speech, and it's a good bet.

For developers, the practical consequence is significant: client code written against OpenAI's Realtime API — browser audio capture, interruption handling, tool-call events — can point at a local endpoint instead. Your voice frontend stops caring whose models are behind the socket. That's the same decoupling that made local LLM serving boring, in the best sense.

What adoption actually looks like

The quickstart is genuinely two commands if you're fine with a hosted LLM:

pip install speech-to-speech
speech-to-speech   # OPENAI_API_KEY set → Realtime server on :8765

The fully local version pairs it with a llama.cpp server — the README's recommended pairing is Gemma 4 E4B in GGUF form — and points the pipeline's Responses-API URL at localhost:8080. Because the LLM lives in a separate process, you can put it on a different box entirely: STT and TTS on a Mac's Apple Silicon via MLX, the LLM on a CUDA machine down the hall, or vLLM in your VPC. That topology — voice loop at the edge, brain wherever GPUs are cheap — is the deployment shape most teams actually want and most hosted voice platforms won't give you.

The trade-offs are real, though. Component licensing varies even when the pipeline is Apache 2.0, so check each model before shipping commercially. The minimal TCP socket mode drops interruption handling and tool-call events, so anything conversational needs the WebSocket path. There's dependency friction typical of audio ML (the README flags a DeepFilterNet/Pocket TTS numpy conflict, and Qwen3-TTS's GGML backend assumes CUDA 12.8 unless you grab a custom wheel). And nothing published yet speaks to multi-tenant scale — this is architected as one pipeline per conversation, which is perfect for a robot, a kiosk, or an on-prem assistant, and not an answer for hundred-concurrent-call telephony. For that, orchestration frameworks like Pipecat and LiveKit Agents still own the conversation, and notably both can now sit in front of open components too.

Who should care

If you're paying per-minute for a hosted voice API to power a single-tenant product — a device, an in-car assistant, an internal tool, anything with privacy constraints — this is now the obvious first thing to prototype, because your existing Realtime client code mostly carries over. If you're running a contact center, keep your orchestrator and watch this space. And if you're a hosted voice-API vendor whose moat is gluing STT to an LLM to TTS, the moat is now a pip install.

The honest verdict: this is the moment the open voice stack went from "impressive demo" to "default starting point." Not because any single model beat GPT-4o's voice — none has — but because the boring things finally got done: a standard protocol, swappable parts, production mileage, and defaults that work. That's how open source wins these fights: not with a better model, but with a better socket.