What you’ll build
A voice assistant you talk to naturally: it notices when you start and stop speaking, answers out loud with latency you measure, can be interrupted, remembers what you allow it to, and can take actions through tools (calendar, tasks, deploys) with confirmation for anything risky.
A prototype built by IonBin’s founder uses this cascade: Deepgram streaming speech-to-text, Claude as the brain, and ElevenLabs streaming text-to-speech, with barge-in, a self-echo guard and a name check before actions. Its design targets are first audio in under 800 ms, interruption handling and read-back confirmation before high-stakes actions. These are design targets, not measured results for this guide.
Architecture and data flow
- Wake: a local wake-word engine or a push-to-talk key opens the microphone. Keep wake detection on the device.
- Listen: audio streams to Deepgram Flux over
/v2/listen. Flux emits turn events:StartOfTurn,EagerEndOfTurn,TurnResumedandEndOfTurn. - Think: use the final
EndOfTurntranscript for tool decisions. OptionalEagerEndOfTurncan prepare a speculative response when configured, but must not commit consequential actions. Cancel or replace that work onTurnResumed. Tools require a compatible MCP client, scoped credentials and enforced authorization. - Speak: stream the LLM’s tokens into ElevenLabs’ text-to-speech WebSocket as they arrive; send
flushat the end of the turn; play audio chunks as they land. - Barge-in: with active microphone input and tested echo cancellation, handle
StartOfTurnby stopping playback, cancelling pending generation and discarding stale audio buffers. Tag turns so delayed responses cannot resume the old answer. Cancellation cannot undo a tool action already committed. - Remember: write short summaries and explicit preferences to a local store you can inspect and delete.
Managed alternative: Deepgram’s Voice Agent API runs listen, think and speak over one WebSocket (Settings, then streamed audio, with events such as UserStartedSpeaking for barge-in). It can reduce custom orchestration; compare its supported models, tools, settings, data handling and billing with the separate-service design.
Components and why
| Component | Choice | Why |
|---|---|---|
| Speech-to-text | Deepgram Flux | Model-integrated end-of-turn detection, configurable thresholds (eot_threshold, eager_eot_threshold, eot_timeout_ms) and a recommended 80 ms chunk size |
| Brain | An LLM or agent platform with tools | Claude, Codex-family models or your agent of choice; tools through MCP |
| Text-to-speech | ElevenLabs Flash over WebSocket | Flash models are documented at about 75 ms model latency; the WebSocket accepts streaming text from an LLM |
| Memory | Local SQLite | Inspectable local storage; access control, encryption, backups and deletion need a design |
| Secrets | OS keychain | Keep long-lived keys outside source code and restrict runtime access |
| Code home | GitHub | Reviewable changes; Actions runs your tests |
Step outline
- Spec and repo: define three voice tasks you actually want, the tools each needs, and which actions need confirmation.
- Text-first loop: build the agent with tools as a chat app first. Voice comes after the brain works.
- Add speech-to-text with Flux; log turn events and tune thresholds on your own voice and room.
- Add text-to-speech streaming with flush at end of turn; measure time to first audio.
- Choose an interruption design: full duplex keeps listening with tested echo cancellation and can support spoken barge-in. Half duplex mutes during playback and therefore needs a button or hotkey to interrupt; it cannot hear a spoken interruption while muted.
- Latency budget: measure each hop (speech end → final transcript → first token → audible audio), including buffering, network and tool time. ElevenLabs’ roughly 75 ms Flash figure describes model inference, not this complete pipeline. Test reconnects and long pauses as well as the best case.
- Safety rails: read back and confirm before sending, paying, deleting or deploying; fall back to text when audio quality drops.
- Memory with consent: store only what the user approves; give a “forget that” command.
- Wake word last, once push-to-talk works well.
AI platforms that help
A voice assistant is several services wired together (speech-to-text, an agent with tools, text-to-speech), so the agent you build with should be the one that is comfortable with orchestration code. Claude Code fits the orchestration, the MCP tool wiring and the latency tests. Cursor fits the client app. OpenAI Codex can take well-specified pieces such as the wake-word adapter in the background. Google Antigravity can run the speech, agent and voice modules as parallel tasks. Grok Bot is not a build tool here; it is a reference for an agent that works in the background and asks for approval before sensitive steps, which is the behaviour this build needs. Keep the code in a GitHub repository from day one so every agent change arrives as a reviewable diff; the platform comparison lists what each one documents.
Costs
- Deepgram bills streaming speech-to-text by audio duration; ElevenLabs bills text-to-speech by characters or credits. To verify: both pricing pages for your models and volume.
- LLM costs scale with conversation length and tool calls. To verify with your provider.
- A local wake-word engine and SQLite add no usage cost; check the wake-word engine’s license.
Risks
- Consent and recording: tell people when the microphone is live; don’t record others without consent where the law requires it.
- Voice cloning: only clone voices you have the right to use; ElevenLabs’ Prohibited Use Policy forbids cloning or impersonating a living person’s voice without their explicit and informed consent.
- Wrong actions: speech recognition errors can change names, amounts or intent. Read back consequential changes and enforce account authorization; a spoken name check is not identity authentication.
- Always-on privacy: keep wake-word detection local and show a clear listening indicator.
- Latency creep: measure slow tool calls and acknowledge pending work without claiming it succeeded. A dropped connection must not automatically repeat a charge, message or deployment.
Sources
Deepgram and ElevenLabs official docs are the primary sources for the speech pipeline. Design targets draw on prototype notes and require measurement in your implementation.