IonBin
All pages
Search
Draft, pending editorial review · not indexed

Build your own "Jarvis" voice assistant

Advanced · several weeks · checked 2026-10-18

A full voice assistant: wake word or push-to-talk, Deepgram streaming speech-to-text with turn detection, an LLM or agent platform that can use tools through MCP, and ElevenLabs streaming text-to-speech. Covers turn-taking, latency, memory, safety and consent.

What you’ll build

A voice assistant you talk to naturally: it notices when you start and stop speaking, answers out loud with latency you measure, can be interrupted, remembers what you allow it to, and can take actions through tools (calendar, tasks, deploys) with confirmation for anything risky.

A prototype built by IonBin’s founder uses this cascade: Deepgram streaming speech-to-text, Claude as the brain, and ElevenLabs streaming text-to-speech, with barge-in, a self-echo guard and a name check before actions. Its design targets are first audio in under 800 ms, interruption handling and read-back confirmation before high-stakes actions. These are design targets, not measured results for this guide.

Architecture and data flow

  1. Wake: a local wake-word engine or a push-to-talk key opens the microphone. Keep wake detection on the device.
  2. Listen: audio streams to Deepgram Flux over /v2/listen. Flux emits turn events: StartOfTurn, EagerEndOfTurn, TurnResumed and EndOfTurn.
  3. Think: use the final EndOfTurn transcript for tool decisions. Optional EagerEndOfTurn can prepare a speculative response when configured, but must not commit consequential actions. Cancel or replace that work on TurnResumed. Tools require a compatible MCP client, scoped credentials and enforced authorization.
  4. Speak: stream the LLM’s tokens into ElevenLabs’ text-to-speech WebSocket as they arrive; send flush at the end of the turn; play audio chunks as they land.
  5. Barge-in: with active microphone input and tested echo cancellation, handle StartOfTurn by stopping playback, cancelling pending generation and discarding stale audio buffers. Tag turns so delayed responses cannot resume the old answer. Cancellation cannot undo a tool action already committed.
  6. Remember: write short summaries and explicit preferences to a local store you can inspect and delete.

Managed alternative: Deepgram’s Voice Agent API runs listen, think and speak over one WebSocket (Settings, then streamed audio, with events such as UserStartedSpeaking for barge-in). It can reduce custom orchestration; compare its supported models, tools, settings, data handling and billing with the separate-service design.

Components and why

Component Choice Why
Speech-to-text Deepgram Flux Model-integrated end-of-turn detection, configurable thresholds (eot_threshold, eager_eot_threshold, eot_timeout_ms) and a recommended 80 ms chunk size
Brain An LLM or agent platform with tools Claude, Codex-family models or your agent of choice; tools through MCP
Text-to-speech ElevenLabs Flash over WebSocket Flash models are documented at about 75 ms model latency; the WebSocket accepts streaming text from an LLM
Memory Local SQLite Inspectable local storage; access control, encryption, backups and deletion need a design
Secrets OS keychain Keep long-lived keys outside source code and restrict runtime access
Code home GitHub Reviewable changes; Actions runs your tests

Step outline

  1. Spec and repo: define three voice tasks you actually want, the tools each needs, and which actions need confirmation.
  2. Text-first loop: build the agent with tools as a chat app first. Voice comes after the brain works.
  3. Add speech-to-text with Flux; log turn events and tune thresholds on your own voice and room.
  4. Add text-to-speech streaming with flush at end of turn; measure time to first audio.
  5. Choose an interruption design: full duplex keeps listening with tested echo cancellation and can support spoken barge-in. Half duplex mutes during playback and therefore needs a button or hotkey to interrupt; it cannot hear a spoken interruption while muted.
  6. Latency budget: measure each hop (speech end → final transcript → first token → audible audio), including buffering, network and tool time. ElevenLabs’ roughly 75 ms Flash figure describes model inference, not this complete pipeline. Test reconnects and long pauses as well as the best case.
  7. Safety rails: read back and confirm before sending, paying, deleting or deploying; fall back to text when audio quality drops.
  8. Memory with consent: store only what the user approves; give a “forget that” command.
  9. Wake word last, once push-to-talk works well.

AI platforms that help

A voice assistant is several services wired together (speech-to-text, an agent with tools, text-to-speech), so the agent you build with should be the one that is comfortable with orchestration code. Claude Code fits the orchestration, the MCP tool wiring and the latency tests. Cursor fits the client app. OpenAI Codex can take well-specified pieces such as the wake-word adapter in the background. Google Antigravity can run the speech, agent and voice modules as parallel tasks. Grok Bot is not a build tool here; it is a reference for an agent that works in the background and asks for approval before sensitive steps, which is the behaviour this build needs. Keep the code in a GitHub repository from day one so every agent change arrives as a reviewable diff; the platform comparison lists what each one documents.

Costs

  • Deepgram bills streaming speech-to-text by audio duration; ElevenLabs bills text-to-speech by characters or credits. To verify: both pricing pages for your models and volume.
  • LLM costs scale with conversation length and tool calls. To verify with your provider.
  • A local wake-word engine and SQLite add no usage cost; check the wake-word engine’s license.

Risks

  • Consent and recording: tell people when the microphone is live; don’t record others without consent where the law requires it.
  • Voice cloning: only clone voices you have the right to use; ElevenLabs’ Prohibited Use Policy forbids cloning or impersonating a living person’s voice without their explicit and informed consent.
  • Wrong actions: speech recognition errors can change names, amounts or intent. Read back consequential changes and enforce account authorization; a spoken name check is not identity authentication.
  • Always-on privacy: keep wake-word detection local and show a clear listening indicator.
  • Latency creep: measure slow tool calls and acknowledge pending work without claiming it succeeded. A dropped connection must not automatically repeat a charge, message or deployment.

Sources

Deepgram and ElevenLabs official docs are the primary sources for the speech pipeline. Design targets draw on prototype notes and require measurement in your implementation.

Source list

To verify Measure full-pipeline latency, interruption, echo, resumed turns and reconnect behavior. · Choose a licensed wake-word engine and verify selected-model costs and applicable recording consent. · The complete assistant and its action-authorization controls have not been tested in this audit.

Related in the wiki

ElevenLabs is a trademark of its owner. IonBin is an independent publisher and is not affiliated with, sponsored by, or endorsed by ElevenLabs. Product names are used for identification and review purposes only.