IonBin
All pages
Search
Draft, pending editorial review · not indexed

Build your own dictation tool with Deepgram (Wispr Flow-style)

Intermediate · a week · checked 2026-10-18

A proposed desktop dictation tool: hold a hotkey, speak, review the transcript and paste it into a supported app. Combine Deepgram streaming transcription with optional cleanup and OS-specific paste handling. Validate focus, clipboard preservation and offline behavior before relying on it.

What you’ll build

A desktop dictation tool in the style of Wispr Flow: hold a hotkey, talk, release, review and paste into a tested application. Global shortcuts, microphone capture and simulated paste need separate OS-specific implementation. Some fields and applications may reject simulated input.

A prototype built by IonBin’s founder includes this flow (hotkey → speech recognition → cleanup → editable preview → clipboard), with a local speech model as the default. This guide uses Deepgram for the cloud speech step and keeps a local fallback.

Architecture and data flow

  1. Hotkey down (a global shortcut, e.g. Ctrl+Shift+Space) starts microphone capture.
  2. Audio streams in small PCM chunks over a WebSocket to Deepgram’s live transcription endpoint; interim and final transcripts stream back.
  3. Hotkey up stops capture, sends remaining audio and a Nova CloseStream message, then keeps reading the final results and server close. Do not close the socket immediately. For a connection you keep open, Finalize flushes buffered audio; its from_finalize acknowledgment is not guaranteed when there is little buffered audio. Handle timeout and disconnect explicitly.
  4. Cleanup: an optional LLM pass proposes punctuation and style changes. Preserve the raw transcript and show changes for review; an instruction to preserve meaning is not a guarantee. If cleanup fails, offer the raw transcript instead of inventing missing words.
  5. Paste: check the intended target, preserve supported clipboard formats, write the reviewed text and request OS-specific paste. Restore only if the clipboard still contains this app’s temporary value; do not overwrite something the user copied meanwhile. If full preservation is unsupported, offer an explicit copy action.
  6. Optional: show an editable preview first, or append the text to a daily note.

Components and why

Component Choice Why
App shell Tauri (Rust + web UI) Small native app; plugins for global shortcuts and the clipboard
Speech-to-text Deepgram streaming (Nova family for dictation) Low-latency streaming; smart formatting adds punctuation and numerals; keyterm prompting helps with names and jargon
Local fallback whisper.cpp Can run locally after downloading its model and dependencies
Cleanup A small, fast LLM, or a local model Proposes cleanup that the user reviews against the raw transcript
Paste Clipboard plugin + OS paste keystroke Requires OS-specific paste code and tests in each supported app
Code home GitHub Versioned, reviewable changes from your coding agent

Deepgram’s Flux model is built for voice agents and turn detection; for push-to-talk dictation the hotkey already marks the turn, so a Nova model with smart formatting is the simpler fit. Test both on your own voice.

Step outline

  1. Repo and spec: create a GitHub repo; list the hotkey, the apps you’ll test in, and a latency target.
  2. Hotkey + mic: register the global shortcut and capture audio while it’s held.
  3. Stream to Deepgram: open the WebSocket on key-down, send audio chunks, read transcripts; keep the API key in the OS keychain, never in code.
  4. Add keyterms for your company, product and people names.
  5. Cleanup pass with a reviewable diff and a raw-transcript fallback.
  6. Clipboard handling: test text, images, rich text, changed focus and a concurrent copy. Tauri’s clipboard plugin is not a general snapshot/restore API for every clipboard format, nor does it simulate a paste keystroke.
  7. Offline mode: preload whisper.cpp and its model. Local-only must also disable cloud cleanup and other outbound data paths; test with networking blocked.
  8. Package and test in your top five apps.

AI platforms that help

Most of the work here is streaming-audio plumbing and operating-system integration rather than UI, so pick the agent by where the glue code lives. Claude Code in the terminal fits the WebSocket client, the cleanup pass and the tests. Cursor fits the small settings window. OpenAI Codex can run the parser tests and packaging in the background. Google Antigravity can split the macOS and Windows hotkey and paste handling across parallel agents. Keep the code in a GitHub repository from day one so every agent change arrives as a reviewable diff; the platform comparison lists what each one documents.

Costs

  • Deepgram bills streaming transcription by audio duration. To verify: the current rate for your model on Deepgram’s pricing page.
  • Calculate cleanup cost from actual input/output tokens and the selected provider rate; include retries. A consumer chat subscription does not necessarily include API usage.
  • Local inference avoids a hosted inference fee but consumes hardware, storage and electricity.

Risks

  • Privacy: audio leaves the device when you use cloud speech-to-text. Make the mode visible and offer local-only.
  • Clipboard loss or wrong target: test supported formats and focus changes; use copy-only mode where preservation or paste cannot be verified.
  • Over-editing: the cleanup prompt must not add facts or change meaning; show a diff or preview for long dictation.
  • OS support: Tauri’s global-shortcut plugin targets desktop systems, not mobile. Microphone, shortcut registration and simulated input have distinct platform permissions and restrictions; test each supported OS.

Sources

Deepgram, Tauri and MDN docs are the primary sources. Wispr Flow is cited only as the reference product category.

Source list

To verify Run OS/app-specific microphone, hotkey, focus, clipboard and final-audio tests. · Measure selected-model API cost and dictation quality; no complete implementation was tested in this audit.

Related in the wiki