Building a voice assistant with local LLMs means combining three components: a speech-to-text model that transcribes your voice, a language model that understands and responds, and a text-to-speech model that speaks the response aloud. None of these need to touch external servers — Whisper handles transcription locally, Ollama runs the LLM, and tools like Piper or Coqui handle speech synthesis on your hardware. The result is a fully private, offline-capable voice assistant that responds in natural language to whatever you ask.
Why Build Locally
Cloud voice assistants — Siri, Alexa, Google Assistant — send your voice to external servers for processing. For most casual use that is fine, but there are compelling reasons to run locally:
- Privacy — voice recordings of your home, office, or conversations never leave your machine
- No latency spikes — no dependency on internet connection quality or API availability
- Customisation — swap any component (different STT model, different LLM, different voice) without changing the rest of the pipeline
- Cost — once hardware is in place, marginal cost is zero regardless of usage volume
The tradeoff is quality at the edges — local STT and TTS are not quite as polished as the best cloud alternatives, and the LLM quality depends on your hardware. For factual Q&A, home automation, and personal assistant tasks, local models are more than capable.
The Three Components
Every local voice assistant pipeline needs the same three building blocks:
Speech-to-text (STT): Whisper is the standard choice — accurate, multilingual, and available in sizes from tiny (fast, lower accuracy) to large-v3 (slower, high accuracy). For a voice assistant where you want responsive real-time transcription, whisper-medium or whisper-small is the right balance. faster-whisper is a drop-in replacement that runs 2–4x faster than the original implementation with the same accuracy.
Language model: Any model running through Ollama works as the brain of the assistant. Llama 3.1 8B and Qwen2.5 7B are good starting points for general Q&A on modest hardware — both respond in a few seconds on a modern CPU and near-instantly on a mid-range GPU. For more capable responses on better hardware, Qwen2.5 32B or Llama 3.3 70B (quantised) produce noticeably richer answers.
Text-to-speech (TTS): Piper is the recommended local TTS engine — fast enough for real-time response (typically under one second), reasonable voice quality, and supports many voices and languages. Coqui TTS is an alternative with higher quality but slower generation. For a voice assistant, Piper’s speed advantage is usually worth its modest quality tradeoff.
Figure 1 — Local voice assistant: three-component pipeline
Wake Word Detection
A production-quality voice assistant needs wake word detection so it is not constantly transcribing ambient audio. OpenWakeWord is the recommended open-source option — it runs extremely lightweight (under 100MB RAM), supports custom wake words, and integrates cleanly into a Python pipeline. Porcupine from Picovoice is a polished commercial alternative with a generous free tier and better accuracy on noisy audio.
The wake word loop works like this: the program continuously listens on the microphone at low CPU cost using the wake word detector. When the wake word is detected, it switches to recording mode, captures audio until a silence threshold is reached, sends the captured audio to Whisper for transcription, passes the transcript to the LLM, and speaks the response via TTS. Then it returns to wake word listening mode. This idle-listen-respond cycle is the core loop of any voice assistant.
Latency: The Key User Experience Factor
The biggest challenge in local voice assistants is latency — the gap between finishing your question and hearing the response. Users tolerate up to about two seconds; beyond that it feels broken. Latency comes from three sources:
- STT latency — Whisper transcription of a 5-second clip takes 0.3–1.5 seconds depending on model size and hardware. faster-whisper reduces this significantly.
- LLM latency — first token latency on a local model is typically 0.5–3 seconds depending on model size and hardware. Streaming the response to TTS as tokens arrive (rather than waiting for the full response) cuts perceived latency dramatically.
- TTS latency — Piper generates speech in near-real-time, typically under 0.5 seconds for a short response. The key is to start playing audio as soon as the first sentence is ready rather than waiting for the full response to be synthesised.
The most effective latency optimisation is streaming: start TTS as soon as the first complete sentence arrives from the LLM rather than waiting for the full response. This means a user hears the beginning of the answer within one to two seconds of finishing their question, with the rest following naturally — the same experience as a human conversation rather than a noticeable processing pause.
System Prompt Design for a Voice Context
A voice assistant LLM needs a different system prompt from a text-based chatbot. Key differences:
- Keep responses short — spoken responses above 60 words feel too long. Instruct the model to give concise answers: “Respond in 1–3 sentences unless asked for more detail.”
- No markdown formatting — bullet points, headers, and code blocks make no sense when spoken aloud. Instruct the model to respond in plain prose only.
- Handle ambiguity by asking — tell the model to ask a clarifying question rather than guessing when the request is unclear, since misunderstandings in voice are harder to correct than in text.
- Natural spoken language — instruct the model to use contractions, natural phrasing, and conversational transitions rather than formal written language that sounds robotic when read aloud by TTS.
Adding Memory and Conversation History
A single-turn voice assistant that forgets each query immediately is limited. Adding a conversation history — maintaining the last 5–10 turns in the LLM context — makes the assistant feel significantly more natural. The user can say “tell me more about that” or “how does it compare to what you mentioned earlier” and the assistant understands the reference. The context window limitation of local models means you cannot maintain arbitrarily long histories, but 5–10 turns covers the typical length of a focused conversation on a single topic without running into context limits on 7B–14B models.
For persistent memory across sessions — the assistant remembering your preferences, ongoing projects, or regular tasks — a simple JSON file storing key facts the user has shared works well. Load it into the system prompt at startup: “Things the user has told you: prefers Celsius, works on machine learning projects, lives in Sydney.” This lightweight memory makes the assistant meaningfully more personalised without the complexity of a full vector database.
Hardware Requirements
A responsive local voice assistant needs modest hardware. On an Apple Silicon Mac (M1 or later), the combination of faster-whisper-small, Llama 3.1 8B via Ollama, and Piper TTS runs with total latency under two seconds and minimal battery impact — the M-series chips are well-optimised for the inference patterns all three components use. On Windows or Linux with a mid-range NVIDIA GPU (RTX 3060 or better), GPU-accelerated Whisper and Ollama inference produces comparable or better latency. On a CPU-only system, stick to the smallest models (whisper-tiny, a 3B or 7B LLM) and expect 3–5 second response latency — usable but noticeably slower than GPU-accelerated setups.
Extending the Assistant with Tools
The baseline voice assistant answers questions from the LLM’s training knowledge. Extending it with tools makes it genuinely useful for real-world tasks. Common extensions include: a web search tool (send the query to DuckDuckGo or Brave Search and include the results in the LLM context), a home automation integration (control smart devices via Home Assistant’s local API), a calendar integration (read and create calendar events), and a timer tool (start timers and alert when they expire). Each tool is a Python function the LLM can call via function calling or a simple prompt-based tool use pattern. The voice interface makes these tools feel natural — asking your assistant to “set a timer for 20 minutes” or “what’s on my calendar tomorrow” is a more comfortable interaction than reaching for a phone or typing a command.
Building a local voice assistant is genuinely achievable with available open-source tools and runs well on hardware most developers already own. The pipeline is modular — each component can be swapped independently as better models become available — and the privacy and customisation benefits over cloud alternatives are substantial for anyone who wants an assistant that does not report home. The main ongoing investment is keeping up with the fast-moving local model ecosystem, where meaningfully better STT, LLM, and TTS options appear every few months.
Integrating with Smart Home Systems
One of the most practical extensions for a local voice assistant is smart home integration. Home Assistant — the popular open-source home automation platform — exposes a local REST API that makes it straightforward to query and control devices by name: turn lights on and off, read temperature sensors, lock doors, or trigger automations. Because Home Assistant runs locally on your network, integrating it with your local voice assistant means the entire pipeline — from speaking a command to the smart device responding — never touches the internet. This combination is increasingly attractive for privacy-conscious users who want smart home functionality without sending commands through cloud servers. The integration works by giving the LLM a tool it can call: a Python function that sends the appropriate Home Assistant API request when the model decides a device action is needed. The LLM handles the natural language understanding of commands like “make the living room warmer” or “turn off everything downstairs” and maps them to specific device actions, while Home Assistant handles the actual hardware control.
Voice Assistant vs Chatbot: What Changes
Building for voice changes several design decisions that text-based chatbots do not need to consider. Response length matters enormously — a 200-word text response reads fine on screen but takes over a minute to speak, which feels interminable in conversation. Instructing the model to keep spoken responses to two or three sentences, and to offer to elaborate rather than front-loading detail, produces a much more natural interaction. Error recovery is also different: in a text chatbot, misunderstandings are visible and easy to correct by scrolling up and re-reading. In voice, misunderstandings feel more disorienting because the interaction is ephemeral. Building in explicit confirmation for high-stakes actions — “I heard you say to delete the reminder, is that right?” — prevents the most damaging class of voice assistant errors. Finally, personality consistency matters more in voice because the interaction feels more personal; investing time in a system prompt that gives the assistant a consistent tone and communication style produces a noticeably more satisfying experience than a generic assistant prompt.