OpenAI’s Whisper is an open-source speech recognition model that runs locally and transcribes audio with impressive accuracy across dozens of languages. It handles podcasts, video interviews, meetings, lectures, voice memos, and phone call recordings — any audio content you want converted to text. This guide covers setting up local Whisper transcription, the practical workflow for video and podcast files, and how to connect transcription to downstream LLM analysis.
Why Local Transcription?
Transcribing audio through cloud services sends potentially sensitive audio to external servers. Meeting recordings contain confidential discussions. Podcast interviews may have off-the-record portions. Medical or legal audio files contain protected information. Local Whisper transcription keeps all audio content on your machine. Beyond privacy, local transcription is free — cloud transcription APIs charge per minute of audio, which accumulates quickly for regular podcast or video processing. A local setup processes unlimited audio at no marginal cost after the initial setup.
Installing Whisper
Whisper runs via Python. The fastest setup uses whisper.cpp (a C++ implementation) or the faster-whisper Python library, both of which are significantly faster than the original OpenAI implementation:
# Option 1: faster-whisper (recommended — GPU accelerated, accurate)
pip install faster-whisper
# Option 2: original OpenAI whisper
pip install openai-whisper
# Option 3: whisper.cpp (fastest on CPU, including Apple Silicon)
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make # or make WHISPER_METAL=1 for Apple Silicon GPU acceleration
bash ./models/download-ggml-model.sh medium.enFor Python workflows, faster-whisper is the recommended option. It uses CTranslate2 for efficient inference, supports CUDA and CPU backends, and achieves 4-8x speed compared to the original implementation at the same accuracy.
Whisper Model Selection
Whisper comes in several size tiers. The right choice depends on your accuracy requirements and hardware:
tiny / tiny.en (~40MB): Fastest, lowest accuracy. Adequate for clear audio with a single speaker and simple vocabulary. Useful for high-volume processing where speed matters more than perfect accuracy.
base / base.en (~140MB): Good balance for simple transcription tasks. Works well for podcasts with clear audio.
small / small.en (~460MB): Good accuracy, reasonable speed. Handles accents and moderate background noise. A solid default for most use cases.
medium / medium.en (~1.5GB): High accuracy. Handles challenging audio well — multiple speakers, background noise, accents, technical vocabulary. The practical recommendation for podcast and video transcription where quality matters.
large-v3 (~3GB): Best accuracy. Use when you need the highest quality — medical dictation, legal proceedings, audio with significant noise or multiple overlapping speakers. Noticeably slower than medium.
Basic Transcription Script
from faster_whisper import WhisperModel
from pathlib import Path
import json
# Load model once (reuse across files)
model = WhisperModel(
"medium",
device="cuda", # or "cpu" or "mps" for Apple Silicon
compute_type="float16" # or "int8" for CPU
)
def transcribe(audio_path: str, language: str = None) -> dict:
segments, info = model.transcribe(
audio_path,
language=language, # None = auto-detect
beam_size=5,
vad_filter=True, # remove silence
word_timestamps=True # optional: get word-level timing
)
transcript_segments = []
full_text_parts = []
for seg in segments:
transcript_segments.append({
"start": round(seg.start, 2),
"end": round(seg.end, 2),
"text": seg.text.strip()
})
full_text_parts.append(seg.text.strip())
return {
"language": info.language,
"duration_s": info.duration,
"full_text": " ".join(full_text_parts),
"segments": transcript_segments
}
# Transcribe a podcast episode
result = transcribe("podcast_episode_42.mp3")
print(f"Language: {result['language']}, Duration: {result['duration_s']/60:.1f} min")
print(result["full_text"][:500]) # first 500 chars
# Save to file
Path("transcript.txt").write_text(result["full_text"])
Path("transcript_segments.json").write_text(json.dumps(result["segments"], indent=2))Figure 1 — Whisper Model Comparison
Extracting Audio from Video Files
For video files (MP4, MKV, MOV, WebM), extract the audio track first using ffmpeg. Whisper can accept MP4 directly in many cases, but extracting audio first is more reliable and reduces processing time:
import subprocess
import tempfile, os
def transcribe_video(video_path: str) -> dict:
# Extract audio to temp WAV file
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
tmp_path = tmp.name
subprocess.run([
"ffmpeg", "-i", video_path,
"-ar", "16000", # Whisper expects 16kHz
"-ac", "1", # mono
"-y", # overwrite if exists
tmp_path
], check=True, capture_output=True)
try:
result = transcribe(tmp_path)
finally:
os.unlink(tmp_path) # clean up temp file
return result
# Transcribe a recorded interview
result = transcribe_video("interview_recording.mp4")
print(result["full_text"])The -ar 16000 -ac 1 flags convert to the 16kHz mono format that Whisper expects, which slightly reduces file size and improves processing speed. For long videos (over an hour), the processing time on a GPU is typically 5-20% of the video duration — a 2-hour video transcribes in 6-24 minutes depending on model and hardware.
Connecting Transcription to LLM Analysis
The real value of local transcription comes when you connect it to an LLM for analysis. Transcribe a meeting recording, then use Ollama to extract decisions and action items. Transcribe a podcast, then summarise it. Transcribe a lecture, then generate study notes. The pipeline is two local steps with no external services:
import ollama
def transcribe_and_analyse(audio_path: str, analysis_type: str = "meeting") -> dict:
# Step 1: transcribe
print("Transcribing...")
transcript = transcribe(audio_path)
# Step 2: analyse with LLM
print("Analysing...")
prompts = {
"meeting": "Extract: (1) Key decisions made, (2) Action items with owners, (3) Open questions. Format clearly.",
"podcast": "Summarise the main topics covered and key insights from this transcript in 5 bullet points.",
"lecture": "Create structured study notes with key concepts, definitions, and examples from this lecture.",
"interview": "Extract the key quotes and main themes from this interview transcript."
}
system_prompt = prompts.get(analysis_type, prompts["meeting"])
response = ollama.chat(
model="llama3.1",
messages=[{
"role": "system",
"content": system_prompt
},
{"role": "user", "content": transcript["full_text"]}],
options={"temperature": 0.2, "num_ctx": 32768} # long context for transcripts
)
return {
"transcript": transcript["full_text"],
"analysis": response["message"]["content"],
"language": transcript["language"],
"duration_min": transcript["duration_s"] / 60
}
# Analyse a team meeting
result = transcribe_and_analyse("weekly_standup_2026-08-26.mp3", "meeting")
print(result["analysis"])Set num_ctx to 32768 or higher for long transcripts — a one-hour meeting at average speaking speed generates roughly 8,000-12,000 words of transcript, which exceeds the default 2048 context window.
Speaker Diarization: Who Said What
Whisper alone does not identify different speakers — it transcribes all speech into a single stream. For meetings or interviews where you need to know who said what, speaker diarization is a separate step. The pyannote.audio library provides speaker diarization that can be combined with Whisper timestamps:
pip install pyannote.audioPyannote requires a Hugging Face token and model access request (free). The combination of Whisper timestamps and pyannote speaker labels produces a transcript with speaker attribution. This pipeline is more complex to set up than Whisper alone but the output — a labelled transcript showing “Speaker 1: ” and “Speaker 2: ” — is substantially more useful for meeting analysis and interview transcription. Everything runs locally: the audio never leaves your machine.
Processing Speed and Hardware
Whisper processing speed depends heavily on whether you are using GPU or CPU. On NVIDIA GPU with faster-whisper: medium model processes at 4-8x realtime (a 30-minute podcast in 4-8 minutes). On Apple Silicon with Metal acceleration via whisper.cpp: similar speed to NVIDIA for the medium model. On CPU: 0.3-0.5x realtime with the medium model (a 30-minute podcast takes 60-100 minutes). For regular podcast or video processing, a GPU or Apple Silicon Mac is worthwhile for the time savings. For occasional transcription, CPU is perfectly adequate — start the process, do something else, and come back to the completed transcript.
Accuracy on Different Audio Types
Whisper’s accuracy varies significantly with audio quality and content type, and knowing what to expect prevents frustration. Clean studio-recorded podcasts with a single English-speaking host: large-v3 achieves near-perfect accuracy, medium achieves 95%+. Two-person conversation podcasts with clear audio: medium achieves 90-95%, with occasional errors on speaker transitions. Phone call recordings (compressed, often 8kHz): accuracy drops for all models — large-v3 is the only model that handles phone-quality audio well. Video conference recordings (Zoom, Teams, Google Meet): quality varies with each participant’s microphone; the model handles high-quality participants well and struggles with low-quality microphones. Background noise (coffee shop, conference room with ambient noise): vad_filter=True helps significantly by removing silent segments, but accuracy still degrades. Technical vocabulary and proper nouns: Whisper handles these better than older ASR systems but still occasionally produces phonetically similar wrong words for technical terms, names, and acronyms — these are the most common errors in domain-specific content and worth checking manually.
Subtitle and Caption Generation
The segment timestamps from Whisper can be formatted directly as SRT subtitle files, making local transcription useful for captioning video content:
def to_srt(segments: list[dict], output_path: str):
def fmt_time(seconds: float) -> str:
h = int(seconds // 3600)
m = int((seconds % 3600) // 60)
s = int(seconds % 60)
ms = int((seconds % 1) * 1000)
return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"
lines = []
for i, seg in enumerate(segments, 1):
lines.append(str(i))
lines.append(f"{fmt_time(seg['start'])} --> {fmt_time(seg['end'])}")
lines.append(seg["text"])
lines.append("")
from pathlib import Path
Path(output_path).write_text("\n".join(lines), encoding="utf-8")
# Generate subtitles from transcription
result = transcribe("video.mp4")
to_srt(result["segments"], "video.srt")This generates standard SRT subtitle files compatible with video players, video editors, and platforms like YouTube (which accepts SRT uploads for manual captions). For YouTube content creators, local Whisper transcription produces better auto-captions than YouTube’s built-in system for technical, accented, or domain-specific content, and keeps the transcript private during editing before publishing.
Batch Processing: Transcribing a Podcast Archive
For content creators, researchers, or journalists with a backlog of audio content to transcribe, a batch pipeline processes an entire archive overnight. Load the model once, process all files, save transcripts and metadata to a structured output:
from pathlib import Path
import json
def batch_transcribe(audio_dir: str, output_dir: str, model_size: str = "medium"):
model = WhisperModel(model_size, device="cuda", compute_type="float16")
Path(output_dir).mkdir(exist_ok=True)
audio_files = list(Path(audio_dir).glob("*.{mp3,wav,m4a,mp4,mkv}"))
for i, audio_path in enumerate(audio_files, 1):
out_name = audio_path.stem
txt_path = Path(output_dir) / f"{out_name}.txt"
if txt_path.exists():
print(f"[{i}/{len(audio_files)}] Skipping {audio_path.name} (already done)")
continue
print(f"[{i}/{len(audio_files)}] Transcribing {audio_path.name}...")
result = transcribe(str(audio_path))
txt_path.write_text(result["full_text"])
(Path(output_dir) / f"{out_name}.json").write_text(
json.dumps(result["segments"], indent=2)
)
batch_transcribe("./podcast_episodes", "./transcripts")The skip-if-exists check is important for large batches — if the process is interrupted, it picks up where it left off without reprocessing already-completed files. Transcripts and timed segment JSON are saved separately: the plain text for LLM analysis, the JSON for subtitle generation or time-aligned search.
Privacy for Audio Content
The same privacy considerations that apply to text and image processing apply to audio. Medical consultations, therapy sessions, legal proceedings, business strategy discussions, personal voice memos — audio content is often more sensitive than the equivalent text because it captures tone, emotion, and context alongside the words. Sending audio files to cloud transcription services creates privacy exposure that is qualitatively different from sending documents, because audio recordings of private conversations cannot be anonymised after the fact. Local Whisper transcription processes your audio on your hardware. The audio file never travels anywhere. This makes local transcription the appropriate choice for any audio content that contains sensitive information, and a simple default for all audio processing once you have the setup in place — there is no meaningful cost to transcribing locally rather than through a cloud service once the local pipeline is working.
Getting Started in 20 Minutes
The practical path from zero to working local transcription: install faster-whisper, download the medium model (it pulls automatically on first use), point it at one of your actual audio files, and run the basic transcribe function. Check the output quality against the audio — for podcast-quality recordings, it should be excellent with minimal errors. For noisier or lower-quality audio, test large-v3 to see whether the quality improvement justifies the slower speed. Once you have confirmed quality on your actual audio, add the Ollama analysis step for the use case you care most about — meeting summaries, podcast notes, or lecture study guides. The entire pipeline from raw audio to structured LLM analysis output runs locally, processes your most sensitive audio without any network transmission, and costs nothing per transcription after the initial setup. For anyone who regularly works with audio content, this is one of the most immediately useful local AI workflows available.