What Is OpenAI Whisper?
Whisper is OpenAI’s open-source automatic speech recognition (ASR) model, released in 2022 and continuously improved since. It transcribes audio to text across 97 languages with state-of-the-art accuracy, handles noisy environments and accented speech remarkably well, and can translate non-English audio directly to English text in a single step. For LLM applications that need to process spoken input — voice assistants, meeting transcription, call centre analysis, podcast processing — Whisper is the de facto standard in 2026.
Two deployment paths exist: the OpenAI Whisper API (managed, per-minute billing, no hardware required) and self-hosted Whisper (open-source model weights, free to run, requires a GPU for practical speed). The choice depends on your volume, latency requirements, data residency constraints, and operational preferences.
OpenAI Whisper API: Quickstart
pip install openaifrom openai import OpenAI
client = OpenAI()
# Transcribe an audio file
with open("meeting_recording.mp3", "rb") as audio_file:
transcript = client.audio.transcriptions.create(
model="whisper-1",
file=audio_file,
language="en", # Optional — auto-detected if omitted
response_format="text" # "text", "json", "srt", "vtt", or "verbose_json"
)
print(transcript) # Plain text transcription# With timestamps and word-level detail
with open("interview.mp3", "rb") as audio_file:
result = client.audio.transcriptions.create(
model="whisper-1",
file=audio_file,
response_format="verbose_json",
timestamp_granularities=["segment", "word"]
)
print(f"Language detected: {result.language}")
print(f"Duration: {result.duration:.1f}s")
for segment in result.segments:
print(f"[{segment.start:.1f}s - {segment.end:.1f}s] {segment.text}")The API accepts audio up to 25 MB. Supported formats: mp3, mp4, mpeg, mpga, m4a, wav, webm. Pricing is $0.006 per minute of audio — a 60-minute meeting costs $0.36 to transcribe.
Translation: Non-English Audio to English Text
# Transcribe AND translate to English in one call
with open("french_interview.mp3", "rb") as audio_file:
translation = client.audio.translations.create(
model="whisper-1",
file=audio_file
)
print(translation.text) # English text, regardless of input languageThe translation endpoint only outputs English. For transcription in the original language with a specific language hint, use the transcriptions endpoint with the language parameter set to the ISO 639-1 code (e.g., language="fr" for French).
Handling Long Audio Files
The API’s 25 MB file size limit means a 60-minute MP3 at 128kbps (~57 MB) exceeds the limit. Two approaches handle long audio: compress to a lower bitrate, or chunk the audio into segments and transcribe each separately:
from pydub import AudioSegment
import io, math
def transcribe_long_audio(file_path: str, chunk_minutes: int = 10) -> str:
audio = AudioSegment.from_file(file_path)
chunk_ms = chunk_minutes * 60 * 1000
n_chunks = math.ceil(len(audio) / chunk_ms)
transcripts = []
for i in range(n_chunks):
chunk = audio[i*chunk_ms:(i+1)*chunk_ms]
buffer = io.BytesIO()
chunk.export(buffer, format="mp3", bitrate="64k")
buffer.seek(0)
buffer.name = f"chunk_{i}.mp3" # API requires a filename
result = client.audio.transcriptions.create(
model="whisper-1", file=buffer
)
transcripts.append(result.text)
print(f"Transcribed chunk {i+1}/{n_chunks}")
return " ".join(transcripts)
full_transcript = transcribe_long_audio("2hr_meeting.mp3")When chunking, use overlapping segments (e.g., each chunk includes the last 2 seconds of the previous one) and then deduplicate the overlap in post-processing. This avoids words at chunk boundaries being cut off or duplicated.
Self-Hosted Whisper: Local Deployment
For data privacy, high volume, or offline use, run Whisper locally with OpenAI’s original library or the faster faster-whisper implementation:
pip install faster-whisperfrom faster_whisper import WhisperModel
# Model sizes: tiny, base, small, medium, large-v2, large-v3
# larger = more accurate but slower and more VRAM
model = WhisperModel("large-v3", device="cuda", compute_type="float16")
segments, info = model.transcribe("meeting.mp3", beam_size=5, language="en")
print(f"Language: {info.language} (probability: {info.language_probability:.2f})")
for segment in segments:
print(f"[{segment.start:.1f}s -> {segment.end:.1f}s] {segment.text}")faster-whisper uses CTranslate2 for inference and is 4–8x faster than the original OpenAI library with the same accuracy. On an RTX 4090, large-v3 transcribes a 60-minute recording in roughly 3–5 minutes. On CPU (no GPU), the tiny or base model is the practical choice — large-v3 on CPU takes much longer than real-time.
Model Size vs. Accuracy Trade-offs
Model | Params | VRAM | Rel speed | WER (English)
----------|---------|-------|-----------|---------------
tiny | 39M | ~1GB | ~32x | ~11%
base | 74M | ~1GB | ~16x | ~7%
small | 244M | ~2GB | ~6x | ~4%
medium | 769M | ~5GB | ~2x | ~3%
large-v3 | 1550M | ~10GB | ~1x | ~2.5%For clear, single-speaker English audio in quiet environments, the small or medium model delivers excellent accuracy at much lower compute cost. Reserve large-v3 for challenging audio: noisy environments, heavy accents, technical vocabulary, multiple overlapping speakers, or non-English languages where the accuracy gap between large and smaller models is more pronounced.
Diarisation: Who Said What
Whisper transcribes speech but does not identify individual speakers (diarisation). Combine Whisper with pyannote.audio for speaker-attributed transcripts:
from pyannote.audio import Pipeline
from faster_whisper import WhisperModel
# Diarisation pipeline (requires HuggingFace token)
diarisation = Pipeline.from_pretrained(
"pyannote/speaker-diarization-3.1",
use_auth_token="your-hf-token"
)
# Transcription
whisper = WhisperModel("medium", device="cuda")
segments, _ = whisper.transcribe("meeting.mp3")
transcription = [(s.start, s.end, s.text) for s in segments]
# Diarise
diarisation_result = diarisation("meeting.mp3")
# Align speaker labels with transcript segments
def assign_speakers(transcript, diarisation):
output = []
for start, end, text in transcript:
midpoint = (start + end) / 2
speaker = "UNKNOWN"
for turn, _, label in diarisation.itertracks(yield_label=True):
if turn.start <= midpoint <= turn.end:
speaker = label
break
output.append({"speaker": speaker, "start": start, "end": end, "text": text})
return outputWhisper in LLM Pipelines
Whisper is the audio input layer for voice-enabled LLM applications. The standard pipeline: record audio from the user's microphone, transcribe with Whisper, send the transcript to Claude or GPT-4o, stream the text response back, optionally convert to speech with a TTS model. For real-time voice applications, use Whisper's streaming-compatible mode with smaller models for lower latency, or OpenAI's new Realtime API which handles audio-to-audio directly without explicit transcription. For batch processing of recorded audio — call centre recordings, meeting notes, podcast transcripts — faster-whisper with large-v3 on GPU delivers production-quality transcription at low per-minute cost.
Best Practices for Production Transcription
Several practices reliably improve transcription quality. Audio preprocessing: normalise volume, reduce background noise with a noise reduction filter (the noisereduce library works well), and trim silence from the beginning and end. Language hints: always specify the language if known — auto-detection adds latency and occasionally misidentifies similar languages. Prompt conditioning: for domain-specific vocabulary (medical terms, company names, product codes), pass a short prompt with key terms to the API's prompt parameter — Whisper uses it to bias towards expected vocabulary. Post-processing: run the transcript through an LLM for punctuation correction, paragraph segmentation, and named entity normalisation before storing or displaying. Raw Whisper output has inconsistent punctuation and capitalisation that simple LLM post-processing cleans up effectively at minimal cost.
Cost Comparison: API vs. Self-Hosted
The OpenAI Whisper API costs $0.006 per minute — a 10-hour call centre recording day costs $3.60 to transcribe. At higher volumes — 1,000 hours per month — API costs reach $360/month. Self-hosting Whisper on a single GPU server (RTX 4090, roughly $0.15/hour on spot cloud instances) and processing 1,000 hours of audio at 4x realtime speed with large-v3 takes approximately 250 GPU-hours, costing around $37.50/month in compute — about 10x cheaper than the API. The break-even between API and self-hosted is roughly 200–300 hours of audio per month, depending on your compute costs and operational overhead. Below that volume, the API is cheaper and simpler. Above it, self-hosting pays for itself within the first month.
Handling Multilingual Audio
Whisper large-v3 supports 97 languages with varying accuracy. Performance is strongest on English, Spanish, French, German, Portuguese, Italian, Japanese, Korean, and Chinese — languages well-represented in its training data. For less common languages, accuracy degrades and alternative specialised models may outperform Whisper. Always specify the language code explicitly rather than relying on auto-detection for production systems — auto-detection adds 1–2 seconds of latency and occasionally misidentifies closely related languages. For mixed-language audio where speakers switch between languages mid-conversation, segment detection is less reliable and you may get better results processing each speaker's segments separately once diarisation has been applied.
Whisper vs. Other ASR Options in 2026
Whisper remains the default choice for most use cases, but alternatives exist for specific requirements. Google Speech-to-Text v2 supports streaming transcription (Whisper does not) and integrates naturally with Google Cloud infrastructure — the right choice for real-time streaming applications. AWS Transcribe is the equivalent for AWS-native applications, with strong integration into Bedrock and Lambda workflows. AssemblyAI and Deepgram offer managed ASR APIs with features Whisper lacks: real-time streaming, speaker diarisation, sentiment analysis, and topic detection as managed services. For applications where transcription is the core feature rather than a supporting component, these specialised providers offer more capability out of the box. For applications where transcription is one step in a larger LLM pipeline and you want minimal vendor dependencies, Whisper's combination of open-source availability, strong multilingual accuracy, and simple API makes it the right default.
Whisper with Real-Time Audio Input
While the Whisper API and standard faster-whisper library operate on complete audio files, real-time transcription from a microphone stream is possible using faster-whisper's streaming capabilities with a rolling audio buffer. The pattern: capture audio in 3–5 second chunks, transcribe each chunk, and append results to a running transcript. This produces near-real-time transcription with a small delay equal to your chunk size. For truly low-latency applications (sub-second), OpenAI's Realtime API handles audio-to-audio processing without explicit transcription, or specialised streaming ASR services like Google Speech-to-Text streaming mode deliver 200–400ms latency at the cost of higher per-minute pricing. For most practical applications — meeting transcription, voice notes, dictation — faster-whisper with 3-second chunks produces accurate transcripts at acceptable latency without requiring a streaming ASR service subscription.
Integrating Whisper with Claude for Meeting Intelligence
A complete meeting intelligence pipeline combines Whisper transcription with Claude analysis. Transcribe the meeting audio with Whisper (including speaker diarisation if available), then send the full transcript to Claude with a structured analysis prompt: identify action items with owners and deadlines, summarise decisions made, highlight open questions, and extract key discussion points by topic. Claude handles meetings up to several hours in its 200K token context window — a 2-hour meeting transcript is typically 15,000–25,000 words, well within context. The output is a structured meeting summary that would take a human 30–60 minutes to produce manually, delivered in under 2 minutes of combined transcription and analysis time. This pipeline represents one of the highest-value, most immediately demonstrable LLM applications available — the input (recorded audio) and output (structured meeting notes) are both familiar to business users, making it easy to evaluate and easy to deploy.