Transcribing and analysing video with AI combines two capabilities — speech-to-text for the audio track and vision or language understanding for the content — to produce searchable transcripts, summaries, topic extractions, and structured insights from video at scale. The tooling for this has matured significantly: Whisper handles transcription reliably, and frontier LLMs handle analysis of the resulting text. The challenge is engineering the pipeline to handle long videos, multiple speakers, and the variety of output formats that different use cases require.
The Two-Stage Architecture
Most video transcription and analysis pipelines follow a two-stage architecture. Stage one: extract the audio track from the video and run it through a speech-to-text model to produce a timestamped transcript. Stage two: send the transcript — or chunks of it — to an LLM for analysis. The analysis stage can produce summaries, extract key topics, identify action items, generate chapter markers, answer questions about the content, or output any structured format the downstream use case needs.
This separation matters. Running video directly through a multimodal model is possible for short clips, but for videos longer than a few minutes it is expensive, slow, and usually unnecessary — the analysis tasks that matter (summarisation, topic extraction, Q&A) work just as well from a high-quality transcript as from the raw video. Reserve full multimodal processing for tasks that genuinely require visual frames: detecting on-screen text, analysing presenter slides, or tracking visual events that are not described in speech.
Audio Extraction and Preprocessing
Extract the audio track from the video before transcription. FFmpeg is the standard tool for this and handles virtually any video format:
ffmpeg -i input.mp4 -vn -acodec pcm_s16le -ar 16000 -ac 1 output.wavThe flags here produce 16kHz mono WAV — the format Whisper expects. A few preprocessing considerations that improve transcription quality:
- Noise reduction — background noise (office ambient sound, recording artifacts) degrades Whisper accuracy noticeably on lower-quality recordings. A simple noise reduction pass with a tool like noisereduce or SoX before transcription helps on poor-quality audio.
- Volume normalisation — uneven volume across a recording (loud presenter, quiet Q&A) causes the model to miss quieter sections. Normalise to a consistent level before transcribing.
- Long file splitting — Whisper’s local models handle files up to about 25MB natively. For longer videos, split the audio into overlapping chunks (e.g. 10-minute segments with 30-second overlaps) to avoid cutting words at segment boundaries.
Figure 1 — Video transcription and analysis pipeline
Transcription with Whisper
OpenAI’s Whisper is the standard choice for transcription in custom pipelines — accurate across accents and languages, available as a local model (no API cost), and produces word-level timestamps that are useful for downstream processing. The key model size tradeoffs:
- whisper-large-v3 — highest accuracy, requires ~10GB VRAM, ~2x real-time on GPU. Best for production pipelines where accuracy matters most.
- whisper-medium — good accuracy, ~5GB VRAM, ~4x real-time. Good balance for most use cases.
- whisper-small / base — fast and lightweight, noticeably lower accuracy on challenging audio. Useful for rough drafts or when latency matters more than accuracy.
For cloud-based transcription without local hardware, the OpenAI Whisper API and AssemblyAI both produce high-quality results with speaker diarisation (separating who said what) available as an option. Speaker diarisation is worth adding when the transcript will be used for meeting notes, interview analysis, or any use case where knowing which speaker said what is important.
Chunking Long Transcripts for LLM Analysis
A one-hour video produces roughly 8,000–12,000 words of transcript — well within the context window of frontier models like Claude or GPT-4o. For most analysis tasks, send the full transcript in one request. For very long videos (several hours) or when using models with smaller context windows, chunk the transcript by time segments and either summarise each chunk independently or use a map-reduce pattern: summarise each chunk, then summarise the summaries.
When chunking, include overlapping context at chunk boundaries — the last paragraph of the previous chunk at the start of the next. This prevents analysis from missing content that spans chunk boundaries, which is particularly important for topic extraction where a topic might be introduced at the end of one chunk and developed in the next.
Analysis Prompts for Different Use Cases
The same transcript supports many different analysis outputs depending on how you prompt the model. A few effective patterns:
Meeting summary with action items: “This is a transcript of a team meeting. Produce: (1) a 3-sentence summary of what was discussed, (2) a bullet list of decisions made, (3) a bullet list of action items with the owner’s name where mentioned. Format as JSON.”
Chapter markers for a long video: “This is a transcript with timestamps. Identify the main topic changes and produce chapter markers in the format: [MM:SS] Topic title. Aim for 5–10 chapters.”
Key quote extraction: “Extract the 5 most quotable or insightful statements from this transcript. For each, include the quote, the approximate timestamp, and a one-sentence explanation of why it is significant.”
Question answering: “Using only the information in this transcript, answer the following question: [question]. If the transcript does not contain enough information to answer, say so explicitly.”
Adding Visual Analysis for Slide-Heavy Videos
For recorded presentations, webinars, or lectures where slides carry significant information, supplement transcript analysis with frame extraction. Extract a frame every 30–60 seconds, filter to frames where the slide has changed (compare frame hashes or use a simple pixel difference threshold), and send changed frames to a vision model to extract the slide text and visual content. Combine the slide content with the transcript for richer analysis — the model can now understand both what was said and what was shown at each point in the video.
Storing and Searching Transcripts
For pipelines processing many videos, storing transcripts with vector embeddings enables semantic search across the full corpus — finding all videos that discuss a specific topic, or retrieving the relevant segment of a specific video in response to a natural language query. Chunk the transcript into paragraph-sized segments, embed each chunk with a text embedding model, store in a vector database (Chroma, Qdrant, or pgvector), and query semantically. This transforms a collection of video files into a searchable knowledge base that is queryable in natural language — a high-value outcome for organisations with significant recorded meeting, training, or interview archives.
Video transcription and analysis pipelines have become straightforward to build with available open-source tools. Whisper handles transcription with production-quality accuracy; frontier LLMs handle analysis with a well-crafted prompt; the engineering challenge is the plumbing that makes the pipeline reliable at scale — audio preprocessing, chunking, retry handling, and structured output parsing. Get those pieces right and the result is a pipeline that turns unstructured video content into structured, searchable, actionable information.
Speaker Diarisation for Multi-Speaker Recordings
For meetings, interviews, and panel discussions, knowing which speaker said what transforms a raw transcript into a structured record. Speaker diarisation — the process of segmenting the transcript by speaker — can be added to the pipeline using pyannote.audio (open source, runs locally) or cloud services like AssemblyAI and Deepgram that include diarisation as a built-in option. The output labels each transcript segment with a speaker identifier (Speaker 1, Speaker 2, etc.) which the LLM analysis stage can then use to attribute statements, identify who raised each action item, or produce per-speaker summaries. Diarisation quality degrades with more than four or five simultaneous speakers, significant audio overlap, or poor recording quality — in these cases, the segment labels are a useful starting point for manual correction rather than a reliable automatic output.
Handling Multiple Languages
Whisper handles multilingual audio natively — it auto-detects the language and transcribes accordingly without any configuration. For pipelines that process content in multiple languages, this automatic detection is usually sufficient. For analysis, frontier LLMs handle transcripts in most major languages directly; you do not necessarily need to translate to English before analysis. However, for consistent downstream processing — especially when building searchable archives or feeding into structured data pipelines — translating all transcripts to a single language (typically English) at the transcription stage simplifies the analysis prompts and the data schema. Request translation by adding “translate to English” to the Whisper call: the model transcribes and translates in a single pass, adding minimal latency compared to a separate translation step.
Quality Evaluation and Pipeline Monitoring
Transcription accuracy varies significantly with audio quality, accent, domain-specific vocabulary, and background noise. Before deploying a pipeline to production, evaluate transcription quality on a sample of your actual content — not just clean speech benchmarks. Word error rate (WER) is the standard metric: the percentage of words in the reference transcript that differ from the model output. A WER under 5% is good for most use cases; under 10% is acceptable for rough summaries; above 15% means the audio quality or model configuration needs improvement before analysis results will be reliable. For ongoing monitoring, spot-check a random sample of transcripts weekly and track accuracy over time — changes in recording equipment, room acoustics, or speaker demographics can silently degrade pipeline quality without a monitoring process to catch it.