What Is Multimodal RAG?
Standard RAG pipelines process text: documents are chunked, embedded as vectors, and retrieved based on semantic similarity to a text query. Multimodal RAG extends this to handle images, diagrams, charts, PDFs with mixed content, and other non-text data alongside text. The motivation is practical — most real enterprise documents are not pure text. A technical manual contains diagrams. A financial report contains charts and tables. A product catalogue has images. A regulatory filing mixes dense prose with structured tables. Text-only RAG over these documents discards the visual content entirely, which often contains the most important information for specific queries.
Multimodal RAG systems can handle these mixed-content documents by either converting visual content to text (via OCR, image captioning, or layout-aware parsing) before indexing, or by storing and retrieving images directly and passing them to a vision-capable LLM at query time. Each approach has different trade-offs in pipeline complexity, retrieval accuracy, and the types of visual content handled well.
Three Approaches to Multimodal RAG
There is no single standard architecture for multimodal RAG — three main approaches each suit different use cases.
Approach 1: Convert everything to text first. Extract text from documents using layout-aware parsers that preserve structure, generate captions for images using a vision model, convert tables to markdown, and then treat the whole corpus as a text RAG problem. This is the simplest approach because it requires no changes to the retrieval layer — any vector database and embedding model can handle text. The weakness is that auto-generated captions for complex diagrams or technical charts often miss important details that a human or the original document author would convey.
Approach 2: Multi-vector retrieval with image embeddings. Embed text and images separately using their respective embedding models. Store both in the vector database with shared metadata linking them to the same source document. At query time, embed the query as both text and (if it contains visual references) an image query, retrieve from both indices, and pass the top-K results from each to a vision-capable LLM that can reason over both text and images together. More complex to build but preserves visual fidelity better than captioning alone.
Approach 3: Late interaction with ColPali. ColPali is a recent approach that embeds entire document pages as visual objects — no OCR or text extraction required. Pages are treated as images and indexed using a vision-language model that understands both the visual layout and any text present. Retrieval is based on page-level visual similarity to the query. This is the most capable approach for complex mixed-layout documents but requires specialised infrastructure and is computationally heavier than text-only RAG.
PDF Parsing for Mixed-Content Documents
PDFs are the dominant format for multimodal RAG and also the most challenging. A PDF can contain flowing text, embedded images, scanned pages (image-only), tables, mathematical formulas, headers and footers, and complex multi-column layouts. Choosing the right parser is critical.
PyMuPDF (fitz) is fast and handles standard PDFs well, extracting text with layout information and saving embedded images for separate processing. It struggles with scanned PDFs where all content is images.
Unstructured.io is a more sophisticated parser that handles PDFs, Word documents, PowerPoints, HTML, and images. It uses a combination of OCR (via Tesseract or cloud services), table detection, and layout analysis to extract structured content from complex documents. Its “hi_res” strategy with a document layout detection model produces much better results than naive text extraction for documents with complex layouts.
Azure Document Intelligence (formerly Form Recognizer) is a cloud-based service that provides state-of-the-art layout-aware extraction including table structure, figure detection with captions, and reading order normalisation. For enterprise documents where extraction quality is critical, it is the strongest option — though it is a managed cloud service with per-page pricing rather than a local library.
LlamaParse from LlamaIndex parses PDFs with LLM assistance, producing clean markdown output that preserves tables and figures better than rule-based parsers. Useful for documents that other parsers mangle.
Image Captioning for Visual Content
Once images are extracted from documents, generating high-quality captions is the key to making them retrievable in a text-based RAG system. The caption becomes the “text representation” of the image that gets embedded and retrieved.
For technical diagrams, architecture charts, and domain-specific visuals, a good captioning prompt matters enormously. Generic captions like “A diagram showing a process flow” are not useful for retrieval. Effective captions for technical content describe the specific components shown, the relationships between them, the labels visible in the diagram, and what the diagram is illustrating in the context of the surrounding document. Providing the surrounding text context to the vision model alongside the image dramatically improves caption quality — the model can use the document context to understand what the diagram is representing.
Claude, GPT-4o, and Gemini Flash all support image input and can generate detailed, contextually aware captions. Gemini Flash’s low cost ($0.10 per million tokens with image processing included) makes it practical for captioning large corpora at scale. For a document set with 10,000 images, Gemini Flash captioning costs roughly $2–5 depending on image sizes — negligible for the quality improvement it provides.
Table Extraction and Representation
Tables in documents are a particularly valuable source of structured data for RAG, and also particularly easy to handle badly. Extracting a table as flat text loses the row-column structure that gives the data meaning. A table comparing product specifications across five vendors, extracted naively, might produce a sequence of numbers with no indication of what row or column each belongs to.
Better approaches preserve table structure. Extracting tables as markdown preserves the visual structure in a format that LLMs understand well. Extracting tables as JSON or CSV preserves structure for programmatic processing. For very large or complex tables, extracting them as images and captioning with a vision model may preserve more meaning than any text extraction approach. Tools like Unstructured.io’s table extraction, Camelot (for PDF tables specifically), and Azure Document Intelligence’s table detection all handle table structure better than generic text extraction.
At retrieval time, tables often need special handling. A user asking “what is the price of product X?” may retrieve a table chunk that contains the answer, but if the chunk is a raw markdown table, the LLM needs to understand that the answer is in the intersection of the “Price” column and the “Product X” row — which modern LLMs handle well. The more problematic case is when the chunk is too small to include the table headers, making the column and row context ambiguous. Always chunk tables as complete units rather than splitting mid-table, even if this produces chunks larger than your standard chunk size.
Multimodal Embedding Models
For retrieval that goes beyond text — where you want images themselves (not just their captions) to be retrieved based on a text query or another image — multimodal embedding models create a shared vector space where text and images can be compared directly. The leading options in 2026 are CLIP and its successors (OpenCLIP, EVA-CLIP), which produce comparable embeddings for images and text, enabling cross-modal retrieval.
The practical limitation of CLIP-based retrieval for document RAG is that CLIP embeddings capture visual similarity and high-level semantic content well, but struggle with text-heavy images like slides, charts with labels, and documents. An image of a slide about “transformer attention mechanisms” will have similar CLIP embeddings to other technology-related images, but not necessarily to text queries about attention mechanisms. For document images, caption-based retrieval (embed the caption, retrieve the image) often outperforms direct image embedding retrieval because captions capture the semantic content in a form that text embedding models handle well.
Retrieval Strategy for Multimodal Corpora
For a corpus containing mixed text and images, a practical retrieval strategy combines multiple sources. Retrieve text chunks based on semantic similarity to the query embedding. Retrieve image captions whose embeddings are similar to the query. Use metadata filtering to bias retrieval toward the same source document when text chunks and image captions are from the same page. Rerank the combined results using a cross-encoder or LLM-based reranker that considers both text and image relevance to the query.
The reranking step is particularly important for multimodal retrieval because the initial embedding-based retrieval from different modalities (text chunks vs. image captions vs. table text) produces results at different effective similarity scales. A cross-encoder that jointly scores each candidate against the query is more effective than trying to normalise similarity scores across modalities.
Passing Multimodal Context to the LLM
At generation time, a vision-capable LLM receives both text chunks and images as context. The query is answered using all available context — the model can reason over a retrieved diagram alongside the text that describes it, or compare a retrieved chart to a retrieved table from a different section of the document. This joint reasoning over retrieved visual and textual context is the core value proposition of multimodal RAG over text-only RAG.
Constructing the multimodal context for the LLM requires care. Order retrieved items by relevance and interleave images with their associated text for maximum coherence. Keep the total context within the model’s context window — images consume tokens proportional to their resolution. Gemini Flash’s 1M token context window makes it practical to include many images in a single call; Claude’s 200K context and GPT-4o’s 128K are more constrained. For very large retrieval sets, summarise or compress lower-relevance items before including them.
When Multimodal RAG Is Worth the Complexity
Standard text RAG handles the majority of enterprise document use cases adequately. Multimodal RAG is worth the additional complexity when: the documents contain charts, diagrams, or figures that are central to answering common queries; when tables with complex structure are frequently referenced; when documents are scanned PDFs where OCR quality is insufficient for reliable text extraction; or when users frequently ask questions whose answers are more precisely expressed in visual form than in text. For document corpora that are primarily running text with occasional simple images, the marginal quality improvement from multimodal RAG often does not justify the additional pipeline complexity. Start with high-quality text extraction and measure whether visual content is a bottleneck before adding multimodal retrieval infrastructure.