Multimodal local LLMs — models that accept both images and text — can read text from images with surprising accuracy. While they are not replacements for dedicated OCR engines on all tasks, they excel at extracting structured information from complex documents, understanding context around text, and handling layouts that traditional OCR struggles with. This guide covers how to use vision models in Ollama for OCR and text extraction, when they outperform traditional approaches, and practical Python workflows.
Vision Models in Ollama: What They Can Do
Several models available through Ollama accept image input alongside text prompts. The leading options for OCR and text extraction tasks:
LLaVA (1.5, 1.6): The original widely-used vision model in Ollama. Competent at general image understanding and text reading, though newer alternatives outperform it on structured document extraction.
Llama 3.2 Vision (11B, 90B): Significantly better than LLaVA on text extraction from images. The 11B version is the practical choice for most tasks — it reads printed text accurately, handles mixed text/image documents, and follows extraction instructions reliably. The 90B version is notably better on complex documents and small or degraded text.
Qwen2-VL (7B, 72B): Among the strongest vision models for OCR tasks, particularly on documents with structured layouts — tables, forms, multi-column text. The 7B version delivers impressive accuracy for its size.
Moondream2 (1.8B): Very small and fast, useful for simple text extraction from clean images where speed matters more than accuracy on complex layouts.
ollama pull llama3.2-vision:11b # recommended general purpose
ollama pull qwen2-vl:7b # strong on structured documents
ollama pull moondream2 # fast, for simple extractionsBasic Image Text Extraction
import ollama
import base64
from pathlib import Path
def extract_text_from_image(image_path: str, model: str = "llama3.2-vision:11b",
instructions: str = None) -> str:
# Read and encode image
image_data = Path(image_path).read_bytes()
b64_image = base64.b64encode(image_data).decode("utf-8")
prompt = instructions or "Extract all text from this image. Return only the extracted text, preserving the original structure and formatting as much as possible. Do not add any commentary or explanation."
response = ollama.chat(
model=model,
messages=[{
"role": "user",
"content": prompt,
"images": [b64_image]
}],
options={"temperature": 0, "num_predict": 2000}
)
return response["message"]["content"].strip()
# Extract text from a scanned document
text = extract_text_from_image("invoice.png")
print(text)
# Extract with specific instructions
receipt_items = extract_text_from_image(
"receipt.jpg",
instructions="Extract all line items from this receipt. Format as a list with item name and price on each line."
)Structured Data Extraction from Documents
Where vision LLMs genuinely outshine traditional OCR is structured extraction — pulling specific fields from documents rather than raw text. A traditional OCR pipeline extracts all text and then requires a separate parsing step to find the fields you care about. A vision LLM can do both in one step:
import json
def extract_invoice_data(image_path: str) -> dict:
image_data = base64.b64encode(Path(image_path).read_bytes()).decode("utf-8")
prompt = """Extract invoice data and return as JSON only. No other text.
{
"invoice_number": "",
"date": "",
"vendor_name": "",
"vendor_address": "",
"bill_to": "",
"line_items": [{"description": "", "quantity": 0, "unit_price": 0, "total": 0}],
"subtotal": 0,
"tax": 0,
"total": 0,
"payment_terms": ""
}"""
response = ollama.chat(
model="qwen2-vl:7b",
messages=[{"role": "user", "content": prompt, "images": [image_data]}],
options={"temperature": 0, "num_predict": 800}
)
raw = response["message"]["content"].strip()
if raw.startswith("```"):
raw = raw.split("\n", 1)[1].rsplit("```", 1)[0].strip()
return json.loads(raw)
# Also useful for: business cards, forms, ID documents (within legal limits),
# receipts, shipping labels, handwritten notesThe JSON extraction pattern works across document types. Define the schema for what you need from a document, and the vision model populates it. This approach handles variations in layout, formatting, and typography far more robustly than regex-based parsing of traditional OCR output.
Figure 1 — Vision LLM vs Traditional OCR: When to Use Each
Processing PDFs: Convert Pages to Images First
PDF files need to be converted to images before vision models can process them. The pdf2image library handles this cleanly:
from pdf2image import convert_from_path
import tempfile, os
def extract_pdf_text(pdf_path: str, model: str = "llama3.2-vision:11b") -> str:
pages = convert_from_path(pdf_path, dpi=200) # 200 DPI is good for text
all_text = []
for i, page in enumerate(pages):
# Save page as temp image
with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmp:
page.save(tmp.name, "PNG")
page_text = extract_text_from_image(tmp.name, model=model)
all_text.append(f"--- Page {i+1} ---\n{page_text}")
os.unlink(tmp.name)
return "\n\n".join(all_text)
# Process a scanned PDF
text = extract_pdf_text("scanned_contract.pdf")
print(text)200 DPI is the minimum for reliable text recognition; 300 DPI is better for small text or degraded documents. Higher DPI increases file size and processing time without quality gains for standard document text. For searchable PDFs (where text is already embedded), use a PDF text extraction library (pdfminer, pdfplumber) instead — vision models are unnecessary when text is already machine-readable.
When Vision LLMs Outperform Tesseract
Traditional OCR engines like Tesseract are fast and accurate on clean, high-resolution printed text — they are the right tool when you need to process thousands of documents quickly and the documents are standard printed forms. Vision LLMs outperform Tesseract in specific scenarios. Handwritten text: Vision LLMs handle handwriting significantly better than traditional OCR, which often produces garbled output. Low quality or degraded documents: Faded, skewed, or noisy scans that produce poor Tesseract output often yield reasonable results from a vision LLM because the model uses context to infer text it cannot fully see. Mixed layout documents: Documents mixing text, tables, and graphics that are difficult to parse structurally are processed holistically by vision LLMs. Context-sensitive extraction: When you need to understand what a field means to extract it correctly (finding “the billing address” rather than just “the second block of text”), vision LLMs apply semantic understanding that rule-based OCR post-processing cannot match. Non-standard formats: Business cards, receipts, handwritten notes, photographs of whiteboards — documents without a predictable structure where Tesseract struggles without careful tuning.
Batch Processing Images
For processing folders of images, the same batch pattern used for other local LLM tasks applies — with the important addition of error handling for images where extraction fails or produces garbled output:
import json
from pathlib import Path
def batch_extract(image_dir: str, output_path: str, model: str = "llama3.2-vision:11b"):
results = []
images = list(Path(image_dir).glob("*.{png,jpg,jpeg,webp}"))
for i, img_path in enumerate(images, 1):
print(f"Processing {i}/{len(images)}: {img_path.name}")
try:
text = extract_text_from_image(str(img_path), model=model)
results.append({"file": img_path.name, "text": text, "success": True})
except Exception as e:
results.append({"file": img_path.name, "text": "", "error": str(e), "success": False})
with open(output_path, "w") as f:
json.dump(results, f, indent=2)
success = sum(1 for r in results if r["success"])
print(f"Done: {success}/{len(images)} successful")Privacy: Why Local OCR Matters
Document OCR often involves the most sensitive content imaginable — financial statements, medical records, legal documents, identification documents. Sending images of these documents to cloud OCR APIs creates the same privacy concerns as sending text. A scanned passport photo, an image of a medical bill, or a photograph of a contract all contain sensitive information that is transmitted to third-party servers when processed by cloud vision APIs. Local vision models process these images entirely on your hardware. The document content never leaves your machine. For individuals, small businesses, and professional service firms handling sensitive client documents, local OCR via vision LLMs provides the same privacy guarantee for image-based content that local text LLMs provide for document content.
Performance and Hardware Requirements
Vision models require significantly more VRAM than text-only models of similar parameter counts, because they process image patches (often thousands of tokens per image) alongside the text prompt. Llama 3.2 Vision 11B requires approximately 8-10GB VRAM for inference. Qwen2-VL 7B requires 6-8GB. Processing time per image is 5-30 seconds depending on image size, model, and hardware — much slower than traditional OCR but fast enough for batch processing workloads that run overnight or in the background. On Apple Silicon, vision models use the same unified memory pool and Metal GPU acceleration as text models, with similar memory bandwidth advantages. An M4 Pro 24GB handles Llama 3.2 Vision 11B comfortably. An M4 Max 64GB+ can run Qwen2-VL 72B for maximum accuracy on complex documents. On NVIDIA, any GPU with 8GB+ VRAM runs the 7-11B vision models; 24GB+ is needed for 70B+ vision models that approach frontier accuracy.
Combining OCR with Text LLMs: A Two-Model Pipeline
For workflows that need both accurate text extraction and intelligent analysis of the extracted content, a two-model pipeline gives the best results: a vision model for extraction, a text model for analysis. Extract all text from the image with the vision model, then pass the extracted text to a capable text model for summarisation, classification, question answering, or any other analytical task. This separation is more reliable than asking a single vision model to both extract and analyse in one step, because the extraction model focuses on accurate transcription while the analysis model can be tuned for reasoning and output quality independently. It also lets you cache the extraction step — extract once, analyse many times with different queries without re-processing the image.
Getting Started: Fastest Path to Working Local OCR
The quickest setup: pull llama3.2-vision:11b from Ollama, copy the extract_text_from_image function from this guide, point it at a test image — ideally a printed document with clear text — and compare the output to what you see visually. If the extraction is accurate, try it on representative documents from your actual use case. For structured extraction (invoices, forms), switch to the JSON extraction pattern with a schema matching your document type and test it on five to ten real examples before committing to it in a production workflow. Vision models have quirks — they occasionally misread similar characters, invent text that is not present, or struggle with very small print — and testing on your actual documents before scaling is the only reliable way to understand which of these quirks affect your specific use case.
Accuracy Tips: Getting Better Extraction Results
A few prompt and workflow adjustments consistently improve vision LLM extraction accuracy. Image preprocessing helps significantly — ensure images are at least 150 DPI, crop tightly to the document removing excess whitespace and background, and convert to grayscale for pure text documents (colour information is irrelevant for OCR and can confuse the model with noise). For low-contrast images, increase contrast before processing. In your prompts, be specific about format: “return a markdown table” for tabular data, “return as a numbered list” for itemised documents, “preserve line breaks” for forms. For multi-page documents with consistent structure, extract one page at a time rather than combining pages — the model handles a single focused document better than a composite of multiple pages that may have slightly different layouts. For critical extractions, run the same image through two different models and compare — discrepancies highlight the fields most likely to contain errors, which you can then verify manually.