How to Use LLaVA with Ollama for Local Image Analysis

LLaVA (Large Language and Vision Assistant) was one of the first open-source multimodal models that could meaningfully analyse images locally. You give it an image alongside a text prompt and it responds based on what it sees. While newer models like Gemma 3 have surpassed it on benchmarks, LLaVA remains widely used and is one of the simplest ways to add local image understanding to any project. With Ollama, it’s a single pull command away.

This guide covers the available LLaVA variants, how to use image analysis from the CLI and API, and what it’s good at in practice.

LLaVA Variants in the Ollama Library

Several LLaVA versions are available, built on different base models:

ollama pull llava           # LLaVA 1.6 7B — default, best balance
ollama pull llava:13b       # LLaVA 1.6 13B — better quality, more memory
ollama pull llava:34b       # LLaVA 1.6 34B — highest quality, needs 24GB+
ollama pull llava-llama3    # LLaVA on Llama 3 — improved text quality
ollama pull llava-phi3      # LLaVA on Phi-3 — small, fast, capable

The default llava (7B, ~4.7GB) is the right starting point for most people. llava-phi3 is worth trying if you want faster inference on limited hardware — it’s smaller but still produces good image descriptions. llava-llama3 gives noticeably better text output quality since it’s built on Llama 3’s stronger language foundation.

Running LLaVA from the CLI

Start an interactive session with LLaVA:

ollama run llava

Once loaded, you can pass an image path directly in the prompt using the image attachment syntax:

>> Describe this image [/path/to/your/image.jpg]

Or from the command line directly:

ollama run llava "What is in this image?" --image /path/to/image.jpg

LLaVA accepts JPEG, PNG, and WebP formats. The image is resized internally — you don’t need to pre-process it. For most practical uses (screenshots, photos, diagrams), the image can be passed as-is.

Using LLaVA via the Python API

The Ollama Python library handles image encoding automatically:

import ollama

response = ollama.chat(
    model='llava',
    messages=[{
        'role': 'user',
        'content': 'Describe what you see in this image in detail.',
        'images': ['/path/to/image.jpg']
    }]
)
print(response['message']['content'])

For images loaded from memory (e.g. downloaded from a URL or generated by another process), pass them as base64-encoded bytes:

import ollama
import base64
from pathlib import Path

image_data = base64.b64encode(Path('image.jpg').read_bytes()).decode()

response = ollama.chat(
    model='llava',
    messages=[{
        'role': 'user',
        'content': 'What text can you read in this image?',
        'images': [image_data]
    }]
)
print(response['message']['content'])

What LLaVA Is Good At

LLaVA handles a useful range of visual understanding tasks:

Scene description. Describe the contents of a photo, identify objects, describe the setting. Works well for most everyday photos — outdoors, interiors, crowds, products.

Text in images (OCR-style). LLaVA can read text visible in images — signs, labels, screenshots, documents. It’s not a dedicated OCR tool and won’t preserve exact formatting, but for extracting text content from photos it works surprisingly well.

Screenshot analysis. Paste a UI screenshot and ask what’s on the screen, what buttons are visible, what the error message says. Useful for automated UI analysis pipelines.

Chart and diagram reading. Basic charts and graphs — bar charts, line graphs, simple diagrams. It can describe what the chart shows and extract approximate values, though it struggles with complex or dense charts.

Document parsing. Photos of printed documents, whiteboards, handwritten notes. The quality varies — clean printed text works well, handwriting is hit-or-miss.

Building an Image Analysis Pipeline

LLaVA’s real value shows when you build it into a pipeline rather than using it interactively. Here are a few practical patterns:

Batch image description:

import ollama
from pathlib import Path

images = list(Path('photos').glob('*.jpg'))
descriptions = []

for img_path in images:
    response = ollama.chat(
        model='llava',
        messages=[{
            'role': 'user',
            'content': 'Describe this image in one sentence.',
            'images': [str(img_path)]
        }]
    )
    descriptions.append({
        'file': img_path.name,
        'description': response['message']['content']
    })
    print(f"{img_path.name}: {response['message']['content']}")

Screenshot-to-text extractor:

import ollama

def extract_text_from_screenshot(image_path: str) -> str:
    response = ollama.chat(
        model='llava',
        messages=[{
            'role': 'user',
            'content': 'Extract all text visible in this image. Return only the text, no descriptions.',
            'images': [image_path]
        }]
    )
    return response['message']['content']

text = extract_text_from_screenshot('screenshot.png')
print(text)

Image Q&A with follow-up questions:

import ollama

messages = [{
    'role': 'user',
    'content': 'What do you see in this image?',
    'images': ['photo.jpg']
}]

# First question
response = ollama.chat(model='llava', messages=messages)
print('Initial:', response['message']['content'])

# Follow-up (image context carries through conversation)
messages.append({'role': 'assistant', 'content': response['message']['content']})
messages.append({'role': 'user', 'content': 'How many people are in the image?'})
response2 = ollama.chat(model='llava', messages=messages)
print('Follow-up:', response2['message']['content'])

LLaVA vs Newer Vision Models

LLaVA is mature and reliable, but newer vision models have overtaken it in several areas. If you’re starting a new project and care about vision quality, it’s worth knowing where alternatives have an edge:

Gemma 3 4B/12B: Better overall quality, stronger text generation, native multimodal from Google. The 4B is smaller than LLaVA 7B and better on most benchmarks. If you can run LLaVA 7B, you can run Gemma 3 4B — and the quality is better. For new projects, Gemma 3 is the better choice.

Qwen-VL / Qwen 2.5-VL: Exceptional for document parsing, table extraction, and Chinese-language image content. Significantly better at reading dense text in images. Available via ollama pull qwen2.5vl.

Moondream: An extremely small vision model (1.9B) optimised for edge deployment. Faster than LLaVA on CPU-only setups. Available as ollama pull moondream and worth trying on resource-constrained hardware.

Where LLaVA still has an edge: it’s battle-tested, has extensive community tooling built around it, and is deeply integrated into projects like Open WebUI where it was an early default. If you’re working with an existing codebase or tutorial that uses LLaVA, sticking with it is perfectly reasonable. If starting fresh, try Gemma 3 4B first.

Open WebUI Integration

If you’re using Open WebUI as your Ollama frontend, adding LLaVA or any vision model is automatic — just select the model from the dropdown and an image attachment button appears in the chat interface. You can drag and drop images directly into the chat window and ask questions about them in the same turn. Open WebUI handles the image encoding and API call transparently.

For production pipelines where you need to process many images, Open WebUI’s pipeline feature (available in newer versions) lets you build multi-step workflows that combine image analysis with follow-on text processing — useful for document extraction, automated reporting, or content moderation workflows that need both visual and textual understanding in sequence. LLaVA’s API consistency with the Ollama standard means any integration that works with text models also works with LLaVA with a simple model name change, which is part of what makes it a practical choice for production work despite its age.

Prompt Engineering for Better Vision Results

LLaVA responds better to specific, directed prompts than open-ended ones. Instead of “describe this image,” try “list every object you can see in this image” or “read all text visible in this screenshot, preserving the layout as best you can.” For chart analysis: “this is a bar chart — what does each bar represent and approximately what is its value?” Breaking the task into specific sub-questions consistently produces more accurate and complete outputs than asking for a general description. When asking about specific regions of an image, describe them spatially: “what is in the top-right corner of this image?” or “what does the label on the bottle say?” LLaVA has decent spatial reasoning for a model its age — it can follow directional instructions and focus on specific parts of an image when prompted correctly. For multi-turn conversations, keep your follow-up questions grounded in what it told you in the previous turn rather than introducing entirely new concepts — this maintains coherence and reduces hallucination where the model invents details it can’t actually see.

Troubleshooting LLaVA Issues

The most common LLaVA issue is the model producing generic descriptions that ignore the specific content of your image — often a sign that the image wasn’t processed correctly. Check that you’re passing the image path correctly and that the file exists. For the Python API, confirm the image path is a string and the file is readable. If descriptions are consistently too generic, try increasing the context length (num_ctx 4096 or higher) and being more specific in your prompt about what you want described. If the model loads but gives strange outputs when images are provided, check that you’re using a vision-capable variant — some models in the Ollama library have vision tags in their names but the default pull may not include the vision weights. Running ollama show llava and looking at the capabilities field confirms whether vision is active. On machines with limited memory, LLaVA can sometimes crash when processing large images — try resizing your input image to 1024×1024 or smaller before passing it to the model, which reduces memory pressure during the vision encoding step without significantly impacting output quality for most tasks.

LLaVA for Automated Document Processing

One of the most practical use cases for LLaVA is processing photos of physical documents — forms, receipts, invoices, ID documents, whiteboards. The workflow combines image capture (phone photo or scanner), LLaVA for text and structure extraction, and a follow-on LLM call for parsing and structuring the extracted content. A receipt photo pipeline might look like: capture image → LLaVA extracts all text and prices → structured LLM call converts to JSON with line items and totals. This kind of pipeline is genuinely useful for expense management, document digitisation, and data extraction from paper records — and it all runs locally with no data sent to external services, which matters for any documents containing personal or sensitive information. LLaVA’s OCR quality on clean printed text is good enough for production use on many document types, and for cases where precision is critical, pairing it with a dedicated OCR tool like Tesseract for the text extraction step while using LLaVA for structural understanding (what kind of document is this? where is the total? what are the line items?) gives better results than either tool alone.

The combination of local processing, reasonable quality, and a well-documented API makes LLaVA a practical building block for any project that needs to understand images without sending them to a cloud service — and despite being superseded on benchmarks by newer models, that practical reliability is why it remains one of the most widely used vision models in the Ollama ecosystem.