How to Use Ollama with LlamaIndex for Local RAG

LlamaIndex is one of the most capable frameworks for building RAG systems — retrieval-augmented generation pipelines that let your LLM answer questions grounded in your own documents. Connecting it to Ollama means you get the full LlamaIndex feature set (document loading, chunking, vector indexing, query engines, agents) backed by a local model with no API costs and no data leaving your machine. This guide walks through a complete working setup: ingesting documents, building a vector index, and querying it with a local Ollama model.

Installation

pip install llama-index llama-index-llms-ollama llama-index-embeddings-ollama

LlamaIndex uses a modular architecture — the core package plus specific integrations. The two Ollama packages handle LLM inference and embeddings respectively. You’ll also want a vector store; for local development, the built-in in-memory store is fine. For persistent storage, ChromaDB is a clean option:

pip install llama-index-vector-stores-chroma chromadb

Basic Setup: LLM and Embeddings

Before building a RAG pipeline, configure LlamaIndex to use your Ollama models:

from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.core import Settings

# Set the LLM
Settings.llm = Ollama(
    model='llama3.2',
    request_timeout=120.0,  # seconds — increase for slower hardware
    base_url='http://localhost:11434'
)

# Set the embedding model
Settings.embed_model = OllamaEmbedding(
    model_name='nomic-embed-text',
    base_url='http://localhost:11434'
)

# Optional: set context window to match your Ollama num_ctx setting
Settings.context_window = 8192
Settings.num_output = 512

These settings apply globally — every LlamaIndex component will use them by default. The embedding model is separate from the LLM; use a dedicated embedding model like nomic-embed-text rather than a general-purpose model for better retrieval quality. Pull it first:

ollama pull nomic-embed-text

Loading Documents and Building an Index

LlamaIndex can load documents from many sources. The simplest is a directory of text or PDF files:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Load all documents from a directory
documents = SimpleDirectoryReader('my_docs/').load_data()
print(f"Loaded {len(documents)} documents")

# Build a vector index — this chunks documents, generates embeddings, and stores them
index = VectorStoreIndex.from_documents(documents, show_progress=True)

# Create a query engine
query_engine = index.as_query_engine(similarity_top_k=3)

# Ask a question
response = query_engine.query("What are the main conclusions in these documents?")
print(response)

The first time you run this, LlamaIndex chunks your documents, calls Ollama’s embeddings API for each chunk, and builds an index in memory. For a directory of 50–100 documents this takes a few minutes. Subsequent queries are fast — the retrieval step finds the most relevant chunks and the LLM synthesises an answer from them.

Figure 1 — LlamaIndex + Ollama RAG Pipeline

Persisting the Index with ChromaDB

Building the index from scratch every time you run your script is slow for large document sets. Persist it to disk with ChromaDB:

import chromadb
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.core import Settings

Settings.llm = Ollama(model='llama3.2', request_timeout=120.0)
Settings.embed_model = OllamaEmbedding(model_name='nomic-embed-text')

# Persistent ChromaDB client
chroma_client = chromadb.PersistentClient(path='./chroma_db')
chroma_collection = chroma_client.get_or_create_collection('my_docs')
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# First run: build and persist the index
documents = SimpleDirectoryReader('my_docs/').load_data()
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    show_progress=True
)
print("Index built and persisted.")

# Subsequent runs: load from disk (no re-embedding)
index = VectorStoreIndex.from_vector_store(
    vector_store,
    storage_context=storage_context
)
query_engine = index.as_query_engine(similarity_top_k=5)
response = query_engine.query("Summarise the key points across all documents.")
print(response)

After the first run, the ChromaDB files on disk contain all your embeddings. Subsequent runs skip the embedding step entirely and load directly from the persisted store — which makes startup near-instant even for large document collections. The get_or_create_collection call handles both cases: creates a new collection on first run, loads the existing one thereafter.

Controlling Chunking

How you chunk documents significantly affects retrieval quality. LlamaIndex’s default is 1024-token chunks with 20-token overlap — sensible for most use cases, but worth tuning for specific document types:

from llama_index.core.node_parser import SentenceSplitter

# Custom chunking: smaller chunks for dense technical docs
Settings.node_parser = SentenceSplitter(
    chunk_size=512,      # tokens per chunk
    chunk_overlap=50     # overlap between consecutive chunks
)

# Or semantic chunking (groups related sentences)
from llama_index.core.node_parser import SemanticSplitterNodeParser
Settings.node_parser = SemanticSplitterNodeParser(
    embed_model=Settings.embed_model,
    breakpoint_percentile_threshold=95
)

Smaller chunks (256–512 tokens) give more precise retrieval but lose context within each chunk. Larger chunks (1024–2048) preserve more context but can retrieve irrelevant content alongside the relevant section. For most document Q&A, 512 tokens with 50-token overlap is a good default. Semantic chunking groups sentences by meaning rather than character count — it produces more coherent chunks but takes longer to process.

Chat Engine: Conversational RAG

The query engine gives you single-turn Q&A. For a conversational interface that remembers previous questions about your documents, use the chat engine:

from llama_index.core.memory import ChatMemoryBuffer

chat_engine = index.as_chat_engine(
    chat_mode='condense_plus_context',
    memory=ChatMemoryBuffer.from_defaults(token_limit=4096),
    verbose=True
)

# First question
response = chat_engine.chat("What are the main topics covered?")
print(response)

# Follow-up — uses conversation history
response = chat_engine.chat("Which of those topics has the most detail?")
print(response)

# Reset conversation
chat_engine.reset()

The condense_plus_context mode rewrites follow-up questions to be standalone (removing references to “it”, “that”, “the previous point”) before retrieval — which significantly improves retrieval accuracy for conversational queries that reference earlier messages.

Figure 2 — LlamaIndex Query Modes Compared

Loading Different Document Types

LlamaIndex’s SimpleDirectoryReader automatically handles multiple file types. For more specific loaders:

from llama_index.core import Document
from llama_index.readers.web import SimpleWebPageReader

# Load from URLs
web_docs = SimpleWebPageReader(html_to_text=True).load_data([
    'https://docs.python.org/3/tutorial/index.html',
    'https://realpython.com/python-basics/'
])

# Load from a string directly
custom_doc = Document(
    text="Your custom text here...",
    metadata={'source': 'manual', 'topic': 'LLMs'}
)

# Combine sources
all_docs = web_docs + [custom_doc]
index = VectorStoreIndex.from_documents(all_docs)

The metadata dictionary on each document is passed through to retrieved chunks and can be used to filter retrieval — useful when you want to query only documents from a specific source or category.

Evaluating RAG Quality

LlamaIndex includes evaluation utilities to measure how well your RAG pipeline is actually working:

from llama_index.core.evaluation import FaithfulnessEvaluator, RelevancyEvaluator

faithfulness_evaluator = FaithfulnessEvaluator()
relevancy_evaluator = RelevancyEvaluator()

query = "What are the main advantages of transformer models?"
response = query_engine.query(query)

# Check if the response is grounded in the retrieved context
faith_result = faithfulness_evaluator.evaluate_response(response=response)
print(f"Faithful: {faith_result.passing} (score: {faith_result.score:.2f})")

# Check if the retrieved context is relevant to the query
rel_result = relevancy_evaluator.evaluate_response(query=query, response=response)
print(f"Relevant: {rel_result.passing} (score: {rel_result.score:.2f})")

These evaluators use the LLM to judge quality — the faithfulness evaluator asks whether the response is supported by the retrieved context, and the relevancy evaluator asks whether the context is relevant to the query. Running evaluations on a sample of queries helps you tune your chunk size, similarity_top_k, and retrieval strategy before deploying a RAG system to real users. LlamaIndex with Ollama gives you a fully local evaluation pipeline — the judge LLM can be the same model you’re using for generation, keeping everything on your hardware with no external API calls needed.

Choosing the Right LLM for Your RAG Pipeline

The choice of Ollama model for LlamaIndex RAG matters more than it might seem. For generation quality — synthesising an answer from retrieved chunks — a general-purpose instruction model like Llama 3.1 8B or Mistral 7B works well for most use cases. For tasks where you specifically need the answer grounded tightly in the provided context (and the model to resist drawing on its own knowledge), Command R or Command R+ are specifically trained for this and produce cleaner citations. For coding documentation RAG — where your document set is code files, API docs, or README files — a coding-tuned model like Qwen 2.5 Coder as the generation model gives noticeably better code examples in the responses. The embedding model choice is separate and equally important: nomic-embed-text and mxbai-embed-large consistently outperform using a general-purpose LLM for embeddings, because they’re trained specifically to produce meaningful semantic representations rather than next-token predictions. Always use a dedicated embedding model for the retrieval step, and use whichever generation model best fits your task type for the synthesis step. Mixing these — a fast 3B model for generation with a high-quality dedicated embedding model — often gives better overall results than using a single large model for both tasks.

What to Build With This

With LlamaIndex connected to Ollama, a few practical projects stand out as natural starting points. A personal knowledge base that indexes your notes, PDFs, and bookmarks and lets you query across them with natural language is achievable in under 100 lines of code using the patterns above. A local documentation assistant for a codebase or technical reference — index the docs, connect the chat engine, and query it from your IDE — replaces the need to manually search documentation for the answers you need. A research paper summariser that loads a directory of PDFs, builds an index, and lets you ask synthesis questions across the whole paper set is genuinely useful for literature reviews. All of these run entirely locally, index once and query many times, and cost nothing per query — which is the fundamental advantage of combining LlamaIndex’s retrieval capabilities with Ollama’s local inference.

Troubleshooting Common LlamaIndex + Ollama Issues

The most common issue when setting up LlamaIndex with Ollama is the embedding step hanging or timing out on large document sets. Each chunk requires an API call to Ollama’s embeddings endpoint, and for a directory of 200 documents chunked into 500-token pieces, that’s potentially thousands of API calls. The fix: increase the request_timeout on the OllamaEmbedding constructor, and consider using a faster embedding model — nomic-embed-text is fast enough that most embedding jobs complete in a few minutes rather than tens of minutes. If embedding is still slow, reduce your chunk size to 256 tokens to generate fewer total chunks. The second common issue is retrieval returning irrelevant chunks despite a good embedding model — this usually means similarity_top_k is too high (retrieving 10 or 20 chunks introduces noise) or too low (missing the relevant chunk). Start with similarity_top_k=3 and increase to 5 if answers feel incomplete. The third issue is generation ignoring the retrieved context and answering from the LLM’s own knowledge — this is a prompt issue; LlamaIndex’s default system prompt instructs the model to use the context, but some models follow this more reliably than others. If you see this, try Command R+ which has specific training for grounded generation, or explicitly add “Answer only using the provided context. If the answer is not in the context, say ‘I don’t know.'” to your system prompt via the text_qa_template parameter on the query engine.

Once LlamaIndex and Ollama are working together reliably on a small test set, scaling to larger document collections is largely a matter of choosing the right persistent vector store — ChromaDB for local persistent storage, Qdrant for production-grade performance, or pgvector if you’re already running PostgreSQL — and the same code patterns above transfer directly with only the vector store initialisation changing.

LlamaIndex vs LangChain for Local RAG

If you’re coming from LangChain, the core question is which framework fits your use case better. LlamaIndex is purpose-built for data ingestion and retrieval — the document loading, chunking, embedding, and query engine architecture is its entire focus, and it’s generally simpler to set up a working RAG pipeline in LlamaIndex than in LangChain. LangChain is a broader framework that covers agents, chains, tool use, memory, and RAG — more flexible but more complex to configure. For a project where the primary task is document Q&A, LlamaIndex with Ollama is often the faster path to a working, production-quality system. For a project where RAG is one component among several (alongside agents, tool calling, and complex chain logic), LangChain’s broader ecosystem might justify the additional complexity. Both integrate with Ollama cleanly and the patterns are similar enough that migrating between them is manageable if your requirements evolve. The good news is that trying LlamaIndex costs you very little — the setup above is 20–30 lines of code, and you’ll know within an hour whether it meets your retrieval quality requirements before committing to it as the foundation of your project.