RAG (Retrieval-Augmented Generation) lets you give a language model access to your own documents — PDFs, notes, wikis, code — without fine-tuning. The model answers questions using content retrieved from your document store rather than only its training data. This guide builds a complete local RAG pipeline: Ollama for LLM inference, ChromaDB for vector storage, and nomic-embed-text for embeddings — everything running on your machine, no API keys required.
How RAG Works
RAG has two phases. The indexing phase runs once (and whenever your documents change): load documents, split them into chunks, embed each chunk into a vector using an embedding model, and store the chunk text and vector in a database. The retrieval phase runs on each query: embed the query using the same model, find the most similar document chunks in the database, and pass those chunks as context to the LLM along with the original question. The LLM generates an answer grounded in the retrieved content rather than relying only on training knowledge. The quality of your RAG system depends heavily on chunking strategy (how you split documents), embedding model quality, and how many chunks you retrieve.
Installation
# Install Python dependencies
pip install chromadb ollama langchain-text-splitters pypdf
# Pull the embedding model and a capable LLM in Ollama
ollama pull nomic-embed-text # fast, high quality embedding model
ollama pull llama3.1 # for generating answersnomic-embed-text is the recommended local embedding model: fast (embeds thousands of chunks per minute on CPU), high quality (competitive with OpenAI’s text-embedding-ada-002), and designed specifically for RAG retrieval. It outputs 768-dimensional vectors and supports long documents with its 8192-token context window.
Document Loading and Chunking
from langchain_text_splitters import RecursiveCharacterTextSplitter
import pypdf, pathlib
def load_pdf(path: str) -> str:
reader = pypdf.PdfReader(path)
return "\n\n".join(page.extract_text() or "" for page in reader.pages)
def load_txt(path: str) -> str:
return pathlib.Path(path).read_text(encoding="utf-8", errors="ignore")
def chunk_text(text: str, chunk_size: int = 512,
chunk_overlap: int = 64) -> list[str]:
splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
separators=["\n\n", "\n", ". ", " ", ""]
)
return splitter.split_text(text)
# Load and chunk a PDF
text = load_pdf("my_document.pdf")
chunks = chunk_text(text)
print(f"Split into {len(chunks)} chunks")Chunk size of 512 tokens with 64-token overlap is a solid starting point for most documents. Smaller chunks (256) improve precision for factual Q&A; larger chunks (1024) work better for summarisation tasks. The overlap prevents information loss at chunk boundaries. Adjust based on your specific document type and query patterns.
Embedding and Storing in ChromaDB
import chromadb
import ollama
client = chromadb.PersistentClient(path="./rag_db")
collection = client.get_or_create_collection(
name="documents",
metadata={"hnsw:space": "cosine"} # cosine distance for text embeddings
)
def embed(texts: list[str]) -> list[list[float]]:
"""Embed texts using nomic-embed-text via Ollama."""
return [
ollama.embeddings(model="nomic-embed-text", prompt=t)["embedding"]
for t in texts
]
def index_document(file_path: str, source_name: str):
"""Load, chunk, embed, and store a document."""
ext = file_path.rsplit(".", 1)[-1].lower()
text = load_pdf(file_path) if ext == "pdf" else load_txt(file_path)
chunks = chunk_text(text)
embeddings = embed(chunks)
ids = [f"{source_name}_{i}" for i in range(len(chunks))]
collection.add(
ids=ids,
embeddings=embeddings,
documents=chunks,
metadatas=[{"source": source_name, "chunk": i} for i in range(len(chunks))]
)
print(f"Indexed {len(chunks)} chunks from {source_name}")
# Index your documents
index_document("docs/company_handbook.pdf", "handbook")
index_document("docs/product_spec.txt", "product_spec")Retrieval and Generation
def retrieve(query: str, n_results: int = 5,
source_filter: str = None) -> list[str]:
"""Retrieve the most relevant chunks for a query."""
query_embedding = ollama.embeddings(
model="nomic-embed-text", prompt=query
)["embedding"]
where = {"source": source_filter} if source_filter else None
results = collection.query(
query_embeddings=[query_embedding],
n_results=n_results,
where=where
)
return results["documents"][0] # list of chunk texts
def answer(query: str, n_results: int = 5) -> str:
"""Retrieve context and generate an answer with Ollama."""
chunks = retrieve(query, n_results=n_results)
context = "\n\n---\n\n".join(chunks)
prompt = f"""Answer the question using only the context below.
If the answer is not in the context, say "I don't have that information."
Context:
{context}
Question: {query}
Answer:"""
response = ollama.chat(
model="llama3.1",
messages=[{"role": "user", "content": prompt}],
options={"temperature": 0.1} # low temperature for factual answers
)
return response["message"]["content"]
# Use the RAG pipeline
print(answer("What is the company's remote work policy?"))
print(answer("What are the key features of the new product?"))Figure 1 — Local RAG Pipeline Architecture
Improving Retrieval Quality
The basic pipeline above works but has room for improvement. Several techniques consistently improve RAG answer quality. Increase retrieval count, then re-rank: retrieve 10-20 chunks instead of 5, then use a cross-encoder re-ranker to select the best 3-5 for the LLM context. Cross-encoders compare query and chunk together (rather than independently embedding them) and produce more accurate relevance scores. Query expansion: before retrieving, ask the LLM to generate 2-3 alternative phrasings of the question, retrieve for all phrasings, and deduplicate results. Different phrasings surface different relevant chunks. Contextual chunking: when splitting documents, include surrounding context (the section heading, the document title) in each chunk so the LLM can understand context even when a chunk is retrieved in isolation. Smaller chunks with larger context window: index with small chunks (256 tokens) for precise retrieval, but when a chunk is retrieved, include the surrounding paragraph (the “parent” chunk) in the context passed to the LLM. This provides precise retrieval with rich context for generation.
Handling Multiple Document Collections
For RAG systems that need to search across different document types or topics, ChromaDB collections provide natural separation:
# Separate collections for different document types
technical_docs = client.get_or_create_collection("technical_docs")
business_docs = client.get_or_create_collection("business_docs")
def answer_with_routing(query: str) -> str:
# Retrieve from both collections, combine results
tech_chunks = retrieve_from(technical_docs, query, n=3)
biz_chunks = retrieve_from(business_docs, query, n=3)
all_chunks = tech_chunks + biz_chunks
# Pass combined context to LLM
return generate_answer(query, all_chunks)Alternatively, use metadata filtering to search within a specific subset of a single collection — more efficient than separate collections when the distinction is a metadata attribute rather than a fundamentally different embedding space.
Adding a Simple Chat Interface with Gradio
pip install gradio
import gradio as gr
def chat(message, history):
return answer(message)
interface = gr.ChatInterface(
fn=chat,
title="Local Document Q&A",
description="Ask questions about your indexed documents.",
examples=["What are the key points in the handbook?",
"Summarise the product specification."]
)
interface.launch(server_name="0.0.0.0", server_port=7860)This launches a chat interface accessible at localhost:7860 — a clean web UI for querying your document collection that any browser can access. Add authentication (gradio’s auth parameter) if you are running on a shared machine or network.
Performance Optimisation: Batch Embedding
Embedding chunks one at a time (as in the basic example above) is slow for large document collections. Batch embedding significantly improves indexing speed:
def embed_batch(texts: list[str], batch_size: int = 32) -> list[list[float]]:
"""Embed texts in batches for efficiency."""
all_embeddings = []
for i in range(0, len(texts), batch_size):
batch = texts[i:i+batch_size]
# nomic-embed-text processes batches via Ollama's batch API
embeddings = [
ollama.embeddings(model="nomic-embed-text", prompt=t)["embedding"]
for t in batch
]
all_embeddings.extend(embeddings)
if i % 100 == 0:
print(f"Embedded {i}/{len(texts)} chunks...")
return all_embeddingsFor very large document collections (tens of thousands of chunks), consider using a faster embedding model (all-minilm-l6-v2 via sentence-transformers is 5-10x faster than nomic-embed-text at some quality cost) or running embedding in parallel across multiple Ollama instances. The embedding step is the bottleneck for large-scale indexing — generation is fast but embedding each chunk sequentially accumulates.
Keeping the Index Fresh
Documents change, and your ChromaDB index needs to stay in sync. The simplest approach: track a hash or modification timestamp for each source document and re-index when it changes. ChromaDB’s collection.update() and collection.delete() methods allow targeted updates without rebuilding the entire index. For production systems with frequently changing documents, consider a change-data-capture pattern that watches your document source (a folder, a database, a CMS) and triggers incremental re-indexing when documents are added, modified, or deleted. The key invariant: the ChromaDB collection should always reflect the current state of your documents. Stale embeddings from outdated document versions produce confusing answers that are harder to debug than outright retrieval failures.
Why Local RAG Matters for Privacy
The privacy argument for local RAG is the same as for local LLMs generally, but stronger for document-based workloads. When you use a cloud RAG service or paste documents into a cloud chatbot, your documents are sent to external servers, potentially used for training, logged for audit purposes, and subject to the provider’s data retention policies. For company handbooks, internal product specifications, client case files, medical records, legal documents, and any other sensitive text, this is a real concern that local RAG eliminates entirely. Your documents stay on your machine. Your questions stay on your machine. The ChromaDB database on disk contains your document embeddings — dense numerical vectors that cannot be directly reversed into the original text, though they do reveal semantic content. Everything in the pipeline — embedding with nomic-embed-text via Ollama, storage in ChromaDB, generation with your Ollama LLM — processes data locally with no network calls to external services. For individuals and small teams handling sensitive documents who want AI assistance without cloud exposure, local RAG is the appropriate architecture.
Common Issues and How to Fix Them
A few problems come up regularly when building RAG systems. The LLM says it does not have information even though the document contains the answer: this usually means retrieval is failing — the query embedding is not similar enough to the relevant chunks. Try retrieving more chunks (increase n_results to 10 or 15), check that you are using the same embedding model for indexing and querying, and verify the relevant content is actually in the collection (use collection.get() to confirm). The LLM gives an answer not grounded in the retrieved context: the system prompt is too weak or the LLM is ignoring the context instruction. Strengthen the prompt to explicitly say “answer only using the provided context, do not use outside knowledge” and lower the temperature to 0.0. Slow indexing for large document collections: embed in batches and consider running embedding overnight for large archives. Inconsistent chunk quality: experiment with chunk size — if chunks are too small they lose context, too large they dilute the relevant content. 512 tokens with 64-token overlap is a good starting point but may need adjustment for your specific document type.
Scaling Up: From Prototype to Production
The local RAG pipeline described in this guide works well for personal use and small team deployments with document collections up to a few hundred thousand chunks. When you need to scale further — larger collections, more concurrent users, higher query volume, or multi-user access with different permission levels — the same logical architecture applies but the infrastructure evolves. ChromaDB’s client-server mode handles multiple concurrent clients. Qdrant replaces ChromaDB for larger collections with better performance and filtering. A proper web framework (FastAPI) wraps the retrieval and generation logic behind an API. Authentication and per-user collection access control layer on top. But none of these scaling steps require rebuilding the core logic — the chunking, embedding, retrieval, and generation pipeline remains identical. Getting the prototype working correctly is the valuable engineering work; the infrastructure scaling is incremental from there.
The Complete Working Pipeline
To recap the minimal working implementation: install chromadb and ollama with pip, pull nomic-embed-text and llama3.1 in Ollama, create a persistent ChromaDB client, write an index_document function that loads, chunks, embeds, and stores your files, and write an answer function that embeds the query, retrieves top-k chunks, and calls Ollama for generation. The entire functional pipeline is under 60 lines of Python, requires no API keys, no cloud services, and costs nothing per query beyond electricity. For document collections up to a few hundred PDFs or text files, the retrieval latency is under 100ms and the total generation latency is whatever your Ollama model speed produces — typically 2-10 seconds for a thoughtful answer. This is a practical, production-usable RAG system that handles the majority of personal and small team document Q&A use cases without any external dependencies.