LangChain wraps ChromaDB (and most other vector databases) behind a unified interface, making it easy to swap components and prototype quickly. But LangChain adds abstraction overhead, makes debugging harder, and changes frequently enough that tutorials become outdated quickly. This guide covers both approaches: the LangChain integration for teams already invested in the ecosystem, and the direct Python approach for those who want full control without the framework layer.
ChromaDB with LangChain: The Integrated Approach
LangChain provides a Chroma vector store class that handles embedding, storage, and retrieval behind a consistent interface. The advantage: LangChain’s retriever interface plugs directly into chains, agents, and other LangChain components. The disadvantage: the abstraction hides what is actually happening, making debugging harder, and LangChain’s API changes frequently enough that code written six months ago often requires updates.
pip install langchain langchain-community langchain-ollama chromadbfrom langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain_community.vectorstores import Chroma
from langchain_community.document_loaders import PyPDFLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
# Embeddings and LLM via Ollama
embeddings = OllamaEmbeddings(model="nomic-embed-text")
llm = ChatOllama(model="llama3.1", temperature=0.1)
# Load and split documents
loader = PyPDFLoader("./docs/handbook.pdf")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_documents(docs)
# Create vector store (embeds and stores automatically)
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory="./chroma_langchain"
)
# Create retriever
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
# Build RAG chain using LCEL (LangChain Expression Language)
prompt = ChatPromptTemplate.from_template(
"""Answer using only the context below. If the answer is not in the
context, say you don't have that information.
Context: {context}
Question: {question}"""
)
def format_docs(docs):
return "\n\n---\n\n".join(d.page_content for d in docs)
chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
# Use the chain
response = chain.invoke("What is the remote work policy?")
print(response)The LCEL (LangChain Expression Language) pipe syntax composes components cleanly. The retriever automatically retrieves, the prompt template formats context and question, the LLM generates, and StrOutputParser extracts the string. Each step is clear in isolation but the inter-component plumbing is hidden.
Loading an Existing ChromaDB Collection in LangChain
# Load previously created collection (no re-embedding)
vectorstore = Chroma(
persist_directory="./chroma_langchain",
embedding_function=embeddings
)
# Different retrieval modes
retriever_mmr = vectorstore.as_retriever(
search_type="mmr", # Max Marginal Relevance — diversifies results
search_kwargs={"k": 5, "fetch_k": 20}
)
retriever_threshold = vectorstore.as_retriever(
search_type="similarity_score_threshold",
search_kwargs={"score_threshold": 0.7, "k": 5}
)
# Metadata filtering
retriever_filtered = vectorstore.as_retriever(
search_kwargs={"k": 5, "filter": {"source": "handbook.pdf"}}
)Max Marginal Relevance (MMR) diversifies the retrieved chunks — instead of returning the 5 most similar chunks (which may all be very similar to each other), MMR balances similarity to the query with diversity among results. This often improves answer quality for broader questions by retrieving more distinct pieces of information.
RAG Without LangChain: The Direct Approach
The LangChain-free implementation uses the native ChromaDB Python client and the Ollama Python library directly. The result is more code but complete visibility into every step — no magic, no hidden API calls, no version compatibility surprises.
import chromadb
import ollama
from langchain_text_splitters import RecursiveCharacterTextSplitter # just the splitter
import pypdf
from pathlib import Path
# Setup
client = chromadb.PersistentClient(path="./chroma_direct")
collection = client.get_or_create_collection(
name="docs",
metadata={"hnsw:space": "cosine"}
)
def load_document(path: str) -> str:
p = Path(path)
if p.suffix == ".pdf":
reader = pypdf.PdfReader(path)
return "\n\n".join(page.extract_text() or "" for page in reader.pages)
return p.read_text(encoding="utf-8", errors="ignore")
def embed(text: str) -> list[float]:
return ollama.embeddings(model="nomic-embed-text", prompt=text)["embedding"]
def index(path: str, source: str):
text = load_document(path)
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_text(text)
collection.add(
ids=[f"{source}_{i}" for i in range(len(chunks))],
embeddings=[embed(c) for c in chunks],
documents=chunks,
metadatas=[{"source": source, "chunk": i} for i in range(len(chunks))]
)
print(f"Indexed {len(chunks)} chunks from {source}")
def retrieve(query: str, n: int = 5, source: str = None) -> list[str]:
where = {"source": source} if source else None
results = collection.query(
query_embeddings=[embed(query)],
n_results=n,
where=where
)
return results["documents"][0]
def answer(query: str) -> str:
chunks = retrieve(query)
context = "\n\n---\n\n".join(chunks)
resp = ollama.chat(
model="llama3.1",
messages=[{"role": "user", "content":
f"Answer using only the context below. If not in context, say so.\n\nContext:\n{context}\n\nQuestion: {query}"
}],
options={"temperature": 0.1}
)
return resp["message"]["content"]
# Use it
index("./docs/handbook.pdf", "handbook")
print(answer("What is the remote work policy?"))Figure 1 — LangChain RAG vs Direct Python RAG: Trade-offs
When to Use LangChain vs Direct Python
The decision is straightforward once you understand the trade-off. Use LangChain when you are building within an existing LangChain project, when you need LangChain agents (which provide tool use and multi-step reasoning built on the retriever interface), when you want to experiment with different vector databases by swapping the vector store class, or when your team already knows LangChain well and productivity matters more than transparency. Use direct Python (ChromaDB + Ollama natively) when you are learning how RAG works and want to understand each step, when you are building a standalone application without agent requirements, when API stability matters for long-running production code, or when you are debugging a retrieval quality problem and need to inspect exactly what is being retrieved and why. Many experienced RAG developers start with LangChain for prototyping speed and rewrite the critical retrieval path in direct Python once the architecture is validated — the two approaches solve the same problem at different abstraction levels.
Streaming Responses with LangChain + Ollama
from langchain_ollama import ChatOllama
llm_stream = ChatOllama(model="llama3.1", temperature=0.1)
# Stream the chain output
for chunk in chain.stream("What is the holiday policy?"):
print(chunk, end="", flush=True)
print()LangChain’s LCEL pipe operator propagates streaming automatically through the chain — if the LLM supports streaming (Ollama does), the chain outputs tokens as they generate. This is useful for interactive applications where you want the user to see the response build progressively rather than waiting for the full answer.
Multi-Document RAG with Source Attribution
A common requirement is knowing which document each retrieved chunk came from — so the LLM can cite sources in its answer:
def answer_with_sources(query: str) -> dict:
"""Return answer and source attribution."""
chunks = retrieve(query, n=5)
results = collection.query(
query_embeddings=[embed(query)],
n_results=5,
include=["documents", "metadatas", "distances"]
)
docs = results["documents"][0]
metas = results["metadatas"][0]
distances = results["distances"][0]
# Build context with source labels
context_parts = []
for doc, meta, dist in zip(docs, metas, distances):
similarity = 1 - dist # convert cosine distance to similarity
context_parts.append(
f"[Source: {meta['source']}, chunk {meta['chunk']}, "
f"similarity: {similarity:.2f}]\n{doc}"
)
context = "\n\n---\n\n".join(context_parts)
prompt = f"""Answer using the context below. Include source citations.
Context:
{context}
Question: {query}"""
response = ollama.chat(
model="llama3.1",
messages=[{"role": "user", "content": prompt}],
options={"temperature": 0.1}
)
return {
"answer": response["message"]["content"],
"sources": [(m["source"], round(1-d, 3))
for m, d in zip(metas, distances)]
}
result = answer_with_sources("What are the expense reimbursement limits?")
print(result["answer"])
print("\nSources:", result["sources"])Choosing an Embedding Model for Local RAG
The embedding model choice has a significant impact on retrieval quality. For local RAG with Ollama, the best options: nomic-embed-text is the recommended default — high quality (MTEB leaderboard competitive), 768-dimensional output, fast inference, 8192 token context, free and open source. mxbai-embed-large is a strong alternative with 1024-dimensional output and slightly better performance on some benchmarks at the cost of larger file size and slower inference. all-minilm is smaller and faster (384 dimensions) — useful when embedding speed matters more than retrieval precision, such as for large-scale indexing jobs where you need to process tens of thousands of documents quickly. The critical rule: use the same embedding model for both indexing (when you store documents) and querying (when you search). Mixing models produces meaningless similarity scores. If you switch embedding models, re-index all documents from scratch — partial re-indexing with a different model creates a corrupted collection.
Evaluating Your RAG Pipeline
The most common RAG failure modes are easy to test manually but require systematic evaluation to catch reliably. Retrieval failures (the relevant chunk is not retrieved): test by querying for specific facts you know are in the documents and checking whether the retrieved chunks actually contain those facts. Generation failures (the LLM ignores or misuses the retrieved context): test by providing context that contradicts a common knowledge claim and checking whether the LLM follows the context or falls back to training knowledge. Boundary handling (questions where the answer is not in the documents): test with questions you know are outside your document collection and check whether the LLM correctly says it does not have the information. Systematic evaluation with a small test set of known question-answer pairs takes a few hours to set up and dramatically improves confidence in production RAG quality. Tools like RAGAS provide automated metrics (answer relevancy, context precision, faithfulness) that can be run against your pipeline to track quality over time as you iterate on chunking, retrieval, and prompting strategies.
Production Considerations: Caching and Latency
A local RAG pipeline has two latency components: embedding the query (fast — nomic-embed-text embeds a sentence in under 100ms on CPU) and generating the answer (slower — typically 2-10 seconds depending on model and hardware). The ChromaDB vector search itself is extremely fast, usually under 10ms for collections under a million vectors. The dominant latency is LLM generation. To reduce perceived latency, stream the response — the user sees words appearing immediately rather than waiting for the full answer. To reduce actual latency, keep the model warm in Ollama (set OLLAMA_KEEP_ALIVE=-1 so the model stays loaded between requests), use a smaller but capable model for simple questions (Llama 3.2 3B answers faster than 70B for factual document Q&A), and cache embeddings for repeated queries. The embedding cache is straightforward: before calling embed(), check a dictionary or database for the query string — if found, return the cached vector. Query caching makes sense when users ask the same questions repeatedly, which is common in document Q&A systems where a finite set of questions covers most usage.
When RAG Is Not the Right Tool
RAG is not the answer to every knowledge gap in an LLM application. It works well when you need the model to answer questions grounded in specific documents, when your knowledge base changes frequently (making fine-tuning impractical), and when you need source attribution. RAG works poorly when the relevant information is spread thinly across many documents requiring synthesis rather than retrieval (the model gets fragments rather than a coherent picture), when questions require deep domain expertise that no amount of context can substitute for (retrieval-augmented hallucination is still hallucination), and when your document collection is very large and queries are broad (thousands of potentially relevant chunks cannot all fit in the context window). For questions that require reading an entire document and synthesising across it, long-context models (128K+ context) can outperform RAG by simply reading the full document rather than relying on retrieved fragments. Understanding when retrieval helps and when it hurts is as important as knowing how to build the pipeline.
Getting Started in Under an Hour
The fastest path to a working local RAG system: install chromadb, ollama, pypdf, and langchain-text-splitters with pip; pull nomic-embed-text and llama3.1 in Ollama; copy the direct Python implementation from this guide into a script; point index() at a PDF you have locally; run answer() with a question you know the document can answer. Confirm the retrieved chunks actually contain the relevant information (print them out). If retrieval is working, your RAG pipeline is functionally complete. The entire process takes under an hour for a developer comfortable with Python, and less if you use the LangChain integration. Iterate from there: improve chunking, add more documents, tune the number of retrieved chunks, and test with questions that span multiple sections to understand where the system struggles before moving to production use.
LangChain vs Direct Python: The Honest Summary
After building RAG systems both ways, the honest assessment is: LangChain’s abstractions genuinely help at the start (less boilerplate, faster to a working prototype) and genuinely hurt later (harder to debug, API churn, hidden behaviour). Direct Python takes longer to write initially but produces code that is easier to maintain, debug, and extend. The LangChain community is large and the documentation is extensive, which helps when you hit problems — but the problems you hit with LangChain are often caused by the framework itself, while problems in direct Python are almost always in your own logic and easier to fix. A pragmatic approach: use the LangChain integration for the first version of a RAG system to validate the architecture quickly, then rewrite the retrieval and generation core in direct Python once the approach is proven. This captures the prototype speed advantage without the long-term maintenance cost of deep LangChain dependency in production code.
ChromaDB Persistence and Backup
ChromaDB’s PersistentClient stores the vector index and document metadata in a directory on disk. The directory contains SQLite files for metadata and binary files for the HNSW index. Back up this directory to preserve your index — copying the full directory to another location or cloud storage is sufficient. ChromaDB does not currently provide built-in backup tooling, so scheduled directory copies via rsync, tar, or cloud sync are the standard approach. For critical RAG systems where the indexed documents represent significant embedding compute cost, back up the ChromaDB directory alongside the source documents — losing the index means re-embedding everything from scratch, which for large collections can take hours. On write-heavy workloads, ChromaDB’s WAL (write-ahead log) ensures durability of recent writes even if the process crashes before a full sync — you do not lose data between writes. Index the PersistentClient path in your application configuration so it is easy to redirect to a backup or migrated location without changing code.