Chunking — splitting documents into pieces before embedding — is one of the most impactful decisions in a RAG pipeline, and one that receives far less attention than model choice or retrieval algorithms. The same documents chunked differently can produce dramatically different retrieval quality. This guide covers the three main strategies, when each works best, and how to evaluate chunking quality on your specific documents.
Why Chunking Matters
Embedding models produce a single vector representing the semantic meaning of a piece of text. That vector must capture enough of the content to match relevant queries, while remaining focused enough to retrieve precisely. Two failure modes define the chunking challenge. Chunks that are too large dilute the relevant content — a 2000-word section on company policies embeds to a vector representing the average meaning of all those policies, making it hard to retrieve specifically for narrow questions about any single policy. Chunks that are too small lose context — a single sentence like “the limit is $500” embeds without the surrounding context explaining what the limit applies to, producing a retrieved chunk that is unhelpful. Finding the right granularity for your specific documents and query types is the core chunking design problem.
Fixed-Size Chunking
The simplest approach: split text every N characters or tokens, with optional overlap between adjacent chunks. LangChain’s RecursiveCharacterTextSplitter implements a practical variant that respects natural text boundaries (trying to split at paragraphs, then sentences, then spaces) before falling back to fixed character counts:
from langchain_text_splitters import RecursiveCharacterTextSplitter
# Standard configuration
splitter = RecursiveCharacterTextSplitter(
chunk_size=512, # target chunk size in characters
chunk_overlap=64, # overlap between adjacent chunks
separators=["\n\n", "\n", ". ", " ", ""] # try these in order
)
# Token-aware splitting (more accurate for embedding models)
from langchain_text_splitters import TokenTextSplitter
token_splitter = TokenTextSplitter(
chunk_size=256, # in tokens, not characters
chunk_overlap=32
)
chunks = splitter.split_text(document_text)
print(f"Split into {len(chunks)} chunks")
print(f"Average chunk length: {sum(len(c) for c in chunks) / len(chunks):.0f} chars")Fixed chunking is fast, deterministic, and works well for homogeneous text documents — articles, reports, books — where the content is relatively uniform in structure. The optimal chunk size depends heavily on your embedding model’s effective context window and your query length. A rule of thumb: chunks should be 2-5x the length of typical queries. For conversational Q&A with short questions, smaller chunks (256-512 characters) retrieve more precisely. For longer analytical queries, larger chunks (512-1024) provide more context per retrieved unit.
The Overlap Trade-off
Chunk overlap prevents information loss at boundaries — a sentence split across two chunks will appear in at least one chunk fully if the overlap is sufficient. But overlap increases index size (a 10% overlap increases total stored content by 10%) and can cause near-duplicate chunks to appear in retrieval results, wasting context window space with redundant content. A practical heuristic: set overlap to 10-15% of chunk size. For a 512-character chunk, 50-75 characters of overlap is appropriate. Avoid large overlaps (50%+) that produce heavily redundant chunks. For documents with very short, high-information sentences (code, legal clauses, database schemas), small or zero overlap is often better because chunks are already self-contained.
Semantic Chunking
Semantic chunking splits documents at topic boundaries rather than fixed character counts, using embedding similarity to detect when the content shifts to a new concept. The algorithm: embed consecutive sentences, compute cosine similarity between adjacent sentence pairs, identify points where similarity drops sharply (topic shifts), and split at those breakpoints:
from langchain_experimental.text_splitter import SemanticChunker
from langchain_ollama import OllamaEmbeddings
embeddings = OllamaEmbeddings(model="nomic-embed-text")
semantic_splitter = SemanticChunker(
embeddings,
breakpoint_threshold_type="percentile", # or "standard_deviation", "interquartile"
breakpoint_threshold_amount=95, # split at top 5% similarity drops
)
chunks = semantic_splitter.split_text(document_text)
print(f"Semantic chunks: {len(chunks)}, avg length: {sum(len(c) for c in chunks)/len(chunks):.0f} chars")Semantic chunking produces chunks that are more topically coherent than fixed splitting — each chunk represents a complete thought or topic section rather than an arbitrary text window. This is particularly valuable for documents with clear topic changes: long blog posts, technical manuals, research papers, and meeting transcripts where topic shifts are semantically meaningful. The trade-off: semantic chunking is slower (requires embedding every sentence during indexing) and produces variable-size chunks that are harder to reason about. For documents with a consistent structure and uniform paragraph length, fixed chunking often performs as well as semantic at lower cost.
Figure 1 — Chunking Strategy Comparison
Parent-Child Chunking in Depth
Parent-child chunking (also called small-to-big retrieval) solves the fundamental tension between retrieval precision and context richness. Index small child chunks for precise retrieval, but when a child is retrieved, return its larger parent chunk to the LLM. The child embedding captures a specific claim or fact precisely; the parent provides the surrounding context the LLM needs to formulate a complete answer.
from langchain_text_splitters import RecursiveCharacterTextSplitter
import chromadb, ollama
# Two splitters: large parent, small child
parent_splitter = RecursiveCharacterTextSplitter(
chunk_size=1500, chunk_overlap=100)
child_splitter = RecursiveCharacterTextSplitter(
chunk_size=300, chunk_overlap=30)
client = chromadb.PersistentClient(path="./rag_db")
collection = client.get_or_create_collection("parent_child")
def embed(text: str) -> list[float]:
return ollama.embeddings(model="nomic-embed-text", prompt=text)["embedding"]
def index_with_parent_child(text: str, source: str):
parents = parent_splitter.split_text(text)
ids, embeddings, documents, metadatas = [], [], [], []
for p_idx, parent in enumerate(parents):
children = child_splitter.split_text(parent)
for c_idx, child in enumerate(children):
ids.append(f"{source}_p{p_idx}_c{c_idx}")
embeddings.append(embed(child)) # embed the small child
documents.append(child)
metadatas.append({
"source": source,
"parent_idx": p_idx,
"parent_text": parent, # store parent alongside child
"child_idx": c_idx
})
collection.add(ids=ids, embeddings=embeddings,
documents=documents, metadatas=metadatas)
print(f"Indexed {len(ids)} children across {len(parents)} parents")
def retrieve_parents(query: str, n_parents: int = 3) -> list[str]:
# Retrieve children, deduplicate by parent, return parent texts
results = collection.query(
query_embeddings=[embed(query)],
n_results=n_parents * 4, # over-retrieve children
include=["metadatas"]
)
seen, parents = set(), []
for meta in results["metadatas"][0]:
key = (meta["source"], meta["parent_idx"])
if key not in seen:
seen.add(key)
parents.append(meta["parent_text"])
if len(parents) == n_parents:
break
return parentsParent-child chunking consistently outperforms pure small or pure large chunking on diverse question sets because it optimises each step independently: retrieval uses the most precise unit possible, generation uses the richest context available. The main overhead is storing parent text in each child’s metadata, which increases ChromaDB storage but eliminates a second database lookup to fetch parent content.
Structure-Aware Chunking
For documents with explicit structure — Markdown headers, HTML headings, LaTeX sections, code blocks — splitting along structural boundaries produces semantically complete chunks that respect the document’s own organisation. LangChain provides structure-aware splitters for several formats:
from langchain_text_splitters import MarkdownHeaderTextSplitter, HTMLHeaderTextSplitter
# Split Markdown by heading hierarchy
md_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=[
("#", "h1"), ("##", "h2"), ("###", "h3")
],
strip_headers=False # keep headers in chunk text for context
)
md_chunks = md_splitter.split_text(markdown_text)
# Each chunk includes its heading path as metadata: {"h1": "Section", "h2": "Subsection"}
# Split HTML by heading tags
html_splitter = HTMLHeaderTextSplitter(
headers_to_split_on=[("h1", "h1"), ("h2", "h2"), ("h3", "h3")]
)
html_chunks = html_splitter.split_text(html_content)Structure-aware chunking is the right choice for documentation sites, wikis, and any Markdown or HTML content where headings define coherent topics. The heading path stored in metadata enables metadata filtering during retrieval — “retrieve only from the h2 section ‘Configuration'” — which dramatically improves precision for navigational queries on large documentation sites.
Evaluating Chunking Quality
The best way to evaluate chunking quality is to build a small test set of question-answer pairs where you know which document section contains the answer, then measure whether retrieval finds the right section. For each test question, retrieve the top-5 chunks and check whether the answer is present in any of them — this is recall at 5. Compare recall at 5 across different chunking configurations on the same test set. A chunking configuration that achieves 90% recall at 5 is retrieving the relevant content for 90% of test questions, leaving only 10% unanswered due to retrieval failure. Improvements in chunking that increase recall from 70% to 90% typically translate directly into better final answer quality. Run this evaluation before and after any chunking parameter change to confirm the change helps rather than hurts. The test set size of 20-50 question-answer pairs is sufficient for meaningful comparison; larger sets give more statistical confidence but are more expensive to create.
Document-Type Chunking Recommendations
Different document types benefit from different chunking strategies. For prose documents (books, articles, reports): fixed 512-character chunks with 10% overlap is a reliable default, semantic chunking if topic boundaries are important. For technical documentation and wikis: structure-aware (Markdown/HTML header) splitting is the clear winner — it mirrors the document’s own organisation. For legal and policy documents: parent-child with large parents (full clauses or sections) and small children for specific provisions; legal documents often contain defined terms that only make sense with surrounding context. For academic papers: split by section (abstract, introduction, methods, results, discussion) and treat each section as a parent, sub-split for retrieval. For code documentation and README files: separate code blocks from prose, index them differently. For conversational transcripts: sentence-window chunking (each sentence with surrounding context) works better than arbitrary fixed chunks because meaning is highly local. For news articles and blog posts: fixed medium-size (512-768 chars) with full paragraphs respected; structure-aware if HTML is available.
The Single Most Impactful Chunking Decision
If there is one chunking choice that consistently matters more than any other, it is whether to include surrounding context in each chunk. A chunk that begins mid-sentence or references “the policy mentioned above” without including what that policy is will retrieve precisely but be unhelpful when the LLM tries to use it. Chunks should be self-contained — a reader seeing only that chunk should be able to understand its meaning without needing adjacent chunks. This principle argues for three practices: include enough overlap to prevent meaningful sentences from being split across chunks, include section headings as part of each chunk so the LLM knows what topic the content is about, and for documents with heavy co-reference (it, this, they referring to earlier entities), use contextual retrieval (generating a context prefix per chunk with an LLM) or larger chunk sizes that are more likely to contain the referenced entities. Self-contained chunks that retrieve precisely are the goal; fixed-size splitting is a practical approximation, and the more your chunks deviate from self-containedness, the more you pay in answer quality.
Iterating on Chunking in Practice
A practical chunking iteration workflow: start with fixed 512-character chunks with 10% overlap — this is your baseline. Build a small evaluation set of 20-30 question-answer pairs drawn from your documents. Measure recall at 5 on the baseline. Try smaller chunks (256), larger chunks (768), and semantic chunking on the same evaluation set. Try parent-child if you have the engineering capacity. Pick the configuration with the highest recall. Redeploy and re-evaluate on a fresh set of real user queries after a week of production use. The evaluation-driven approach prevents intuition-driven chunking decisions that often feel right but measure poorly. The configuration that measures best on your actual documents and queries is almost always different from the theoretical optimum — empirical measurement is the only reliable guide.
Chunking and Embedding Interaction
Chunking and embedding model choice interact in ways that are easy to overlook. Different embedding models have different effective context lengths — the number of tokens they process faithfully before attention degrades. nomic-embed-text supports 8192 tokens and handles long chunks well. all-MiniLM-L6-v2 has a 256-token limit and produces poor embeddings for chunks longer than roughly 200 words. Mismatching chunk size to embedding context length is a common source of unexpectedly poor retrieval. Always verify the maximum sequence length of your embedding model and keep chunks well within it — a chunk at 80% of the model’s maximum length is safer than one at 100%. The quality loss from truncation is severe and silent: the model truncates without warning, and the resulting embedding represents only the first portion of the chunk while appearing to represent the whole thing. For nomic-embed-text’s 8192-token limit, chunks of 512-1024 characters (roughly 100-200 tokens) are well within range. For shorter-context embedding models, smaller fixed chunks are not just preferable but required for correct embeddings.
Late Chunking: A Newer Approach
Late chunking (introduced by JinaAI in 2024) offers an alternative to traditional chunking by embedding the full document first to capture long-range dependencies, then pooling token embeddings into chunk-level vectors. Unlike traditional chunking where each chunk is embedded independently without access to surrounding context, late chunking embeds each chunk with awareness of the full document. The result: chunks at the end of a document that refer to concepts introduced at the beginning are embedded correctly, capturing the cross-reference that traditional independent-chunk embedding misses. Late chunking requires models that expose token-level embeddings (JinaAI’s models support this natively), so it is not a drop-in replacement for standard chunking pipelines. For documents with high inter-sentence co-reference or long-range context dependencies, late chunking measurably improves retrieval quality. For most standard documents, the improvement is modest and the implementation complexity is significant. Watch the embedding model landscape for growing late chunking support — it is a promising direction that will likely become more accessible as more models support it.