What Embeddings Are and Why They Matter
An embedding is a numerical representation of a piece of text — a vector of floating-point numbers that encodes the semantic meaning of the text in a form that mathematics can operate on. Two sentences with similar meanings will have embeddings that are close together in the high-dimensional vector space; sentences with different meanings will be far apart. This mathematical property enables semantic search: finding text that means the same thing as a query, regardless of whether the exact words match.
Embeddings are the foundation of retrieval-augmented generation, semantic search systems, recommendation engines, duplicate detection, and clustering of text data. Understanding how embedding models work, which ones to use for which tasks, and how to implement semantic search reliably is one of the most practically important skills in LLM application development.
How Embedding Models Work
Embedding models are transformer-based neural networks trained to map text to dense vector representations. Unlike generative LLMs that produce token sequences, embedding models produce a single fixed-length vector for an entire input text. They are trained using contrastive learning: positive pairs of semantically similar texts (a question and its answer, a document and a relevant query) are trained to have similar embeddings, while negative pairs of dissimilar texts are trained to have distant embeddings.
The resulting vector space encodes semantic relationships geometrically. The cosine similarity between two vectors — the angle between them in the high-dimensional space — measures their semantic similarity. Cosine similarity of 1.0 means identical direction (highly similar meaning); 0.0 means perpendicular (unrelated); negative values indicate opposite meaning. For practical purposes, embeddings with cosine similarity above 0.85 are typically highly semantically similar, 0.70–0.85 are related, and below 0.70 are loosely related or unrelated — though exact thresholds depend on the embedding model and domain.
Choosing an Embedding Model
Embedding model selection significantly affects semantic search quality, and the right choice depends on your specific use case. OpenAI text-embedding-3-small (1,536 dimensions) is the most widely used embedding model for English text — strong performance, reasonable cost, and the default choice for most RAG applications. OpenAI text-embedding-3-large (3,072 dimensions) improves retrieval quality by 5–10% on hard queries at higher cost and storage. Cohere Embed v3 offers strong multilingual performance and a unique “search” vs “clustering” use case optimisation. BGE-M3 (self-hosted, 1,024 dimensions) delivers excellent multilingual performance across 100+ languages at near-zero inference cost once deployed — the choice for multilingual applications where recurring embedding API costs are a concern. Nomic Embed is a strong open-source English embedding model that matches or exceeds text-embedding-3-small on many benchmarks, available for self-hosting.
The MTEB (Massive Text Embedding Benchmark) leaderboard provides the most comprehensive comparison of embedding model performance across tasks — retrieval, clustering, classification, and others. Consult it for your specific task type before committing to an embedding model, and evaluate the top candidates on a sample of your actual data before production deployment.
Implementing Semantic Search
A basic semantic search system has three components: an embedding pipeline that converts your document corpus to vectors and stores them, a query embedding step that converts the user’s search query to a vector, and a nearest-neighbour search that finds the document vectors most similar to the query vector.
For small to medium corpora (under 1 million documents), pgvector with a well-tuned HNSW index handles semantic search efficiently without additional infrastructure. For large corpora or applications requiring very low latency at high query volume, dedicated vector databases (Pinecone, Qdrant, Weaviate) provide optimised ANN search at scale. For prototyping and small datasets, even in-memory similarity search with numpy is sufficient to validate your approach before investing in persistent storage infrastructure.
The embedding step itself — converting text to vectors — is where model selection has the most impact on search quality. The gap between a well-chosen embedding model and a poorly-chosen one on your specific document domain can be 10–20% in retrieval accuracy, which is significant for any application where retrieval quality directly affects downstream LLM response quality.
Figure 1 — Semantic Search Architecture
Chunking Strategy: The Most Underrated Factor
How you split documents into chunks before embedding has an outsized impact on retrieval quality — arguably more than which embedding model you use. Chunks that are too large contain too much information and produce embeddings that average across many topics, making them less precise for specific queries. Chunks that are too small lose context and produce embeddings that are too narrow to match broader queries. The optimal chunk size depends on your document type and query distribution, but 300–600 tokens per chunk works well for most prose documents. Overlap between adjacent chunks (100–150 tokens of overlap) prevents important information from falling in the gap between chunks and being missed by retrieval.
More sophisticated chunking strategies improve on fixed-size approaches. Semantic chunking splits on natural boundaries — paragraph breaks, section headers, sentence endings — rather than arbitrary token counts, producing chunks that represent coherent units of meaning rather than arbitrary text slices. Hierarchical chunking creates both large summary chunks and small detail chunks, enabling retrieval at different granularities depending on whether the query needs a broad overview or a specific fact. For structured documents (contracts, forms, reports with clear sections), section-aware chunking that respects the document’s own structure consistently outperforms fixed-size chunking.
Hybrid Search: Combining Dense and Sparse Retrieval
Semantic search with embeddings (dense retrieval) excels at finding conceptually related content regardless of exact word matches. Traditional keyword search (sparse retrieval, such as BM25) excels at finding exact term matches — crucial for proper nouns, model names, version numbers, codes, and identifiers that embeddings may not handle well. Hybrid search combines both: run dense and sparse retrieval in parallel and merge the results using Reciprocal Rank Fusion (RRF) or a learned reranker. This hybrid approach consistently outperforms either method alone on real-world RAG benchmarks, particularly for document corpora that mix prose explanation with technical identifiers. Qdrant, Weaviate, and pgvector (with the pg_trgm extension for BM25) all support hybrid search. If you are building a RAG system and are not already using hybrid search, testing it against your pure semantic search baseline should be the next experiment on your roadmap.
Reranking: A Second Pass for Higher Precision
ANN search retrieves the top-K most similar chunks efficiently but not perfectly — approximate nearest neighbour algorithms trade some precision for speed. A reranker performs a more expensive but more accurate assessment of relevance for the top-K retrieved results, reordering them before passing the most relevant to the LLM. Cross-encoder rerankers (Cohere Rerank, BGE Reranker, Jina Reranker) score each query-document pair jointly rather than comparing pre-computed vectors, capturing nuanced relevance that bi-encoder similarity misses. Adding a reranker as a second stage after retrieval typically improves RAG answer quality by 5–15% on hard questions with modest latency cost. The standard pipeline is: retrieve top-20 with ANN, rerank to top-5 with a cross-encoder, pass top-5 to the LLM. This three-stage pipeline consistently outperforms single-stage retrieval on both precision and answer quality.
Evaluating Retrieval Quality
Retrieval quality is the most important and most underreported metric in RAG systems. The standard approach is to evaluate recall@K: for a test set of questions with known relevant documents, what fraction of the time does the relevant document appear in the top-K retrieved results? Recall@5 above 85% is a reasonable target for a well-tuned retrieval system. Below 70%, the retrieval is the primary quality bottleneck and no amount of LLM quality improvement will compensate for missing the relevant context. Measure retrieval and generation quality separately — this diagnostic enables targeted improvement of whichever component is the actual bottleneck, rather than the common pattern of optimising generation when retrieval is the real problem.
Metadata Filtering: Precision Beyond Semantics
Pure semantic search finds semantically similar content regardless of when it was written, which product it applies to, or which department it belongs to. For many real applications, these filters matter as much as semantic similarity. A customer asking about a product’s return policy should get results about their specific product tier, not semantically similar results about a different product’s policy. Metadata filtering combines vector similarity with structured attribute filters — run ANN search on the vector index while filtering for chunks where product_tier = "premium" and document_type = "policy". All major vector databases support metadata filtering natively. Designing your metadata schema thoughtfully — what attributes will users implicitly filter by, even if they do not state them explicitly? — is an important part of retrieval system design that is easy to retrofit poorly and costly to redo once the corpus is indexed.
Production Considerations for Embedding Pipelines
Embedding a large document corpus is a one-time cost, but keeping it current as documents are added, updated, or deleted requires an ongoing incremental embedding pipeline. Design this from the start: when a document changes, identify which chunks were affected, re-embed those chunks, and update or delete the old vectors in the index. Full re-indexing of a large corpus on every change is impractical beyond small corpora. Incremental updates require document identity tracking (each chunk must know which document it came from and what version), deletion support in your vector store, and an event-driven or batch update process that triggers re-embedding on document changes. These are not complex engineering problems, but they are often not thought through until a stale corpus causes visible quality problems in production — at which point the retrofit is more painful than the upfront design would have been.