The Long Context Revolution
Context windows have grown from 4,096 tokens in GPT-3 to 1 million tokens in Gemini 1.5 and 200,000 tokens in Claude. This is not an incremental improvement — it represents a qualitative shift in what LLM applications can do. Tasks that previously required complex chunking pipelines, multi-step summarisation, or external retrieval systems can now be handled by passing the full document, conversation, or codebase directly to the model. Yet most teams using long-context models are not using them effectively. Simply increasing context does not automatically improve results — and in some cases, naive use of large contexts produces worse results than well-designed shorter ones.
This guide covers when long contexts genuinely help, when they hurt, the specific failure modes that emerge at large context sizes, and the patterns that reliably produce better results when you need to operate with tens or hundreds of thousands of tokens.
When Long Context Genuinely Helps
Whole-document analysis. Asking questions about a 200-page document, identifying contradictions across a long contract, or summarising a lengthy report without losing detail from the middle sections — these tasks improve materially with large context. The alternative (chunking and retrieving) loses cross-document coherence and often misses the answer to questions that require synthesising information from multiple sections that a retriever would not return together. When the task requires genuine whole-document understanding, long context is the right tool.
Large codebase understanding. Debugging complex issues that span multiple files, understanding an unfamiliar codebase, performing cross-file refactors, and identifying where a specific behaviour originates in a large project all benefit from loading the full relevant codebase into context. Retrieval-based approaches for code often fail because the right context for a bug is not the most semantically similar code — it is the exact call chain, which a semantic retriever may not identify correctly.
Long conversation continuity. Customer service, tutoring, and complex advisory applications where session context genuinely matters across dozens of turns benefit from large context windows. Instead of summarising and compressing conversation history, you can maintain the full transcript, preserving nuance that summaries lose.
Multi-document comparison and synthesis. Comparing multiple documents — research papers, competitive analyses, legal contracts — and synthesising insights across them is a task that benefits from having all materials in context simultaneously. The model can draw connections, identify contradictions, and synthesise themes that a sequential reading approach would miss.
The Lost-in-the-Middle Problem
The most important thing to understand about long-context LLMs is that they are not uniformly good at using information across the full context window. Research has consistently demonstrated that models perform significantly worse at using information located in the middle of a long context compared to information at the beginning or end — the “lost-in-the-middle” problem. In experiments where the relevant information is placed at different positions in a long context, accuracy is highest when the relevant information is at the very beginning or end, and drops significantly when it is in the middle.
This has practical implications for how you structure your context. The most important information — the instructions, the specific question, the key facts — should appear at the beginning and be repeated or referenced at the end. Information in the middle of a long context is at higher risk of being underweighted. For tasks where position of information matters, design your context layout deliberately rather than assuming the model reads all positions equally.
Latency and Cost at Large Context Sizes
Long contexts are significantly more expensive and slower than short ones. Time-to-first-token scales roughly linearly with input length — a 100,000-token context takes roughly 10x longer to process than a 10,000-token one before generation even begins. For interactive applications, this latency is often unacceptable. And the cost of processing 100,000 input tokens at Claude Sonnet pricing ($3 per million tokens) is $0.30 per request — significant at any meaningful volume. Before defaulting to large contexts, ask whether the full context is actually needed, or whether a smaller, well-selected context would produce equivalent results at a fraction of the cost and latency.
Prompt caching dramatically changes the economics of large contexts when the same content is used repeatedly. If you are asking multiple questions about the same 200-page document, cache the document in the context and pay only 10% of the input token price for subsequent queries. At Claude Sonnet pricing, a 100,000-token cached context costs $0.03 per query rather than $0.30 — a 10x reduction that makes large-context, multi-query workflows economically viable. Design your application to maximise cache hits by keeping the cached prefix stable and appending the user’s question at the end.
When RAG Is Still Better Than Long Context
Long context has made significant advances but does not obsolete RAG for all use cases. RAG remains superior in several specific scenarios. Very large corpora: A corpus of 10 million documents cannot fit in any context window — retrieval is required. Even with 1M token windows, loading the full corpus of a large organisation’s document library is impractical. RAG scales to arbitrary corpus size; long context does not. Frequently updated information: When your document corpus changes frequently, a long-context approach requires re-processing the full context on each update. RAG can update individual documents in the index without reprocessing everything. Citation and source attribution: RAG systems naturally provide the source chunks that contributed to an answer, enabling precise citation. A long-context system that has processed an entire document has no equivalent natural citation mechanism. Cost optimisation at scale: For high-volume applications where each query needs only a small fraction of the available information, retrieving the relevant subset is dramatically cheaper than loading the full corpus for every query.
The practical guidance: use long context when the task genuinely requires whole-document understanding or cross-document synthesis, when the corpus fits comfortably in the context window, and when the cost and latency are acceptable for your use case. Use RAG when the corpus is large, the query typically needs only a small subset of the information, citation is required, or cost at scale is the primary constraint. Hybrid approaches — loading a retrieved subset into a large context alongside the user query — often deliver the best of both.
Strategies for Effective Long-Context Use
Put the task at the beginning and end. Given the lost-in-the-middle problem, sandwich your context: state the task clearly at the start, provide the document content in the middle, and restate the specific question or output requirement at the end. This structure ensures the model encounters the task instructions when its attention is most reliable.
Use structured document formatting. Documents loaded into long contexts benefit from clear section headers, numbered sections, and explicit labelling of what each section contains. “Section 3.2 — Revenue Recognition Policy” is easier for the model to navigate to than an unmarked paragraph. Markdown formatting with headers is processed well by all major LLMs and helps the model orient within a long document.
Break complex multi-document tasks into focused sub-tasks. Rather than loading five documents and asking for a comprehensive synthesis in one pass, break the task: first summarise each document separately (each in its own focused context), then synthesise the summaries. This avoids the lost-in-the-middle problem for the synthesis step and produces more reliable results than one massive context containing all five documents simultaneously.
Validate retrieval before removing it. If you are considering replacing a RAG system with long context for an existing application, evaluate the change rigorously on your actual task distribution before deploying. Long context sometimes performs worse than well-tuned RAG on specific task types, particularly for precise fact retrieval from specific document sections where retrieval precision is high and the document position problem is significant. Measure, do not assume.
Figure 1 — Long Context vs. RAG: When to Use Each
Practical Context Budget Management
Working with large contexts requires deliberate management of the “context budget” — the allocation of tokens across system prompt, documents, conversation history, and the user’s query. A useful approach: define the maximum context size you will use, then allocate it by priority. System prompt gets its allocation first (typically 2–5%). The user’s query and required output format get their allocation (1–3%). The remaining 92–97% is available for document content. Within the document allocation, prioritise by relevance to the current query — if you know which sections are most relevant, put them first and last, with less relevant content in the middle. Explicitly tracking this budget prevents the common failure of accidentally truncating critical content because the context overflowed without warning.
Testing Long-Context Applications
Long-context applications require specific testing strategies that standard LLM evaluation does not cover. Needle-in-a-haystack tests — embedding a specific fact deep in a long document and asking the model to retrieve it — assess whether the model can reliably access information throughout the context, not just at the edges. Multi-hop reasoning tests verify that the model can follow chains of reasoning that cross multiple sections of a long document. Position sensitivity tests place the same information at different positions in the context and measure whether retrieval accuracy changes — if it does, you have a lost-in-the-middle problem in your specific application that requires context structure adjustments. These tests are not replacements for task-specific evaluation but complement it by identifying failure modes that task-level metrics may not surface until they cause problems in production.