Why Context Compression Matters
Even with million-token context windows, there are compelling reasons to compress what you send to the model. Cost scales linearly with input tokens — a 100,000-token prompt costs 10x more than a 10,000-token one. Latency scales with input length, directly affecting user experience in interactive applications. And longer contexts do not always produce better outputs: the lost-in-the-middle problem means information buried in a very long context is less reliably used than information in a focused, shorter one. Context compression — reducing the token count of your inputs without losing information needed to answer the query — addresses all three problems simultaneously.
Summarisation-Based Compression
The most straightforward approach is summarisation: replace verbose content with a compact summary that preserves key information. This works well for conversation history (replace older turns with a summary of what was established), background context documents (summarise before injecting into the prompt), and retrieved RAG chunks that contain more detail than needed for the specific query. The key design decision is what to preserve and what to discard. A good summarisation prompt is highly specific: “Summarise this document preserving all specific dates, numerical figures, obligations, and parties named. Discard background context, examples, and general statements.” Generic instructions produce generic summaries that may omit exactly the details your downstream query needs. Task-specific instructions preserve the right information.
Hierarchical summarisation handles very long documents by summarising in sections first, then summarising the section summaries into a final compressed representation. This preserves more structure than a single-pass summary and tends to produce more accurate retrieval of specific details because section-level summaries maintain the original document’s organisation.
Selective Extraction
Rather than summarising broadly, selective extraction retains only the information relevant to the current query, discarding everything else. For a query about a contract’s payment terms, you extract only the payment clauses rather than loading the full 50-page contract. For a query about a specific function in a large codebase, you extract only that function and its direct dependencies. Selective extraction requires knowing what is relevant before the LLM has processed the full context. The typical solution is a lightweight first pass: keyword matching, embedding similarity, or a fast cheap model that identifies which sections likely contain the relevant information. This first-pass cost is small relative to the savings from not loading the full document into an expensive frontier model.
Conversation History Compression
In multi-turn conversations, history grows with every exchange. Without management, a long session’s context can reach tens of thousands of tokens. Three techniques manage this effectively. A sliding window keeps only the last N turns — appropriate when recent context is all that matters. Progressive summarisation periodically summarises older turns into a compact representation and replaces them, preserving semantic continuity better than a hard cutoff. Semantic compression identifies which turns established important context (decisions, constraints, key facts) and retains those specifically, discarding low-information turns like acknowledgements and filler exchanges. The right technique depends on your application: sliding window for simple Q&A chatbots, progressive summarisation for advisory or coaching applications, semantic compression for complex multi-session workflows where early-established facts continue to matter throughout.
Token-Efficient Formatting
The way you format information significantly affects its token count. Markdown tables, verbose XML, heavily nested JSON, and repetitive prose all consume more tokens than necessary. Several formatting choices reliably reduce token count without losing information. Plain text representations of structured data (comma-separated rather than JSON with field names repeated for every record) reduce overhead significantly for large datasets. Removing HTML formatting, metadata, headers and footers from documents before passing them to the model eliminates tokens that add no semantic value. Replacing verbose descriptions with concise ones — “The meeting will be held on 2026-07-15 at 14:00 UTC” rather than “We are planning to have our meeting scheduled for the fifteenth day of July, in the year two thousand and twenty-six, at two o’clock in the afternoon, coordinated universal time” — sounds obvious but commonly occurring verbose patterns in documents and system prompts add up significantly at scale.
Prompt Caching: Free Compression for Repeated Prefixes
Prompt caching is the most cost-effective form of context compression for applications where the same large prefix (system prompt, document, or context) is used across many requests. Rather than reducing the token count, caching eliminates the per-request cost of re-processing tokens that have already been computed. Anthropic’s prompt caching bills cached tokens at 10% of the standard input price; OpenAI’s caching discounts vary by tier. For a 50,000-token system prompt with a 90% cache hit rate across 10,000 daily requests, caching reduces the daily input token cost for that prefix from $1,500 to $150 — a $49,500/month saving from a one-line code change. Prompt caching is complementary to other compression techniques: compress your context to the minimum necessary, then cache the stable prefix that appears in every request.
Learned Compression and Embeddings
Beyond manual summarisation and extraction, more sophisticated compression approaches use embedding representations to compress context. AutoCompressor and similar research approaches train models to compress long documents into summary vectors that can be prepended to the context, enabling the model to reason about very long documents at a fraction of the token cost of including them in full. Gist tokens represent a similar approach — learned compression that encodes an entire document’s information into a small number of special tokens. These approaches are at the research frontier and not yet widely deployed in production, but they represent the direction context compression is heading: away from manual summarisation and toward learned representations that preserve more information in fewer tokens than human-readable summaries can.
For production applications today, the practical approaches are manual summarisation, selective extraction, and prompt caching — these are mature, well-understood techniques with clear implementation paths. Track the research on learned compression as it matures; within 12–18 months these approaches are likely to become production-ready and will deliver compression ratios that manual approaches cannot match.
Measuring Compression Quality
The risk of context compression is information loss that degrades output quality. Every compression technique must be validated against your specific task to confirm that the compressed context produces equivalent quality to the full context on your evaluation set. The evaluation methodology: run your full evaluation set with full context, then run it with compressed context, and compare quality scores. A compression approach that reduces token count by 60% but degrades quality by 15% may or may not be acceptable depending on your quality threshold and cost sensitivity. For most applications, a quality degradation of under 5% is acceptable for a token reduction of 50%+ — but this must be measured on your specific task, not assumed from general principles.
Also measure the information that was lost: for which query types did compressed context produce worse results? These failure modes often reveal which information was incorrectly identified as non-essential by the compression step. Refining the compression instructions for those failure cases — adding specificity about what to preserve — iteratively improves compression quality without requiring fundamental architectural changes.
Figure 1 — Context Compression Techniques: Trade-offs at a Glance
Putting It Together: A Compression Stack
The highest-impact context compression strategy combines multiple techniques applied in sequence. First, apply prompt caching to any stable system prompt or document context that appears in every request — this is free efficiency with no quality risk. Second, apply selective extraction to retrieved RAG content, keeping only the tokens directly relevant to the current query. Third, apply progressive summarisation to conversation history beyond the last 5–10 turns. Fourth, apply token-efficient formatting to any remaining content — remove unnecessary markup, verbose field names, and redundant language. Applied together on a typical RAG-based conversational application, these four techniques typically reduce input token costs by 60–80% with under 5% quality degradation on the primary task metrics. Measure the impact at each stage to confirm that quality is preserved and to identify where additional refinement is needed. Context compression is not a one-time optimisation but an ongoing discipline — as your application evolves and your context composition changes, revisit the compression strategy to confirm it remains optimal for your current context structure.
Compression for Agentic Systems
Agentic systems that execute multi-step tasks present a particular context management challenge: the agent’s working memory — the record of what it has done, what it found, and what remains to do — grows with every action. Without compression, a long-running agent accumulates thousands of tokens of intermediate state that consumes context space needed for new information. Effective agent context management applies compression selectively to intermediate state: summarise completed steps into a compact record of outcomes rather than retaining the full detail of every action taken. Keep only the information the agent needs to complete the remaining steps, not a full audit trail of everything that has happened. This requires the agent to be designed with explicit state management — what information must be preserved, what can be compressed, and what can be discarded entirely — rather than simply appending everything to the context and hoping it fits. Agents designed this way can run significantly longer task sequences within a fixed context window than agents that accumulate context naively.
The Economics: Making the Case Internally
Context compression is an engineering investment that pays returns in reduced API costs and lower latency. Making the case for that investment requires quantifying the opportunity. Calculate your current average input token count per request, your daily request volume, and your input token cost. Apply a conservative 50% compression ratio estimate to get the projected savings. For most production LLM applications beyond early-stage, this calculation produces numbers that justify dedicated engineering investment in compression. At 1 million daily requests with an average of 5,000 input tokens at Claude Sonnet pricing, input costs are $15,000 per day. A 50% compression reduction saves $7,500 per day — $2.7 million per year. Even a 20% compression at that scale saves over $1 million annually. Frame the investment in these terms, with conservative estimates, and the case for context compression as a priority engineering project becomes straightforward for any production system operating at meaningful scale.
The teams that build context compression infrastructure early — before cost pressures make it urgent — do so at lower cost and with less disruption than those that retrofit compression onto an established application under budget pressure. Start with prompt caching (zero risk, immediate return), measure your current context composition, and plan your compression roadmap based on where the largest token costs reside. Context compression, practised consistently, is one of the compounding operational disciplines that separates LLM applications with healthy unit economics from those that struggle to scale profitably.