Beyond the Basics: Why Advanced Cost Optimisation Matters
Most teams deploying LLMs learn the obvious cost levers quickly: use cheaper models for simple tasks, enable prompt caching, keep prompts concise. These basics can reduce costs by 30–50%. But the teams achieving 70–90% cost reductions without quality loss are applying a more systematic, layered approach that treats cost optimisation as an engineering discipline rather than a set of one-time tweaks. This guide covers the advanced techniques that separate serious cost-optimised LLM systems from those that have done the obvious things and stopped.
Cost Attribution: You Cannot Optimise What You Cannot Measure
Before optimising, you need complete visibility into where costs come from. Most teams track total monthly API spend but not cost per feature, per user cohort, per model tier, or per pipeline stage. Without this breakdown, you are optimising blindly — making changes that may reduce costs in one area while a bigger cost driver goes unaddressed. Build cost attribution into every LLM call from day one: log the model used, the endpoint or feature that triggered it, the user segment (free vs paid vs enterprise), and the token breakdown (system prompt, conversation history, RAG context, user input, output). Aggregate this by feature weekly. The first time you see the breakdown, the largest cost driver is almost always not what you expected.
In practice, three patterns emerge repeatedly. First, one feature — often a background or automated feature rather than a core user-facing one — consumes 40–60% of total spend because it runs at high frequency on large contexts. Second, the system prompt is larger than it needs to be and not cached, so it is processed afresh on every call. Third, a model tier is being used uniformly when 80% of calls do not require that tier’s capability. Each of these has a specific fix, and finding them requires the cost attribution data.
Intelligent Model Routing at Scale
Routing each request to the appropriate model tier is one of the highest-leverage cost optimisations available, but doing it well requires more than a simple rule. Naive routing — route short queries to cheap models, long queries to expensive ones — captures only a fraction of the available savings. Effective routing classifies requests by complexity and required capability, not just surface features like length. A two-sentence query that requires deep reasoning goes to a frontier model; a two-paragraph query that is a simple extraction goes to an economy model. Building this classification requires: defining what “complex” means for your specific task distribution, creating a labelled dataset of requests with their appropriate model tier, and training or prompting a lightweight classifier that makes the routing decision.
The classifier itself should be fast and cheap — a small model or even a rule-based system that takes under 50ms to make the routing decision. The value is asymmetric: routing a complex request to an economy model and getting a wrong answer costs more in user impact than the API savings are worth, but routing a simple request to an economy model when a frontier model was not needed saves meaningfully at scale. Calibrate your classifier to be conservative — when in doubt, route up — and monitor the quality of economy-tier responses to identify cases where your classification boundaries need adjustment.
Request Deduplication and Semantic Caching
Prompt caching handles repeated identical prefixes. Semantic caching handles the broader case: requests that are semantically similar but not textually identical, where the same response would serve both. “What is your refund policy?” and “How do I get a refund?” are different strings that may have the same answer. A semantic cache embeds incoming queries, finds the nearest cached response above a similarity threshold, and returns the cached response without an API call. For applications where users frequently ask the same types of questions in different words — customer service, internal Q&A, FAQ systems — semantic caching can eliminate 20–40% of API calls entirely.
Implementation: embed each incoming query using a fast, cheap embedding model. Search your semantic cache (a vector database keyed by query embeddings) for the nearest cached response. If the similarity score exceeds your threshold (typically 0.92–0.95 for factual Q&A), return the cached response. If not, make the API call, cache the response with its embedding, and return it. Set cache TTLs based on how frequently the underlying information changes — short for news-related queries, long for stable policy or documentation queries. Monitor cache hit rates and adjust the similarity threshold to balance precision (not returning wrong cached answers) against recall (maximising cache hits).
Figure 1 — The LLM Cost Optimisation Stack
Output Length Control at Scale
Output tokens cost 3–5x more than input tokens and are directly controllable through prompt design — yet many teams leave output length largely unconstrained. At production scale, reducing average output length by 30% saves more than optimising input prompts, because output tokens are more expensive and the reduction applies multiplicatively across every request. Several techniques reliably reduce output length without degrading quality. Explicit length instructions (“respond in 2–3 sentences for simple questions, one paragraph for complex ones”) work well when the model follows instructions reliably. Output format constraints (requesting JSON instead of prose for classification or extraction) naturally limit verbosity. And max_tokens limits provide a hard ceiling that prevents runaway verbose responses — set it at 150% of your typical expected response length to allow legitimate long responses while capping outliers.
Monitor your output token distribution, not just the average. A bimodal distribution — most responses are 100–200 tokens but 5% are 2,000+ tokens — suggests that a small number of queries are generating disproportionately expensive responses. Identify those query types, understand why they generate long responses, and either add specific length constraints for those cases or route them to a cheaper model where the long response costs less per token.
Batch Processing: 50% Off for Async Workloads
The Batch APIs from Anthropic and OpenAI process requests asynchronously at exactly half the standard on-demand price. The model, quality, and capabilities are identical — the only difference is that results are available within 24 hours rather than immediately. For any workload that does not require real-time responses, this is the single largest cost reduction available as a single architectural change. Nightly document processing, offline classification pipelines, bulk data enrichment, model evaluation runs, and large-scale content generation are all candidates. The implementation is straightforward: submit a JSONL file of requests, poll for completion, retrieve results. The 50% saving applies immediately, requires no quality compromise, and scales linearly with volume.
Fine-Tuning Small Models for High-Volume Tasks
For applications with one or two dominant high-volume task types that are well-defined and stable, fine-tuning a small model to match the quality of a larger one is often the best long-term cost strategy. A fine-tuned Llama 8B or Phi-4 14B model that achieves 95% of Claude Sonnet’s quality on your specific task costs 20–50x less to run at inference time. The upfront fine-tuning investment — data collection, training, evaluation — is typically recovered within weeks at production volume. The break-even calculation: if your application makes 5 million daily requests at $0.003 each on Sonnet, you pay $450,000 per month. The same requests on a self-hosted fine-tuned 8B model cost roughly $30,000–$50,000 per month in compute. A $100,000 fine-tuning project breaks even in under two weeks at that scale.
The prerequisite is that the task must be narrow and stable enough to fine-tune effectively — as discussed in the fine-tuning guide, this requires at least 500–1,000 high-quality training examples and a task definition that does not change frequently. For tasks that meet these criteria, fine-tuning + self-hosting is the terminal state of cost optimisation for mature, high-volume applications.
The Compounding Effect of Layered Optimisation
Each optimisation layer compounds with the others. Semantic caching eliminates 30% of requests before any other optimisation applies. Of the remaining 70%, intelligent routing sends 60% to economy models at one-tenth the cost. Prompt caching reduces input token costs by 85% for the cached prefix across all remaining requests. Context compression reduces remaining input tokens by 40%. Output length controls reduce output tokens by 25%. Batch processing handles 40% of remaining volume at 50% off. Applied together, these layers reduce total API spend by 75–90% compared to an unoptimised baseline using a single mid-tier model with no caching. The compound saving is significantly larger than any individual optimisation — which is why building the measurement infrastructure first, then addressing each layer systematically, consistently outperforms ad-hoc individual optimisations applied without a complete picture of the cost structure.
Provider Negotiation and Commitment Discounts
At sufficient scale, API cost optimisation extends beyond technical architecture into commercial negotiation. All major LLM providers offer discounted rates for committed spend or volume commitments — typically 20–40% below standard on-demand pricing for annual commitments. These discounts are not advertised on pricing pages; they require a conversation with enterprise sales. If your organisation is spending more than $50,000 per month on LLM APIs, a conversation about volume commitments is almost certainly worth having. The discount negotiated on a committed annual spend typically pays back within months and requires no technical changes. Combine committed spend discounts with the technical optimisations above, and the total cost reduction relative to a naive unoptimised deployment often exceeds 85%.
Monitoring and Preventing Cost Regressions
Cost optimisation without ongoing monitoring is a one-time event rather than a sustained discipline. Three types of cost regressions are common: prompt changes that increase average token count, new features that bypass the existing routing or caching infrastructure, and traffic growth that outpaces the capacity of caching and batching systems. Monitor your cost-per-request metric continuously, not just total monthly spend. A rising cost-per-request with stable traffic indicates a quality regression — some efficiency mechanism has stopped working or a change has increased the token footprint. Alert when cost-per-request rises more than 10% week-over-week and investigate before the change compounds. The discipline of treating cost-per-request as a primary engineering metric — as important as latency and error rate — is what separates teams that maintain efficient operations at scale from those that periodically discover they have an unexpected six-figure monthly API bill.