Why Context Compression Matters Even with million-token context windows, there are compelling reasons to compress what you send to the model. Cost scales linearly with input tokens — a 100,000-token prompt costs 10x more than a 10,000-token one. Latency scales with input length, directly affecting user experience in interactive applications. And longer contexts do not … Read more
Why LLM Product Strategy Is Different Building products on top of large language models requires a different strategic framework than traditional software products. The underlying capability is non-deterministic, rapidly improving, and shared by competitors — three properties that make standard product moats (proprietary technology, switching costs, network effects) harder to build and maintain. A product … Read more
The Benchmark Problem Benchmark rankings are the most visible metric in LLM evaluation, and the least useful for choosing a model for a specific production application. MMLU, HumanEval, GSM8K, and their successors measure performance on curated academic test sets under standardised conditions. They tell you how a model performs on a specific type of question, … Read more
The Long Context Revolution Context windows have grown from 4,096 tokens in GPT-3 to 1 million tokens in Gemini 1.5 and 200,000 tokens in Claude. This is not an incremental improvement — it represents a qualitative shift in what LLM applications can do. Tasks that previously required complex chunking pipelines, multi-step summarisation, or external retrieval … Read more
The Unique Security Challenge of LLM Applications LLM applications introduce attack vectors that traditional application security does not address. The core problem is that LLMs process natural language from untrusted sources and generate actions or outputs based on that processing — creating a channel through which adversarial inputs can influence system behaviour in ways that … Read more
The Fine-Tuning Temptation Fine-tuning is the most over-prescribed solution in LLM application development. When a model produces outputs that do not match expectations, the intuitive response is to train it on better examples. But in the majority of cases, the problem is not that the model lacks the capability — it is that the instructions … Read more
Why Model Selection Is Harder Than It Looks The LLM market in 2026 offers dozens of capable models across every price point, capability level, and deployment mode. Benchmark leaderboards exist, but they measure performance on standardised academic tests that may have little correlation with performance on your specific task. A model that ranks first on … Read more
The Decision That Shapes Everything Downstream Choosing between open-source and proprietary LLMs is not primarily a technical decision — it is a strategic one that determines your cost structure, your data governance posture, your dependency on external vendors, your ability to customise, and the ceiling on what you can build. Get it wrong and you … Read more
Beyond Code Completion: The Full Picture Most discussions of LLMs in software engineering focus on code completion tools like GitHub Copilot. These are genuinely useful, but they represent only a fraction of where LLMs are transforming how engineering teams work. The broader impact spans the entire software development lifecycle: requirements analysis, architecture decision-making, code review, … Read more
