What Is a Token? Every LLM input is converted into tokens before the model processes it. A token is not a word, not a character, and not a syllable — it is a unit that falls somewhere between characters and words, determined by the tokeniser’s vocabulary. In practice, common English words are typically one token … Read more
The Structured Output Problem LLMs generate free-form text by default. When your application needs the output in a specific format — JSON for an API response, a list with defined fields, a table with precise columns — getting reliable structured output requires deliberate engineering. The naïve approach of asking the model to “respond in JSON” … Read more
Why Context Compression Matters Even with million-token context windows, there are compelling reasons to compress what you send to the model. Cost scales linearly with input tokens — a 100,000-token prompt costs 10x more than a 10,000-token one. Latency scales with input length, directly affecting user experience in interactive applications. And longer contexts do not … Read more
Why LLM Product Strategy Is Different Building products on top of large language models requires a different strategic framework than traditional software products. The underlying capability is non-deterministic, rapidly improving, and shared by competitors — three properties that make standard product moats (proprietary technology, switching costs, network effects) harder to build and maintain. A product … Read more
The Benchmark Problem Benchmark rankings are the most visible metric in LLM evaluation, and the least useful for choosing a model for a specific production application. MMLU, HumanEval, GSM8K, and their successors measure performance on curated academic test sets under standardised conditions. They tell you how a model performs on a specific type of question, … Read more
The Long Context Revolution Context windows have grown from 4,096 tokens in GPT-3 to 1 million tokens in Gemini 1.5 and 200,000 tokens in Claude. This is not an incremental improvement — it represents a qualitative shift in what LLM applications can do. Tasks that previously required complex chunking pipelines, multi-step summarisation, or external retrieval … Read more
The Unique Security Challenge of LLM Applications LLM applications introduce attack vectors that traditional application security does not address. The core problem is that LLMs process natural language from untrusted sources and generate actions or outputs based on that processing — creating a channel through which adversarial inputs can influence system behaviour in ways that … Read more
The Fine-Tuning Temptation Fine-tuning is the most over-prescribed solution in LLM application development. When a model produces outputs that do not match expectations, the intuitive response is to train it on better examples. But in the majority of cases, the problem is not that the model lacks the capability — it is that the instructions … Read more
Why Model Selection Is Harder Than It Looks The LLM market in 2026 offers dozens of capable models across every price point, capability level, and deployment mode. Benchmark leaderboards exist, but they measure performance on standardised academic tests that may have little correlation with performance on your specific task. A model that ranks first on … Read more
