What Is a Token?
Every LLM input is converted into tokens before the model processes it. A token is not a word, not a character, and not a syllable — it is a unit that falls somewhere between characters and words, determined by the tokeniser’s vocabulary. In practice, common English words are typically one token each. Less common words are split into two or three tokens. Non-English text tends to use more tokens per word than English. Whitespace, punctuation, and numbers each have their own tokenisation rules. Understanding this is not academic — it directly affects how you design prompts, estimate costs, and debug unexpected model behaviours.
The typical rule of thumb is that 1 token ≈ 0.75 words, or roughly 100 tokens ≈ 75 words. A 10-page document of about 5,000 words uses approximately 6,500–7,000 tokens. But these are averages — the actual token count depends heavily on the vocabulary of the text, the specific tokeniser used, and the language.
How Tokenisers Work: Byte-Pair Encoding
Most modern LLMs use Byte-Pair Encoding (BPE) as their tokenisation algorithm. BPE starts with a vocabulary of individual characters (or bytes), then iteratively merges the most frequent pairs of adjacent tokens into new combined tokens. This process continues until the vocabulary reaches a target size — typically 50,000 to 100,000 tokens. The resulting vocabulary contains common words as single tokens, less common words as two or three tokens, and rare words that may be split into many small fragments.
The vocabulary is learned from a large training corpus, which means the tokeniser reflects the statistical patterns of that training data. English text from common internet sources is well-represented and therefore uses tokens efficiently. Code uses tokens reasonably efficiently for common languages like Python, JavaScript, and SQL. Rare languages, technical jargon not in the training data, and strings of unusual characters (long URLs, hash strings, base64 encoded data) all tokenise inefficiently — a single long URL might cost 20–50 tokens that add little semantic value.
Why the Same Word Can Cost Different Tokens
Token count is affected not just by the word itself but by its position and context. Leading whitespace is often included as part of a token: the word “the” preceded by a space may be a different token from “the” at the start of a sentence without a preceding space. Capitalisation matters: “Python” and “python” may be different tokens. This positional and contextual sensitivity means you cannot reliably predict token counts by counting words — you need to use the tokeniser directly to get accurate counts.
Most LLM providers make their tokenisers available as open-source libraries. OpenAI’s tiktoken library provides the exact tokenisers used by GPT models. Anthropic uses a variant of the Claude tokeniser that can be approximated for cost estimation. For precise token counting before an API call — essential for applications that must stay within a context window — always use the provider’s tokeniser directly rather than estimating from word counts.
Tokenisation Across Languages
The tokenisation efficiency gap between languages is one of the most significant practical considerations for multilingual applications. English is the language best represented in most LLM training corpora, and English tokenises most efficiently — typically 1.0–1.3 tokens per word. French, German, and Spanish tokenise at roughly 1.2–1.5 tokens per word. Japanese, Chinese, and Korean, which use different writing systems, typically tokenise at 1.5–2.5 tokens per character using BPE vocabularies trained primarily on English data. Arabic, Hindi, and other non-Latin script languages often tokenise even less efficiently.
The practical consequence: a 1,000-word document in Japanese may cost 2–3x more tokens (and therefore 2–3x more in API fees) than an equivalent document in English. For applications serving multilingual users at scale, this cost difference is meaningful. Models with tokenisers trained on more balanced multilingual corpora — Qwen 2.5 for Chinese, mT5 for many languages — tokenise non-English text significantly more efficiently and may be the right choice for multilingual applications where cost at scale is a primary constraint.
Tokens and Model Behaviour
Understanding tokenisation explains several LLM behaviours that otherwise seem mysterious. Models often struggle with tasks that seem trivial — counting the letters in a word, for example — because they never see individual characters during inference, only tokens. “strawberry” may be a single token, so the model cannot introspect its individual letters without special reasoning. Tasks involving individual characters, spelling, or letter counting are harder for models than tasks involving semantic meaning, precisely because the tokenisation abstraction hides character-level structure from the model’s perspective.
Arithmetic is another area where tokenisation creates subtle difficulties. Numbers are not tokenised uniformly: “100” may be one token, “1234567” may be split into several tokens in a way that does not preserve the numerical structure humans see. This is one reason LLMs are unreliable at precise arithmetic and benefit from tool use (a calculator) for numerical computation. The model is not reasoning about numbers the way you might expect — it is reasoning about sequences of tokens that happen to represent numbers, which is a very different cognitive operation.
Repeated tokens are another tokenisation effect that affects model behaviour. When a context contains many repetitions of the same token or token sequence — a very long repeated pattern, for example — some models degrade in quality as the repetition accumulates. This “repetition degradation” is a known failure mode linked to the attention mechanism’s difficulty distinguishing semantically meaningful repetition from noise.
Token Limits and Context Windows
Every LLM has a maximum context window measured in tokens — not words, not characters, but tokens. Claude’s 200,000-token context window holds approximately 150,000 words of English text, or roughly 500 pages. Gemini’s 1-million-token window holds approximately 750,000 words. These numbers assume average English prose; technical text with many numbers, URLs, and code may tokenise less efficiently and fit fewer “words” within the same token budget.
When a conversation or document exceeds the context window, something must be truncated or compressed. Different systems handle this differently: some truncate the oldest content, some summarise, some raise an error. Understanding that the limit is in tokens rather than words means you cannot reliably predict whether a piece of content will fit simply by counting words — particularly for non-English text, code, or documents with unusual formatting.
Figure 1 — How the Same Text Tokenises Differently
Practical Implications for Prompt Design
Tokenisation knowledge makes you a more effective prompt engineer in several concrete ways. When you are close to a context window limit, knowing how different types of content tokenise differently lets you make informed decisions about what to cut first. URLs, long numbers, and non-English text are expensive; clean English prose is efficient. Removing a single long URL from your context can free as many tokens as several sentences of explanation.
When counting tokens for cost estimation, always use the provider’s tokeniser rather than word counts. For rough estimates, use the 0.75 words-per-token ratio for English — but expect this to be significantly less accurate for code, technical content, or multilingual text. Building a token counting step into your application’s context assembly pipeline is cheap and prevents the class of bugs where prompts occasionally exceed the context limit under production load.
For applications that process user-generated content in unknown languages, budget conservatively. A user who types their query in Japanese or Arabic will consume 2–3x the tokens of an equivalent English query. If your application has a context window budget, design it to accommodate this variation rather than assuming English-level tokenisation efficiency for all users.
The Future of Tokenisation
Tokenisation is an active research area, and the BPE approach dominant today has known limitations that researchers are working to address. Character-level and byte-level models that process text at lower granularity than BPE tokens improve performance on character-level tasks and reduce the multilingual efficiency gap. Learned tokenisers that adapt their vocabulary to the specific domain or language of the deployment data improve efficiency for specialised applications. And end-to-end models that process raw audio or pixels rather than tokenised text are expanding the scope of what counts as an LLM beyond the text domain entirely. The specific tokenisation approach used by today’s models will likely look dated in five years — but the fundamental principle of representing variable-length natural language as fixed-vocabulary discrete units will remain central to how neural language models work for the foreseeable future.
Why Tokenisation Matters for Cost
API pricing is denominated entirely in tokens — not words, not characters, not requests. Every efficiency improvement in tokenisation translates directly to cost reduction. A prompt that uses 800 tokens instead of 1,000 costs 20% less on every single call. At production scale this compounds: one million daily requests at 200 tokens saved per request is 200 million fewer input tokens per day, saving $600/day at Claude Sonnet pricing. Small per-request token savings become large absolute savings at scale, which is why understanding tokenisation — and designing prompts to be token-efficient — is a genuine operational discipline rather than a theoretical concern. Remove URLs that are not needed for the task, use concise phrasing over verbose explanation, prefer structured formats that encode information densely, and always measure actual token counts rather than estimating from word counts. These habits, applied consistently, produce meaningful cost reductions without any sacrifice in output quality.
Figure 1 — How the Same Text Tokenises Differently
Practical Implications for Prompt Design
Token awareness changes how you design prompts in several concrete ways. Verbose preambles that explain what you are about to ask add tokens without adding value — get to the instruction quickly. Repeated field names in JSON-structured prompts cost tokens for every repetition — consider more compact formats for large structured inputs. Long URLs, file paths, and hash strings cost more tokens than their semantic value warrants — consider summarising or omitting them if they are not essential to the task. And white space, while it improves human readability, adds tokens: a prompt that uses extensive indentation and blank lines costs meaningfully more than an equivalent compact one.
None of these mean you should write cryptic, dense prompts — clarity is worth tokens. But being aware of where tokens are going enables deliberate trade-offs rather than accidental waste. The most expensive token in a prompt is one that adds cost without adding information, and there are more of these in typical prompts than most practitioners realise until they actually count them.
Tokens and Model Context Limits
Every model has a maximum context length measured in tokens — not words, not characters. Claude’s 200,000-token context window is roughly equivalent to 150,000 words or about 600 pages of text. GPT-4o’s 128,000-token window is roughly 96,000 words. Gemini’s 1,000,000-token window is roughly 750,000 words. These numbers give you a practical sense of what fits, but remember that both your input and the model’s output count against the context limit — if you fill 195,000 tokens of a 200,000-token context with input, the model can only generate 5,000 tokens of output. For applications where long outputs are expected, leave headroom by keeping input well below the context ceiling.
Token Pricing and Cost Estimation
All major LLM APIs price by the token, separately for input and output. Output tokens typically cost 3–5x more than input tokens because they require sequential generation rather than parallel processing. Accurate cost estimation requires counting tokens before making the API call, not estimating from word counts. The tiktoken library for OpenAI models, available via pip, counts tokens exactly. For Anthropic models, the API provides a token counting endpoint that returns the exact count for any given prompt and model combination. Building token counting into your application’s cost estimation and budget management infrastructure — rather than approximating — is the difference between predictable API costs and surprise bills at month end.