Ollama Context Length Guide: How to Set num_ctx and Which Models Support Long Context

by

One of the most common surprises when using Ollama is discovering that your model has a context length of 2048 tokens by default — even if the model itself supports 128K. Ollama’s default is conservative to protect users on limited hardware, but for most people the right context length is much higher. This guide explains what context length actually means, how to set it correctly, how much memory each context size costs, and which models offer the best long-context performance locally.

What Context Length Actually Means

Context length — also called context window — is the total number of tokens a model can “see” at once during inference. This includes your system prompt, the entire conversation history, any documents you’ve pasted in, and the response being generated. When you exceed the context length, the oldest tokens get dropped from the window. The model can no longer reference earlier parts of the conversation or document.

A token is roughly 0.75 words in English, so:

  • 2,048 tokens ≈ 1,500 words — about 3 pages
  • 8,192 tokens ≈ 6,100 words — about 12 pages
  • 32,768 tokens ≈ 24,500 words — about 50 pages
  • 128,000 tokens ≈ 96,000 words — about 200 pages

For casual Q&A and short coding tasks, 2048 tokens is often sufficient. For document analysis, long conversations, or pasting in large code files, you need much more. The mismatch between Ollama’s 2048 default and what modern models support is the source of a lot of confusing truncation behaviour.

How to Set Context Length in Ollama

There are three ways to set num_ctx in Ollama, each with different scope:

Per-session via CLI flag (temporary):

ollama run llama3.2 --parameter num_ctx 8192

Sets context length for this run only. Reverts to default next time you run the model.

Inside a chat session (in-session):

>>> /set parameter num_ctx 16384

Type this at the >>> prompt during an active chat session. Takes effect immediately for subsequent messages in that session.

Via Modelfile (persistent per-model config):

cat > Modelfile << 'EOF'
FROM llama3.2
PARAMETER num_ctx 16384
EOF
ollama create llama3.2-long -f Modelfile

Creates a named variant of the model with your preferred context length baked in. ollama run llama3.2-long always uses 16384 tokens. This is the cleanest approach if you always want the same context length for a specific model.

Via the API (per-request):

import ollama

response = ollama.chat(
    model='llama3.2',
    messages=[{'role': 'user', 'content': 'Your prompt here'}],
    options={'num_ctx': 32768}
)

Memory Cost of Context Length

Every token in the context window requires memory in the KV (key-value) cache. This memory is allocated at inference time and grows as you send more tokens. The formula is roughly:

KV cache memory ≈ num_ctx × num_layers × 2 × hidden_size × bytes_per_element

In practice, the memory cost per 1K context tokens varies by model size:

  • 3B model: ~0.5 GB per 8K context
  • 7B model: ~0.8 GB per 8K context
  • 13B model: ~1.2 GB per 8K context
  • 70B model: ~5 GB per 8K context

These are in addition to the base model memory. A Llama 3.1 8B model with 4.7GB base size and 32K context needs roughly 4.7 + 3.2 = ~8GB total. On hardware with tight memory, setting context too high causes OOM errors mid-conversation as the KV cache grows. Start with 8192, which covers most practical use cases without excessive memory overhead.

Figure 1 — Context Length vs Memory Overhead by Model Size

Model Base size +8K ctx +32K ctx +128K ctx Total (128K) Llama 3.2 3B Q42.0 GB+0.3 GB+0.8 GB+3.2 GB~5 GB Llama 3.1 8B Q44.7 GB+0.8 GB+3.2 GB+12 GB~17 GB Mistral 7B Q44.1 GB+0.7 GB+2.8 GB+11 GB~15 GB Phi-4 14B Q48.9 GB+1.2 GB+5 GB+20 GB~29 GB Llama 3.1 70B Q443 GB+5 GB+20 GB+80 GB~123 GB! KV cache overhead is additive — large context on large models can exceed system memory quickly

Context Length Support by Model

Not all models support the same maximum context length. Here's what the most commonly used Ollama models actually support:

128K context (current generation): Llama 3.1/3.2/3.3, Gemma 3, Phi-4, Qwen 2.5, Mistral Small 3, Gemma 2. These support up to 128K tokens — though running the full 128K context requires substantial memory overhead as shown above.

32K context: Mixtral 8x7B, Mistral 7B (v0.2+), CodeLlama, some older Qwen models. Still very capable for most practical use cases.

8K or less: Older models like the original Mistral 7B v0.1, LLaVA, some CodeLlama variants. For these, 8K is already the ceiling.

The key thing to know: Ollama defaults to 2048 tokens regardless of what the model supports. You always have to opt in to longer context. The model won't tell you it was truncated — it simply "forgets" earlier parts of the conversation once the window fills up, which can cause confusing non-sequitur responses in long conversations.

Recommended Context Settings by Use Case

Casual chat and simple Q&A: 4096 tokens is plenty. Most questions and answers fit well within this, and the memory overhead is minimal.

ollama run llama3.2 --parameter num_ctx 4096

Coding sessions: 8192–16384 tokens. You need room for the code file you're discussing, your prompt, and the response. A 300-line Python file is about 3000 tokens — 8192 gives you comfortable headroom.

ollama run qwen2.5-coder --parameter num_ctx 16384

Document analysis (moderate): 16384–32768 tokens. Covers most articles, reports, and papers. A 10,000-word research paper is about 13,000 tokens.

ollama run llama3.1:8b --parameter num_ctx 32768

Long document analysis / book chapters: 65536–131072 tokens. Only practical on models that support it and hardware with 16GB+ free memory after the model loads.

ollama run llama3.1:8b --parameter num_ctx 65536

Full book or very large codebase: 131072 tokens. This is the maximum most current models support. Memory cost is severe — only attempt on 32GB+ RAM or unified memory with a 7B or smaller model.

Fastest Ollama Models for CPU Inference

If you're running without a GPU — or with a GPU that can't fit larger models — speed is dominated by model size. Smaller models generate faster on CPU because they have fewer parameters to process per token and fit better in CPU cache. Here's how the main options stack up on a modern 16-core CPU (e.g. AMD Ryzen 9 7950X or Apple M4):

Sub-2B models (5–15+ tokens/second on CPU):

  • Phi-4-mini (3.8B): ~6–10 t/s. Best quality-to-speed ratio in this class. Strong reasoning for the size.
  • Qwen 2.5 1.5B: ~12–18 t/s. Very fast, basic capability.
  • Gemma 3 1B: ~15–25 t/s. Fastest Ollama model worth running — surprisingly capable for simple tasks.
  • SmolLM2 1.7B: ~12–20 t/s. Efficient architecture, good for lightweight automation.

3B–4B models (4–10 tokens/second on CPU):

  • Llama 3.2 3B: ~6–10 t/s. The best 3B model for general use — balances speed and quality well.
  • Gemma 3 4B: ~5–8 t/s. Multimodal, strong instruction following, worth the slight speed cost.
  • Phi-4-mini: ~6–9 t/s. Excellent reasoning at this size, punches well above weight on structured tasks.

7B–8B models (2–6 tokens/second on CPU):

  • Llama 3.1 8B: ~3–5 t/s. The workhorse. Excellent quality, acceptable CPU speed for non-interactive use.
  • Mistral 7B: ~4–6 t/s. Slightly faster architecture than Llama 3.1, good general performance.
  • Qwen 2.5 7B: ~3–5 t/s. Strong multilingual and coding, similar speed to Llama 3.1 8B.

For interactive CPU-only use, 3B is the practical sweet spot — fast enough to feel responsive, capable enough for real tasks. The 7B models are better quality but the 2–5 t/s range starts to feel slow for back-and-forth conversation. Use 7B for batch processing and 3B for interactive use when you're on CPU only.

Figure 2 — CPU Inference Speed by Model Size (16-core CPU)

Model Size (Q4) CPU t/s Good for gemma3:1b815 MB15–25 t/sFast chat, edge tasks llama3.2:3b2.0 GB8–12 t/sInteractive chat phi4-mini2.2 GB6–10 t/sReasoning, structured tasks mistral:7b4.1 GB4–6 t/sQuality + speed balance llama3.1:8b4.7 GB3–5 t/sBatch, non-interactive Speeds on AMD Ryzen 9 7950X (16-core). Apple M4 CPU-only is 30–50% faster.

Flash Attention: Reducing Context Memory Cost

Flash Attention is an optimised attention algorithm that significantly reduces memory usage during long-context inference — without changing output quality. It restructures how attention computations are done to minimise data movement between GPU HBM and SRAM, which also speeds up generation on long contexts.

To enable it in Ollama:

# Set before starting Ollama (all platforms)
export OLLAMA_FLASH_ATTENTION=1   # Linux / macOS
# or
$env:OLLAMA_FLASH_ATTENTION=1    # Windows PowerShell

Then restart Ollama. Flash Attention is particularly impactful when num_ctx is 8192 or higher. At 32K context it can reduce KV cache memory by 30–50% and speed up generation by 10–30% depending on hardware. It requires a compatible GPU — all modern NVIDIA cards from RTX 2000 series onwards support it, as do Apple Silicon chips via Metal.

For most users on modern hardware, enabling Flash Attention is a pure win for long-context use: less memory pressure and faster generation. The only reason not to enable it is if you encounter compatibility issues (rare) — in which case disabling it returns to the default attention implementation.

Practical Context Length Strategy

The right approach is to start with 8192 for everyday use — it handles most conversations, coding sessions, and short documents without excessive memory overhead. Enable Flash Attention for better efficiency. Scale up to 16384 or 32768 when you're working with specific long documents or large code files, and set it just for those sessions rather than globally. Only push to 64K+ if you're specifically doing whole-book analysis or very large codebase review and have the hardware to support it comfortably.

Monitor actual context usage during long sessions with ollama ps — the output shows how much of the context window is currently occupied. When you're at 80%+ of your num_ctx, the model is about to start dropping early context. Either summarise the conversation at that point, or increase num_ctx if you have the memory headroom. Building this awareness into how you work with local LLMs prevents the confusing "why did the model forget what I said earlier" experience that catches people off guard when they first hit the context limit.

Leave a Comment