Benchmarking intro text. This guide covers how to measure Ollama performance precisely, what to look for in the numbers, how to compare models and settings systematically, and how Flash Attention improves both speed and memory usage. The goal is to give you concrete numbers rather than vague impressions about what is fast or slow.
Built-in Verbose Stats
The verbose flag prints timing stats after every response. Three key metrics appear: eval rate (generation speed in tokens per second, the primary benchmark metric), prompt eval rate (prefill speed for processing input, relevant with long context or large documents), and load duration (model load time from disk, eliminated by OLLAMA_KEEP_ALIVE). Run it on any prompt to get an immediate baseline before and after any configuration change.
ollama run llama3.2 --verbose "Explain how a transformer model works."Reference Numbers by Hardware
Typical eval rates for 7-8B Q4 models: RTX 4090 at 80-110 t/s, RTX 4080 at 55-80 t/s, M4 Max 64GB at 90-120 t/s, M4 Pro 24GB at 55-75 t/s, CPU-only 16-core desktop at 3-8 t/s. If your numbers are significantly below these for your hardware, something is misconfigured. The most likely causes: model not fully on GPU (check with ollama ps), Flash Attention not enabled, or another process competing for GPU memory or compute.
Figure 1 — Typical Ollama Eval Rates by Hardware (7-8B Q4 Models)
Monitoring GPU Usage
Watching GPU utilisation during inference reveals whether you are compute-bound or memory-bandwidth-bound. On NVIDIA, run nvidia-smi dmon -s u -d 1 to stream metrics every second. The key column is sm% (GPU compute utilisation). During active generation it should approach 100% on a fully utilised GPU. If it stays at 40-70%, you are memory-bandwidth limited rather than compute-limited. This is normal for smaller models and means a faster GPU with the same memory bandwidth will not help much. On Apple Silicon, check Activity Monitor GPU History tab during inference.
Flash Attention: What It Is and How to Enable It
Flash Attention is an optimised attention algorithm that restructures how attention computations are performed to minimise memory reads and writes between GPU high-bandwidth memory and compute units. It does not change the mathematical result of the attention operation, only how it is computed. The practical effects are reduced KV cache memory usage (often 30-50% less at long context lengths) and faster generation (10-30% faster depending on hardware and context length). It is particularly impactful at context lengths of 8192 tokens and above. Enable it before starting Ollama:
export OLLAMA_FLASH_ATTENTION=1 # Linux / macOS
$env:OLLAMA_FLASH_ATTENTION = "1" # Windows PowerShellRestart Ollama after setting the variable. To persist it across reboots, add to your shell profile or systemd service override. Verify it is active by running a generation with –verbose and checking that eval rate improves versus the baseline you measured without it. Flash Attention requires GPU inference to be active and is supported on all modern NVIDIA GPUs (RTX 2000 series onwards) and Apple Silicon via Metal.
Benchmarking Flash Attention Impact
The cleanest way to measure Flash Attention improvement is a controlled before-and-after benchmark using the Python script above. Run five iterations without Flash Attention, enable it, restart Ollama, run five more, and compare means. The improvement is most pronounced at longer context lengths. At num_ctx=2048 you may see only 5-10% improvement. At num_ctx=16384, improvements of 20-35% are common on Apple Silicon and 10-25% on NVIDIA. If you see no improvement at all, verify the variable was set and Ollama was restarted, and confirm GPU inference is active.
Context Length and Its Effect on Speed
Context window size (num_ctx) directly affects inference speed because it determines the size of the KV cache that must be maintained in memory during generation. A larger context means more memory bandwidth consumed on every token generation step, which slows the eval rate. For a fixed model and hardware, the relationship is roughly: doubling context length reduces eval rate by 5-20% depending on whether you are memory-bandwidth or compute-bound. This means that setting num_ctx to what you actually need rather than the maximum supported can meaningfully improve interactive speed. For casual chat and short tasks, num_ctx=4096 gives the best interactive speed. For document work, accept the slower rate at higher context as a necessary trade-off rather than always running at maximum context.
Quantization and Speed
The quantization format of your model affects both memory usage and inference speed. Q4_K_M (the default for most Ollama pulls) is an excellent balance point. Q4_0 is slightly smaller and slightly faster at some quality cost. Q2_K is roughly half the size and 20-30% faster at significant quality cost. Q8_0 is close to full float16 quality at roughly 2x the memory of Q4_K_M, and may be slower than Q4 on some GPUs due to higher memory bandwidth requirements. The practical recommendation: benchmark Q4_K_M and Q4_0 for your specific model and task. The quality difference is small for many practical use cases (classification, summarisation, structured extraction) and Q4_0 can give a meaningful speed boost on memory-tight hardware. Only drop to Q2_K if you need the model to fit in limited VRAM and have validated that quality is acceptable for your specific task.
Building a Comparison Dashboard
If you benchmark multiple models and configurations regularly, it is worth keeping results in a structured file for comparison over time. Extend the Python benchmark script to append results to a JSON file with timestamps, hardware info, and configuration details. Over a few weeks of testing, this creates a reference document that makes model selection decisions data-driven rather than based on vague recollections of which model felt faster. Include: model name and tag, quantization, num_ctx, Flash Attention enabled or not, hardware (GPU model and VRAM, CPU, RAM), and the full set of run results including mean, stdev, min, max. When you get new hardware, run the same benchmark suite to get comparable numbers. When you see a new model release that looks promising, run it against your existing benchmark suite before committing to it as your daily driver. This discipline of measurement before and after changes is what separates informed configuration decisions from cargo-culting settings from blog posts without knowing whether they actually help on your specific setup.
Figure 2 — Flash Attention Impact on Speed and Memory by Context Length
Using ollama ps for Real-Time Monitoring
The ollama ps command shows what is loaded in memory right now and how memory is being used. During a benchmark run, open a second terminal and watch it update. The PROCESSOR column shows the GPU vs CPU split, SIZE shows total memory in use including the KV cache, and UNTIL shows when the model will unload. This real-time view is more informative than relying solely on post-generation stats because it shows you the steady-state memory usage as context grows during a conversation, which can differ significantly from the initial load size shown by ollama list. If PROCESSOR shows partial CPU offloading unexpectedly, the model or its context is pushing against VRAM limits. Reducing num_ctx or switching to a lower quantization should push more layers back onto GPU.
Comparing Across Models: A Practical Framework
When benchmarking to choose between models for a specific task, structure your comparison so the results are actionable. Define your evaluation criteria before running benchmarks: minimum acceptable quality (does it correctly complete 90% of your test cases?), minimum acceptable speed (does it respond in under X seconds for your use case?), and maximum acceptable memory (does it fit in your available VRAM/RAM?). Run both quality evaluation and speed benchmarks on the same set of 20-50 representative prompts from your actual use case rather than generic benchmark prompts. A model that scores higher on MMLU but produces worse outputs on your specific tasks is not the right model for your use case. The combination of quality validation and speed measurement on representative workloads is what makes benchmark results actually useful for decision-making rather than an academic exercise.
The Minimum Benchmark Checklist
For anyone setting up Ollama and wanting to confirm everything is optimised, a five-step checklist covers the most impactful items. First, confirm GPU inference with ollama ps during a generation run. Second, enable OLLAMA_FLASH_ATTENTION=1 and restart. Third, check your num_ctx setting is appropriate for your use case rather than the maximum possible. Fourth, run the Python benchmark script above on your primary model with five iterations and record the mean eval rate. Fifth, check nvidia-smi or Activity Monitor during a benchmark run to confirm GPU utilisation is high. If any of these five checks reveals a problem, fixing it should be your first priority before any other optimisation. The gains from these fundamentals — going from CPU to GPU, enabling Flash Attention, right-sizing context — are almost always larger than the gains from any hardware upgrade or exotic configuration change. Get the basics right before reaching for more complex solutions.