If your Ollama inference feels slow, there are usually several things you can do about it. The gains available vary enormously by hardware and configuration — some users get 2–3x faster inference from a single setting change, others find they’re already close to their hardware ceiling. This guide covers every meaningful optimisation, starting with the highest-impact ones, so you can work through them systematically and measure the improvement at each step.
Measure First: Know Your Baseline
Before optimising, establish a baseline. The most useful metrics are tokens per second during generation (eval rate) and time to first token (prompt eval time). Get both from Ollama’s verbose output:
ollama run llama3.2 --verbose "Write a short story about a robot."Look for these lines in the output:
prompt eval count: 28 token(s)
prompt eval duration: 1.2s
prompt eval rate: 23.3 tokens/s
eval count: 156 token(s)
eval duration: 8.4s
eval rate: 18.6 tokens/s # this is generation speedRecord eval rate (generation tokens/second) and prompt eval rate (prefill tokens/second). These are your reference points. Every change you make should be followed by re-running this test to quantify the actual improvement.
1. Get the Model Fully on GPU
The single biggest factor in Ollama speed is whether your model is running on GPU or CPU. GPU inference is typically 10–50x faster than CPU for the same model. Check what’s happening with ollama ps:
ollama ps
# NAME ID SIZE PROCESSOR UNTIL
# llama3.2:latest ... 3.1 GB 100% GPU 5 minutes from nowIf PROCESSOR shows anything less than 100% GPU, you’re not getting full GPU inference. Common causes and fixes:
Model doesn’t fit in VRAM: The model is too large for your GPU memory. Options: use a smaller model, use a more aggressively quantized variant (Q2 instead of Q4), or upgrade GPU memory. Check your GPU VRAM: nvidia-smi on NVIDIA, Activity Monitor → GPU History on Mac.
NVIDIA GPU not detected: Ollama can’t find CUDA. Check: nvidia-smi should show your GPU. If Ollama still uses CPU, reinstall with CUDA support: curl -fsSL https://ollama.com/install.sh | sh on Linux, or download the latest Windows installer. Confirm CUDA is installed: nvcc --version.
Apple Silicon not using Metal: On Mac, Ollama uses Metal (Apple’s GPU API) automatically. If inference is slow, confirm you’re running the arm64 binary, not Rosetta: file $(which ollama) should show arm64.
2. Enable Flash Attention
Flash Attention reduces memory usage during inference and speeds up generation, especially at longer context lengths. Enable it before starting Ollama:
# Linux / macOS
export OLLAMA_FLASH_ATTENTION=1
# Windows (PowerShell)
$env:OLLAMA_FLASH_ATTENTION = "1"
# Persist by adding to your shell profile (~/.bashrc, ~/.zshrc)
echo 'export OLLAMA_FLASH_ATTENTION=1' >> ~/.bashrcRestart Ollama after setting the variable. The speed improvement is model and hardware dependent — typically 10–30% faster generation on longer contexts, with less benefit on very short prompts. On Apple Silicon, Flash Attention via Metal provides particularly consistent gains. There’s no quality impact — it’s purely an algorithmic efficiency improvement.
3. Right-Size Your Context Window
Every token in the context window requires memory in the KV cache. A 32K context setting uses far more memory than an 8K setting — and more memory means fewer GPU layers fit, which means slower inference. Set num_ctx to what you actually need rather than the maximum supported:
# For casual chat — 4096 is usually plenty
ollama run llama3.2 --parameter num_ctx 4096
# For coding with moderate context — 8192
ollama run qwen2.5-coder:7b --parameter num_ctx 8192
# Only use 32768+ when you actually need long context
ollama run llama3.1:8b --parameter num_ctx 32768The memory freed by reducing context directly translates to more GPU layers (faster generation) on hardware near its memory limit. On a 8GB VRAM GPU running a 5GB model, reducing context from 32K to 8K can free enough VRAM to run all layers on GPU instead of partially offloading to CPU RAM.
4. Choose the Right Quantization
The quantization format of your model significantly affects both speed and quality. Lower quantization = smaller model = faster inference at some quality cost:
ollama pull llama3.1:8b-instruct-q4_K_M # default ~4.7GB — good balance
ollama pull llama3.1:8b-instruct-q4_0 # slightly smaller, slightly faster
ollama pull llama3.1:8b-instruct-q2_K # smallest ~2.7GB, fastest, quality dropsQ4_K_M is the standard recommendation for a reason — the quality loss over Q8 is minimal for most tasks. But if you’re specifically trying to fit a model in VRAM or get faster inference on constrained hardware, trying Q4_0 (slightly more aggressive than Q4_K_M) or even Q2_K (significant quality trade-off but fits much smaller hardware) can unlock meaningful speed gains.
Figure 1 — Ollama Speed Optimisations: Impact vs Effort
5. Keep Models Loaded (Eliminate Cold Start Latency)
By default, Ollama unloads a model from memory after 5 minutes of inactivity. The next request triggers a full model reload — 15–45 seconds for most 7B models, 45–90 seconds for larger ones. For interactive use, this is painful. Eliminate it by setting a longer keep-alive duration:
# Keep models loaded for 2 hours
export OLLAMA_KEEP_ALIVE=2h
# Or set on a per-request basis via the API
curl http://localhost:11434/api/chat -d '{
"model": "llama3.2",
"keep_alive": "2h",
"messages": [{"role": "user", "content": "Hello"}]
}'
# Keep loaded indefinitely (until manually unloaded)
export OLLAMA_KEEP_ALIVE=-1During active development or usage sessions, set keep_alive to something longer than your typical break between queries. The memory cost of a loaded model is constant — keeping llama3.2 loaded uses the same ~3.1GB whether you’re actively generating or idle. Only set keep_alive=-1 (indefinite) if you’re dedicated to keeping that model in memory permanently and have headroom on your GPU/RAM.
6. Tune Parallel Requests and Batch Size
For applications making many requests (batch processing, API servers with multiple users), Ollama’s parallel processing settings matter:
# Allow 2 parallel requests (doubles throughput on capable hardware)
export OLLAMA_NUM_PARALLEL=2
# Maximum simultaneous models loaded
export OLLAMA_MAX_LOADED_MODELS=2
# Batch size for prompt processing (higher = faster prefill, more VRAM)
# Set via model option in API calls
# "options": {"num_batch": 512} # default is 512, try 1024 on capable GPUOLLAMA_NUM_PARALLEL=2 allows two simultaneous inference requests, effectively doubling throughput for a multi-user or batch processing scenario — but requires roughly double the VRAM to hold two model contexts simultaneously. Only enable this if you have confirmed VRAM headroom. num_batch controls how many tokens are processed simultaneously during the prompt prefill phase — increasing it speeds up time-to-first-token on long prompts at the cost of peak VRAM during prefill.
7. NVIDIA-Specific Optimisations
On NVIDIA GPUs, a few additional settings can improve performance:
# Prefer CUDA over CPU for any layers that can fit
export CUDA_VISIBLE_DEVICES=0 # Use GPU 0 (or 0,1 for multi-GPU)
# For multi-GPU setups — Ollama distributes layers automatically
# Ensure NVLink is enabled if you have compatible GPUs (rtx 3090/4090)
nvidia-smi nvlink --status # check NVLink statusOn Windows, ensure you’re running the latest NVIDIA drivers — Ollama’s CUDA performance is affected by driver version, and drivers from 6+ months ago may show 10–20% lower performance than the current release. Check and update via GeForce Experience or NVIDIA’s driver download page.
8. Apple Silicon Specific Optimisations
Apple Silicon Macs have unified memory — GPU and CPU share the same memory pool, which gives Ollama a significant advantage over discrete GPU setups for models that slightly exceed typical VRAM sizes. A few Mac-specific optimisations:
# Ensure Ollama is using Metal (Apple's GPU API)
# Check in Activity Monitor — Ollama should show GPU usage
# Close memory-hungry applications before running large models
# Safari, Chrome, and Slack can each use 1-2GB
# For M4 Pro/Max/Ultra — all layers should be on Metal automatically
ollama run llama3.1:8b --verbose # confirm gpu_layers matches total layersOn M-series Macs, the biggest performance factor is available unified memory bandwidth. Close memory-heavy applications before running large models — each GB freed gives Ollama more bandwidth and reduces paging. For users on 16GB M-series Macs running 8B models, this can make the difference between comfortable interactive use and noticeable sluggishness when other apps are competing for memory.
Figure 2 — Expected Tokens/Second by Hardware (Q4_K_M, 7-8B models)
9. When You’ve Hit the Hardware Ceiling
Some users work through all these optimisations and find their inference speed is still not where they need it. At that point the remaining options are hardware-level. A GPU upgrade is the most impactful: going from an RTX 3070 8GB to an RTX 4090 24GB roughly triples tokens per second for a 7B model and enables larger models entirely. For Apple Silicon users, the M4 Max and M4 Ultra chips offer dramatically higher memory bandwidth than base M4 Pro, which directly translates to faster inference — an M4 Max at 100+ tokens/second versus an M4 base at 30–40 tokens/second for the same 7B model. Multi-GPU setups using NVLink (RTX 3090/4090 pairs) effectively combine VRAM and bandwidth, enabling faster inference on models that fit the combined pool. If hardware upgrades aren’t feasible, the practical alternative is to permanently move to smaller models — a Qwen 2.5 3B at 80 tokens/second on modest hardware often provides a better user experience than a Llama 3.1 70B at 3 tokens/second, even at lower quality per token. The right balance depends on your specific tasks and quality requirements.
Diagnosing Unexpected Slowness
If Ollama is significantly slower than you expect for your hardware, a few diagnostic steps help identify the cause. Run nvidia-smi dmon -s u (NVIDIA) or check Activity Monitor’s GPU tab (Mac) during inference — GPU utilisation should be near 100% if the model is on GPU. If it’s low, the model may be partially on CPU or another process is competing for GPU resources. Check ollama ps to see the processor split. If inference starts fast and slows down mid-generation, you may be hitting thermal throttling — check GPU temperature with nvidia-smi -q -d TEMPERATURE and ensure adequate cooling. If the first token takes much longer than subsequent ones, the prefill phase is the bottleneck — reduce your prompt length or increase num_batch. System RAM being fully consumed (check with free -h on Linux or Activity Monitor on Mac) causes swapping that devastates performance — close applications to free RAM before running large models.
The Practical Optimisation Checklist
Working through optimisations in the right order maximises improvement per hour of effort. Start with confirming GPU utilisation — if you’re not on GPU, fix that first since nothing else matters until inference is hardware-accelerated. Then enable Flash Attention and measure again. Then right-size your context to what you actually need. Then consider whether a different quantization or smaller model gives you acceptable quality at meaningfully better speed. Only after exhausting these software-level optimisations should you consider hardware changes, which involve more cost and complexity. Most users find that the first three steps — GPU offload, Flash Attention, and right-sized context — give them 80% of the available improvement. The remaining gains from quantization tuning and hardware upgrades are meaningful but diminishing. Document your measurements at each step: knowing that Flash Attention gave you 22% faster generation on your specific hardware and model is useful when you set up a new machine or advise someone else with similar hardware.