How to Speed Up Ollama: GPU, Quantization, and Performance Tuning

Ollama’s default settings are conservative — designed to work on a wide range of hardware without crashing, not to extract peak performance from yours. If you’re getting fewer tokens per second than you expected, or the model is loading slowly, or you’re hitting memory pressure that prevents you from running your preferred model, there’s a good chance the right configuration changes will significantly improve the experience. This guide covers every meaningful lever for improving Ollama’s speed and efficiency, from GPU setup to quantization choice to runtime configuration.

Baseline: Measuring What You Have

Before tuning anything, establish a baseline. Run a generation with --verbose to see exactly what’s happening:

ollama run llama3.2 --verbose "Tell me about transformers in machine learning."

The output includes several critical metrics:

total duration:       8.234s
load duration:        245ms
prompt eval duration: 312ms
prompt eval rate:     45.2 tokens/s
eval duration:        7.677s
eval rate:            28.3 tokens/s    ← this is your generation speed
gpu_layers:           33              ← layers on GPU (33 = all for Llama 3.2 3B)

The two numbers to track: eval rate (generation tokens per second — the main speed metric) and gpu_layers (layers on GPU vs total layers). If gpu_layers is lower than expected, you’re losing speed to CPU offloading. If eval rate is low despite full GPU usage, the bottleneck is GPU memory bandwidth or compute.

GPU Setup: The Biggest Performance Lever

Nothing improves Ollama speed more than running the model fully on GPU. On NVIDIA, confirm CUDA is set up correctly:

# Verify CUDA is detected
nvidia-smi
ollama run llama3.2 --verbose "hi" 2>&1 | grep gpu_layers

If gpu_layers=0, Ollama isn’t using your GPU. Common causes and fixes:

CUDA not installed: Install the CUDA Toolkit matching your driver version. Run nvcc --version to check. Re-run the Ollama installer after installing CUDA — Ollama detects GPU support at install time.

Driver too old: Check your NVIDIA driver version with nvidia-smi. CUDA 12.x requires driver 525+. Update via your OS package manager or NVIDIA’s site.

Model too large for VRAM: If the model exceeds your VRAM, Ollama falls back to CPU. Switch to a smaller quantization (Q4 instead of Q8, or a smaller model variant).

On Apple Silicon, Metal GPU acceleration is automatic — no configuration needed. Confirm with ollama run llama3.2 --verbose "hi" and look for metal in the output.

Quantization: Quality vs Speed Trade-off

Quantization reduces model precision to save memory and improve inference speed. Lower bit width = smaller size, faster inference, slightly lower quality. The practical options:

Q8_0: 8-bit, closest to full precision, largest file. Best quality but requires more VRAM. Use when quality is the priority and you have ample memory.

Q4_K_M: ~4.5 bits, the recommended default. Excellent quality-to-size ratio. Most models in the Ollama library default to this. For a 7B model: ~4.7GB vs 7GB for Q8.

Q4_0: Standard 4-bit. Slightly lower quality than Q4_K_M. Marginally faster. Not recommended unless you specifically need the speed.

Q2_K: 2-bit, smallest footprint. Noticeable quality reduction on complex tasks. Use only if memory forces it — for example, fitting a 7B model on a 4GB GPU.

Speed difference between Q4_K_M and Q8_0 is roughly 30–50% — Q4 generates faster because loading smaller weights from memory is faster. For GPU-bound inference, quantization matters as much as GPU specs:

# Pull a specific quantization
ollama pull llama3.1:8b-instruct-q4_K_M   # default, recommended
ollama pull llama3.1:8b-instruct-q8_0     # higher quality, slower
ollama pull llama3.1:8b-instruct-q2_K     # smallest, fastest, lower quality

Context Length: The Hidden Memory Cost

Every token in the context window requires KV cache memory. A 7B model with 128K context needs significantly more memory than the same model with 4K context — and that KV cache memory competes with model weights for VRAM. Ollama’s default context of 2048 tokens is conservative but memory-efficient. If you’ve increased num_ctx significantly, try reducing it and check whether more GPU layers load:

# Check whether reducing context frees VRAM for more GPU layers
ollama run llama3.1:8b --parameter num_ctx 4096 --verbose "hi"
# vs
ollama run llama3.1:8b --parameter num_ctx 32768 --verbose "hi"

For a 7B model on a 8GB GPU: at 4096 context, the model fully fits in VRAM. At 32768 context, KV cache growth pushes some model layers to CPU RAM, reducing speed significantly. The right context length is the minimum that covers your use case, not the maximum the model supports.

Flash Attention: Free Speed and Memory Gains

Flash Attention is an optimised attention algorithm that reduces memory bandwidth consumption during inference — making it faster and allowing larger context lengths without running out of memory. Enable it with one environment variable before starting Ollama:

# Linux / macOS
export OLLAMA_FLASH_ATTENTION=1

# Windows PowerShell
$env:OLLAMA_FLASH_ATTENTION=1

# Add to your shell profile for permanent effect
echo 'export OLLAMA_FLASH_ATTENTION=1' >> ~/.bashrc

Flash Attention is supported on all modern NVIDIA GPUs (RTX 2000 series and later) and on Apple Silicon via Metal. On older hardware, Ollama silently falls back to standard attention — you won’t see an error, it just won’t use the optimisation. The speedup is most significant at longer context lengths: at 2048 tokens the difference is small (~5–10%); at 32768 tokens it can be 30–40% faster with 40% less KV cache memory. Enable it by default — there’s no reason not to on compatible hardware.

Parallel Requests: Serving Multiple Users

By default, Ollama processes one request at a time. For multi-user setups, set OLLAMA_NUM_PARALLEL to allow concurrent requests:

export OLLAMA_NUM_PARALLEL=2   # allow 2 simultaneous requests

Each parallel slot uses additional memory — roughly the KV cache size for one context window. On a 16GB machine running a 7B model: one parallel slot uses ~5GB for the model + ~0.5GB KV cache; two parallel slots uses the same model weight (shared) + ~1GB KV cache. The model weights aren’t duplicated, only the KV caches. Setting OLLAMA_NUM_PARALLEL=2 typically adds about 500MB–1GB memory usage for a 7B model at default context length — a modest cost for doubling your concurrent request capacity.

Keep-Alive: Eliminating Reload Latency

Ollama unloads models from memory after 5 minutes of inactivity by default. The next request after unloading triggers a cold load (15–45 seconds for 7B models, longer for larger ones). Extend the keep-alive window for better interactive performance:

# Keep models loaded for 1 hour
export OLLAMA_KEEP_ALIVE=1h

# Or per-request via the API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "keep_alive": "2h",
  "prompt": "Hi"
}'

For active development sessions, set keep-alive to 4–8 hours so your primary model stays loaded all day. The memory cost is whatever the model uses when loaded — not additional. Set it back to a shorter duration (or the default) at the end of your session to free memory for other applications.

Multi-GPU Setup on NVIDIA

If you have multiple NVIDIA GPUs, Ollama automatically distributes model layers across them using NVLink (if available) or PCIe. No configuration is needed — Ollama detects multiple GPUs and uses them. To verify multi-GPU usage:

nvidia-smi   # check VRAM usage on each GPU while a model is loaded
ollama run llama3.1:70b --verbose "hi"  # large model to trigger multi-GPU

For best multi-GPU performance, use GPUs with NVLink (RTX 3090/4090 NVLink bridges, professional Ada cards) rather than PCIe-only connections. NVLink dramatically increases inter-GPU bandwidth, which matters when model layers need to exchange activations between GPUs during inference. Two RTX 3090s with NVLink at 48GB combined VRAM can run Llama 3.1 70B at 15–25 tokens per second — comparable to a single high-end datacenter GPU at a fraction of the cost.

CPU Optimisation for CPU-Only Machines

Without a GPU, Ollama uses CPU inference via llama.cpp. A few settings improve CPU performance. Ollama automatically detects and uses AVX2 and AVX-512 instruction sets on modern CPUs — confirm with cat /proc/cpuinfo | grep avx on Linux. If not available, inference will be slower. Set the number of CPU threads with OLLAMA_NUM_THREADS — Ollama defaults to using all physical cores, which is usually optimal, but on machines with many cores (e.g. 32-core workstations), limiting to 16 threads can sometimes improve throughput by reducing cache contention:

export OLLAMA_NUM_THREADS=16

On Apple Silicon CPU-only mode (running without GPU, which you shouldn’t do intentionally), the Metal GPU is significantly faster than the CPU even for small models — make sure Ollama is using Metal by checking the verbose output. On Linux CPU-only machines, use Q4_K_M or smaller quantizations and smaller models (3B class) to keep generation speed at a usable rate. A 7B model on a modern 16-core CPU generates at 3–6 tokens per second — adequate for batch processing where you’re not watching a progress bar, less comfortable for interactive use. For interactive CPU-only use, the 3B model tier (Llama 3.2 3B, Gemma 3 1B) at 8–20 tokens per second is significantly more pleasant.

Benchmarking Your Setup

After making configuration changes, benchmark consistently to measure actual improvement. A simple benchmark script:

import ollama
import time

def benchmark(model: str, prompt: str, n_runs: int = 3) -> dict:
    times = []
    tokens = []
    for _ in range(n_runs):
        start = time.time()
        response = ollama.generate(model=model, prompt=prompt)
        elapsed = time.time() - start
        times.append(elapsed)
        tokens.append(response.get('eval_count', 0))
    avg_time = sum(times) / n_runs
    avg_tokens = sum(tokens) / n_runs
    return {
        'model': model,
        'avg_time': round(avg_time, 2),
        'avg_tokens_per_sec': round(avg_tokens / avg_time, 1),
        'avg_tokens': round(avg_tokens)
    }

results = benchmark('llama3.2', 'Explain the theory of relativity in detail.')
print(results)

Run this before and after each configuration change to confirm the improvement is real and quantify it. The eval_rate from ollama run --verbose is less reproducible than a programmatic benchmark because the verbose mode adds logging overhead. The Python benchmark runs clean inference and gives more consistent numbers for comparison.

Expected Performance by Hardware Tier

Realistic expectations for Ollama generation speed on common hardware setups help you diagnose whether you’re getting the performance your hardware should deliver. For Llama 3.1 8B Q4_K_M with Flash Attention enabled: an RTX 4090 (24GB) should deliver 60–80 tokens per second fully in VRAM; an RTX 4080 (16GB) gives 45–60 t/s; an RTX 3080 (10GB) gives 35–50 t/s. Apple Silicon M4 Max with 48GB+ delivers 50–70 t/s; M4 Pro with 24GB gives 35–55 t/s; M3 MacBook Pro 18GB gives 25–40 t/s. A CPU-only setup with a modern 16-core desktop chip (AMD Ryzen 9, Intel Core Ultra) gives 4–8 t/s. If your numbers are significantly below these ranges with Flash Attention enabled and the model fully in VRAM, something is misconfigured — the most common cause on NVIDIA is the model partially offloading to system RAM due to KV cache pressure from a large num_ctx setting. Reducing context length is usually the first thing to try when NVIDIA GPU speeds are lower than expected, as the KV cache overhead at large contexts is easy to underestimate and has a disproportionate effect on generation speed.