Ollama on Windows runs on CPU by default, but if you have an NVIDIA GPU, you’re leaving a lot of performance on the table. With CUDA enabled, a 7B model that generates 4–6 tokens per second on CPU can hit 50–80 tokens per second on a mid-range GPU — the difference between something that feels sluggish and something that feels instant. This guide covers exactly how to set up CUDA for Ollama on Windows, verify it’s working, and troubleshoot the common issues that trip people up.
What You Need Before You Start
CUDA acceleration with Ollama requires three things on the Windows side:
An NVIDIA GPU. Any NVIDIA card from roughly the GTX 1000 series onwards supports CUDA. The more VRAM you have, the larger the models you can run fully on GPU — 8GB VRAM handles 7B models, 16GB handles 13B, 24GB handles 30B+.
An up-to-date NVIDIA driver. Ollama requires CUDA 11.3 or later, which corresponds to NVIDIA driver version 452.39 or higher. Most people on reasonably modern machines will already meet this — but if you haven’t updated your GPU driver recently, now is a good time. Download from nvidia.com/drivers, not via GeForce Experience (the full driver package is more reliable).
Ollama installed on Windows. If you haven’t installed it yet, download from ollama.com and run the installer. The installer automatically detects your NVIDIA GPU and installs the necessary CUDA runtime libraries alongside Ollama.
You do not need to separately install the CUDA Toolkit or cuDNN. Ollama bundles the CUDA runtime it needs — the only dependency on your side is having a compatible NVIDIA driver installed.
Verifying Your CUDA Setup
Before touching Ollama, confirm your NVIDIA driver is working correctly. Open a terminal (PowerShell or Command Prompt) and run:
nvidia-smiYou should see a table showing your GPU model, driver version, CUDA version, and current memory usage. If this command isn’t found or returns an error, your GPU driver isn’t installed or isn’t working — fix that first before proceeding.
The CUDA version shown in nvidia-smi is the maximum version your driver supports, not a separately installed component. Ollama uses CUDA 11.3+ libraries it bundles internally, so as long as your driver supports CUDA 11.3 or higher (driver version 452+), you’re compatible.
Checking Whether Ollama Is Using Your GPU
After installing Ollama and pulling a model, run it with the verbose flag:
ollama run llama3.2 --verboseWatch the output as the model loads. Look for a line containing gpu_layers. If it shows a non-zero number — for example gpu_layers: 33/33 — all model layers are running on your GPU. If it shows gpu_layers: 0, Ollama is falling back to CPU-only mode.
You can also check while a model is running using ollama ps in a separate terminal:
ollama psThis shows the currently loaded model, how much VRAM it’s using, and the processor type (GPU or CPU). A line like llama3.2 4.7 GB 100% GPU confirms full GPU acceleration is active.
For a real-time view, open Task Manager (Ctrl+Shift+Esc) → Performance → GPU. When you send a prompt to Ollama, you should see the GPU compute utilisation spike. If it stays flat at 0% during inference, CUDA isn’t being used regardless of what Ollama reports.
What to Do When GPU Isn’t Being Detected
This is the most common problem people hit. Ollama installs correctly but runs on CPU. Here’s how to fix it systematically:
Step 1: Reinstall the NVIDIA driver. Don’t use GeForce Experience — go to nvidia.com/drivers, select your GPU model and Windows version, download the full driver package, and run it. Choose “Custom” installation and check “Clean install” to remove any stale driver components. Restart your machine after installation.
Step 2: Quit and restart Ollama. Right-click the Ollama icon in the system tray → Quit. Then launch Ollama again from the Start menu. The service needs to restart after a driver change to detect the GPU correctly.
Step 3: Re-run your model and check again. Run ollama run llama3.2 --verbose and look for gpu_layers again. If it’s still 0, continue to step 4.
Step 4: Enable debug logging. Set the OLLAMA_DEBUG environment variable to get verbose output from the Ollama service:
# In PowerShell
$env:OLLAMA_DEBUG=1
ollama serveLook through the output for lines mentioning CUDA, GPU detection, or library loading errors. Common messages to look for: CUDA library not found, insufficient driver, or no CUDA devices available. Each of these points to a specific fix.
Figure 1 — CPU vs GPU Inference Speed: Ollama on Windows
Partial GPU Offloading: When the Model Is Bigger Than Your VRAM
If your model is larger than your GPU’s VRAM, Ollama automatically splits it — loading as many layers as possible into VRAM and the rest into system RAM. This is called partial GPU offloading and it still gives you a meaningful speed improvement over CPU-only, even if it’s not as fast as a full GPU load.
For example, running Phi-4 (14B, ~10GB) on an RTX 4060 with 8GB VRAM: the model is too large for full VRAM loading, but Ollama will put as many layers as fit into VRAM and run the rest on CPU RAM. The result might be 15–25 tokens per second instead of 55–75 — slower than full GPU, but much better than the 2–4 tokens per second you’d get on CPU alone.
You can control how many layers go to GPU with the num_gpu parameter:
# Let Ollama decide automatically (recommended)
ollama run phi4
# Force a specific number of GPU layers (tune to your VRAM)
ollama run phi4 --parameter num_gpu 20
# Force CPU-only (for testing or comparison)
ollama run phi4 --parameter num_gpu 0To find the right num_gpu value for your setup, start with Ollama’s automatic choice and increase the number incrementally until you either hit maximum speed or start getting out-of-memory errors from the GPU. The sweet spot is the highest value that loads without error.
Choosing the Right GPU for Local LLMs on Windows
If you’re in the market for a GPU primarily for running local LLMs, VRAM is the most important spec — more important than raw compute power. A card with more VRAM but slower compute will often outperform a faster card with less VRAM, because it can hold more of the model in fast GPU memory.
Current recommendations by budget:
Under $400: RTX 4060 Ti 16GB. The 16GB version specifically — the 8GB variant is significantly less useful for LLMs despite the similar name. 16GB VRAM handles 7B and 13B models fully in GPU memory.
$400–700: RTX 4070 Ti Super (16GB). Fast compute plus 16GB VRAM. Handles 13B models at 50+ tokens per second.
$700–1000: RTX 4080 Super (16GB). Strong across the board. The 16GB keeps you in the same VRAM tier but with meaningfully higher bandwidth.
$1000+: RTX 4090 (24GB). The consumer king — 24GB VRAM opens up 30B models and makes 7B/13B inference extremely fast. Also handles partial offloading of 70B models usably.
For professional use or if you want to run 70B models fully on GPU, you’re looking at enterprise cards (RTX 6000 Ada with 48GB, A6000) which are a different price tier entirely.
Optimising Performance Once GPU Is Working
With CUDA confirmed working, a few tweaks can improve performance further.
Keep the model loaded between requests. By default, Ollama unloads the model 5 minutes after the last request. Set OLLAMA_KEEP_ALIVE to keep it in VRAM permanently if you’re making frequent requests:
# In Windows Environment Variables
OLLAMA_KEEP_ALIVE = -1Use Q4_K_M quantization. This is Ollama’s default for most models and the best quality/size trade-off. Avoid Q2 variants unless you genuinely need the smaller size — the quality drop is noticeable. Q8 gives slightly better quality but needs significantly more VRAM.
Close other GPU-heavy applications. Games, video editing software, and other ML tools compete for VRAM. If you’re running out of VRAM for a model that should fit, check Task Manager → GPU for what else is using GPU memory and close it.
Enable hardware-accelerated GPU scheduling. In Windows Settings → Display → Graphics → Change default graphics settings, enable “Hardware-accelerated GPU scheduling” if it’s available. This reduces latency between the CPU and GPU and can improve inference consistency.
Figure 2 — VRAM Requirements by Model Class
Common Error Messages and Fixes
“CUDA library not found” in debug output. Ollama couldn’t find the CUDA runtime it bundles. Try reinstalling Ollama — download the latest installer from ollama.com and run it. The installer should place the CUDA libraries alongside the Ollama binary. If reinstalling doesn’t help, check that your antivirus isn’t quarantining the CUDA DLL files that Ollama installs.
“Insufficient driver version”. Your NVIDIA driver is too old. Update from nvidia.com — download the full Game Ready or Studio driver for your GPU, run it with Clean Install checked, and restart.
Model loads but GPU utilisation stays at 0% in Task Manager. This usually means the CUDA libraries loaded but something is preventing actual GPU compute. Check that you’re not in a Remote Desktop session (CUDA sometimes doesn’t initialise correctly over RDP). Try running Ollama directly at the physical machine or over a different remote protocol. Also check that your GPU isn’t being shared with a VM that has exclusive access to it.
Out-of-memory errors mid-inference. The model fit during loading but ran out of VRAM during a long response generation. This happens when context length is large and the KV cache grows during inference. Reduce num_ctx to free up VRAM: ollama run modelname --parameter num_ctx 2048. Alternatively, use a more aggressive quantization like Q2 or Q3 to shrink the model’s base footprint.
Getting GPU acceleration working on Windows is almost always just a driver issue — once you have a current NVIDIA driver installed and Ollama restarted, it picks up the GPU automatically. The 10x+ speed improvement over CPU inference makes the five minutes of setup time one of the best returns in the local LLM space.