Windows is a perfectly capable platform for local LLM inference, with access to the full NVIDIA CUDA ecosystem, a choice between native Windows and WSL2 Linux environments, and dedicated GUI tools that make getting started straightforward. This guide covers the complete Windows local AI setup: CUDA drivers, Ollama on Windows, WSL2 as an alternative, and which approach works best for different use cases.
Windows Native vs WSL2: Which to Use
The first decision for Windows users is whether to run local AI tools in native Windows or in WSL2 (Windows Subsystem for Linux). Both work well, and the choice depends more on your existing workflow than on performance differences.
Native Windows (using the Windows versions of Ollama, LM Studio, etc.) is the right choice if you primarily use Windows applications, are not comfortable with the Linux command line, want the simplest setup, or primarily use GUI tools like LM Studio or Open WebUI. Ollama has a polished Windows installer that handles everything. Performance is essentially identical to WSL2 for inference tasks.
WSL2 (running Linux inside Windows) is the right choice if you are a developer who already works in a Linux-like environment, need to run Linux-only tools or scripts alongside your AI tools, want access to the full Python/conda/pip ecosystem in a Linux environment, or are following tutorials written for Linux. WSL2 with GPU passthrough performs identically to native Linux for CUDA inference — the GPU is accessed directly from WSL2 at near-native speed.
Setting Up CUDA on Windows
NVIDIA CUDA is required for GPU-accelerated inference on Windows. The setup:
# Step 1: Install NVIDIA GPU drivers (from nvidia.com/drivers)
# Get the latest Game Ready or Studio driver for your GPU
# After install, verify:
nvidia-smi # in Command Prompt or PowerShell
# Should show driver version and GPU info
# Step 2: Install CUDA Toolkit (optional for Ollama — it bundles its own)
# Required if you want to build llama.cpp or use PyTorch directly
# Download from: developer.nvidia.com/cuda-downloads
# Recommended: CUDA 12.x
# Verify CUDA installation:
nvcc --version
# Should print CUDA compiler versionOllama on Windows does not require a separate CUDA toolkit installation — it bundles the necessary CUDA runtime. Install just the GPU drivers and Ollama handles the rest. If you plan to build llama.cpp from source or use PyTorch directly, you do need the full CUDA toolkit.
Installing Ollama on Windows
# Download the Windows installer from ollama.com
# Run OllamaSetup.exe — installs to %LOCALAPPDATA%ProgramsOllama
# Ollama runs as a system tray application
# Open PowerShell or Command Prompt:
ollama pull llama3.1
ollama run llama3.1
# Check GPU is being used:
ollama run llama3.1 --verbose
# Look for "gpu_layers" in the output — should match total model layersOllama on Windows installs as a system tray app that starts automatically at login. The ollama command is available in PowerShell and Command Prompt immediately after installation. GPU detection is automatic — if your NVIDIA drivers are installed and Ollama finds a CUDA-capable GPU, it uses it without additional configuration.
WSL2 Setup for Local AI
# Install WSL2 (run in PowerShell as Administrator)
wsl --install
# Restart Windows after install
# Install Ubuntu (default, recommended)
wsl --install -d Ubuntu-24.04
# Launch WSL2
wsl
# Inside WSL2 — GPU is accessible immediately (no extra setup needed)
nvidia-smi # should show your GPU from inside WSL2
# Install Ollama inside WSL2:
curl -fsSL https://ollama.com/install.sh | sh
ollama serve & # start in background
ollama pull llama3.1
ollama run llama3.1WSL2 GPU passthrough works because NVIDIA drivers installed on Windows are automatically exposed to WSL2 via a translation layer. You do not install separate NVIDIA drivers inside WSL2 — the Windows drivers handle everything. This means your CUDA-capable GPU is available at full speed from inside the WSL2 Linux environment without any additional GPU driver configuration.
Figure 1 — Windows Local AI Setup Options
LM Studio: The Best Windows GUI for Local AI
LM Studio is a polished Windows desktop application for local LLMs that provides a clean GUI over llama.cpp inference. It handles model downloading from Hugging Face, model management, and provides a ChatGPT-like interface for conversations. It also runs a local server with an OpenAI-compatible API when you need programmatic access.
The workflow: download LM Studio from lmstudio.ai, install it, use its built-in model browser to search for and download models (it shows file sizes and hardware requirements), then chat directly in the app. For Python integration, enable the local server in LM Studio and point your OpenAI client at localhost:1234. LM Studio handles GPU detection automatically — it picks up NVIDIA GPUs via CUDA and shows a GPU layers slider that controls how many model layers run on GPU versus CPU.
LM Studio is the recommended starting point for Windows users who are not developers and want a clean, integrated experience without any command-line setup. Its main limitation compared to Ollama is that the API server requires the GUI application to be running, whereas Ollama runs as a background service independently.
Open WebUI on Windows
Open WebUI provides a full-featured web interface for Ollama that is significantly more capable than the CLI. On Windows, the simplest installation is via Docker Desktop:
# Install Docker Desktop from docker.com, then:
# Run Open WebUI connected to local Ollama
docker run -d -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
--name open-webui \
--restart always \
ghcr.io/open-webui/open-webui:main
# Access at http://localhost:3000Alternatively, install Open WebUI via pip in a Python environment on Windows. The Docker approach is recommended because it handles all dependencies automatically. Once running, Open WebUI provides conversation history, model switching, system prompt management, document upload and RAG, and multi-user support — all backed by your local Ollama models.
Environment Variables for Windows Ollama
Ollama on Windows reads environment variables from the Windows System Environment Variables (not PowerShell session variables, which do not persist to the Ollama service). Set them via System Properties → Advanced → Environment Variables:
OLLAMA_MODELS — change where models are stored (useful if your C drive is small; point to a larger drive). OLLAMA_HOST — set to 0.0.0.0 to allow access from other devices on your network. OLLAMA_FLASH_ATTENTION — set to 1 to enable Flash Attention for all models. OLLAMA_NUM_PARALLEL — number of simultaneous requests (default 1). OLLAMA_KEEP_ALIVE — time to keep models loaded (set to -1 to keep forever, 0 to unload immediately after generation). After changing environment variables, restart the Ollama service from the system tray icon for them to take effect.
Troubleshooting Windows-Specific Issues
A few issues are specific to Windows. GPU not detected: ensure your NVIDIA drivers are current (download from nvidia.com/drivers, not Windows Update which may have older versions). Run nvidia-smi in PowerShell to confirm the GPU is visible. Restart Ollama after driver updates. Antivirus interference: Windows Defender or third-party antivirus may flag llama.cpp binaries or slow down model file access. Add the Ollama models directory and application directory to your antivirus exclusions. Disk space: Ollama stores models in %USERPROFILE%\.ollama\models by default. Set OLLAMA_MODELS to a larger drive if your system drive is tight. WSL2 GPU issues: if nvidia-smi does not work inside WSL2, ensure you are using WSL2 (not WSL1) and that your NVIDIA drivers on the Windows side are 510.06 or newer. Run wsl –update to get the latest WSL2 kernel which includes improved GPU support. Performance lower than expected: check that GPU layers are actually being used by looking at the verbose output during generation — GPU usage should be visible in Task Manager under GPU (CUDA) when generating.
Python Development on Windows for Local AI
For Python development on Windows with local LLMs, two approaches work well: native Windows Python with the Ollama library, or Python inside WSL2. Native Windows Python is simpler to set up and works fine for Ollama-based development. WSL2 Python is better for projects that use Linux-specific tooling, CUDA PyTorch for fine-tuning, or complex environments with many dependencies. Install Python via the Windows Store (simplest) or python.org, then pip install ollama for the Ollama Python library. The full stack — Python, the Ollama library, and Ollama running as a Windows service — gives you a complete local AI development environment without leaving Windows.
Recommended Hardware for Windows Local AI
On Windows, the NVIDIA GPU is the primary hardware decision. The practical tiers in 2026: RTX 3060 (12GB VRAM, ~$200-250 used) runs 7B models at full GPU speed at around 40-55 t/s — capable and affordable. RTX 3090 or 4070 Ti (24GB VRAM, ~$400-600 used / ~$600 new) runs 14B models at full GPU speed and handles 7B at 60-80+ t/s — the sweet spot for serious local AI work on Windows. RTX 4090 (24GB VRAM, ~$1,600-1,800) maximises 7B-14B generation speed at 90-120 t/s — the fastest consumer GPU for models that fit in 24GB. Beyond 24GB VRAM on a single consumer GPU, models begin to require CPU offloading on Windows/NVIDIA setups, which dramatically reduces speed. If your workload requires 30B+ models at interactive speed on Windows, the costs escalate quickly — dual 3090 or 4090 in NVLink, or moving to server GPUs. For those use cases, an Apple Silicon Mac with large unified memory is often the more practical and cost-effective solution. For 7B-14B model work, which covers the majority of everyday local AI use cases, a Windows machine with a recent NVIDIA GPU provides excellent performance at a lower cost than an equivalent Mac configuration.
Getting Started: The 30-Minute Windows Setup
The fastest path from zero to running local AI on Windows: update your NVIDIA drivers from nvidia.com/drivers, download and run the Ollama Windows installer from ollama.com, open PowerShell and run ollama run llama3.1, and confirm you see GPU usage in Task Manager under GPU → CUDA during generation. If you want a better chat interface, install Open WebUI via Docker Desktop (one Docker command from the section above) and access it at localhost:3000. The entire setup takes under 30 minutes on a Windows machine with an NVIDIA GPU and a reasonable internet connection. For the majority of local AI use cases — interactive chat, coding assistance, writing help, document summarisation — this basic setup is everything you need. Additional tooling (LM Studio for GUI model management, WSL2 for Linux development workflows, ExLlamaV2 for maximum speed) can be added later as specific needs emerge.