Privacy is one of the strongest reasons to run AI locally. When you use ChatGPT, Claude, Gemini, or any cloud AI service, your messages travel to and are processed on servers controlled by third parties — under their data policies, their security posture, and their business decisions about how to handle your information. A local AI setup eliminates that entirely. Your questions, documents, and conversations stay on your hardware. This guide covers how to build a genuinely private, completely offline AI assistant — what tools to use, what to watch out for, and what privacy you actually get.
What “Private” Actually Means for Local AI
Privacy in local AI has two distinct dimensions: data privacy and inference privacy. Data privacy means your content — your prompts, your documents, your conversation history — does not leave your machine and is not stored on external servers. Inference privacy means the AI computation itself happens locally, on your hardware, not on a cloud provider’s GPU farm. A fully local setup provides both. Understanding this distinction matters because some “local AI” setups are only partially local — for example, using a local frontend that routes requests to a cloud API, or a RAG system that stores documents locally but sends them to an external embedding service.
True local AI means every component runs on your hardware: the inference engine, the model weights, the embedding model if you use RAG, the frontend, and the data storage. Nothing in the request-response path touches the internet. You can verify this by running in airplane mode — a properly configured local AI setup works identically with no network connection.
The Full Local Stack
A complete, private local AI setup has these components:
Inference server: Ollama — runs on Windows, macOS, Linux. Handles model management and inference. All computation is local. No telemetry by default.
Chat frontend: Open WebUI (self-hosted via Docker) or the Ollama CLI. Open WebUI stores conversation history locally in its Docker volume. No external services called.
Embedding model (for RAG): nomic-embed-text or mxbai-embed-large via Ollama. Both run locally — your documents never leave the machine for embedding.
Vector store (for RAG): ChromaDB (self-hosted), Qdrant (self-hosted), or the built-in SQLite-backed store in Open WebUI. All local.
Model weights: Downloaded once from Hugging Face or ollama.com. Stored locally. After the initial download, no internet access is required.
Once this stack is set up, you can disconnect from the internet entirely and everything continues to work. This is the gold standard for local AI privacy.
Setting Up the Private Stack
# Step 1: Install Ollama
curl -fsSL https://ollama.com/install.sh | sh # Linux/macOS
# Windows: download installer from ollama.com
# Step 2: Download your models (requires internet, one-time)
ollama pull llama3.1 # main chat model
ollama pull nomic-embed-text # embedding model for RAG
# Step 3: Install Open WebUI (requires Docker)
docker run -d -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
# Step 4: Access at http://localhost:3000
# Create a local account (stored in the Docker volume, not external)
# Select your model, start chattingAfter the initial model downloads, set Ollama’s keep-alive to indefinite and turn off your internet connection. Everything still works. Your AI assistant is now completely air-gapped from external services.
Figure 1 — What Leaves Your Machine in Each AI Setup
Privacy Pitfalls: What Can Leak Without You Realising
A local AI setup is only as private as its weakest component. Several common configurations that seem local actually have data leaving the machine:
Using a cloud embedding service for RAG. If you set up RAG in Open WebUI and configure it to use OpenAI’s embedding API rather than a local Ollama embedding model, your document chunks are sent to OpenAI for embedding. The chat inference happens locally but your documents do not stay local. Check your embedding model configuration and confirm it points at a local Ollama model, not an external API.
Ollama telemetry. Ollama does not send your prompts or model outputs anywhere. However, some frontends and applications built on top of Ollama may have their own analytics or error reporting that sends usage data externally. Review the privacy documentation of any frontend you install, particularly Open WebUI, which has configurable analytics settings.
Model download sources. Models are downloaded from Hugging Face or ollama.com — these are external services that receive your IP address and know which models you downloaded. Once downloaded, models run locally. If even the model download is sensitive, download via Tor or a VPN, or have a trusted party prepare a local file server with the models pre-downloaded.
Browser-based frontends. Open WebUI runs in your browser but the server is local — data does not leave your machine. However, the browser itself may have extensions or settings that intercept local traffic. Use a browser with extensions disabled for AI work if this concerns you, or use the Ollama CLI directly to bypass the browser entirely.
Choosing the Right Model for Private Use
For private AI use, any model you pull from Ollama is appropriate — the model weights are just files on your machine after download, and the inference happens locally regardless of which model you choose. The considerations for private use are the same as general use: model capability versus hardware requirements. A few recommendations based on common private AI use cases:
For general private assistant (writing, research, analysis): Llama 3.1 8B is excellent — highly capable, fast on modern hardware, good instruction following. Llama 3.1 70B if you have the VRAM for significantly better quality on nuanced tasks. For private document analysis: any capable 7B+ model with a large context window configured (set num_ctx to 16384 or higher). Qwen 2.5 32B handles long documents well on hardware with sufficient memory. For private coding assistance: Qwen 2.5 Coder 7B or 14B — specifically trained for code, faster and better than general models for coding tasks. For private medical, legal, or financial questions: use a larger model (14B+) where quality on domain-specific questions matters more, and treat the outputs as a starting point for research rather than authoritative advice regardless of the model.
Local AI for Specific Sensitive Use Cases
Medical information. Looking up symptoms, understanding a diagnosis, researching treatment options — these are deeply personal queries that many people are uncomfortable putting into a cloud service. A local AI handles these queries entirely privately. Important caveat: local LLMs can produce confident-sounding but incorrect medical information. Use local AI as a starting point for understanding terminology and generating questions to ask your doctor, not as a substitute for medical advice. The privacy benefit is real; the reliability caveat applies regardless of whether the AI is local or cloud.
Legal and financial research. Researching a legal situation, understanding contract clauses, exploring financial options — similar considerations apply. Local AI can explain concepts, summarise documents, and help you formulate questions for professionals. The privacy of the research remains yours. Again, treat outputs as research assistance rather than authoritative advice.
Confidential work documents. Client proposals, strategic planning documents, internal analysis — pasting these into ChatGPT raises legitimate concerns about corporate data policies. A local AI has no such concerns. Use local AI freely for all the writing assistance, summarisation, and analysis tasks you would hesitate to do with cloud AI on confidential work materials.
Personal journals and creative work. Some writing is deeply personal — journals, memoirs, creative projects you are not ready to share. Local AI assistance for this work remains yours alone. The model never sees your writing in a way that leaves your control.
Offline AI Without a GPU: CPU-Only Setup
Not everyone has a dedicated GPU. CPU-only local AI is slower but viable for private use where response time is less critical than privacy. On a modern 8-16 core CPU, a 3B model generates at 2-6 tokens per second — slow for interactive conversation but perfectly adequate for background processing, document analysis, or tasks where you submit a query and come back in a minute for the result. The setup is identical to the GPU setup — Ollama detects whether a GPU is available and falls back to CPU automatically. Simply follow the same installation steps; Ollama handles the hardware detection. For a privacy-focused batch workflow (classify 200 documents overnight), a CPU-only setup running an 8B model is entirely practical even if it would be frustrating for real-time chat.
The Privacy Guarantee in Plain Terms
With a properly configured local AI stack — Ollama as the inference server, Open WebUI or GPT4All as the frontend, local embedding models for RAG — you have a complete AI assistant where: your prompts are never transmitted to any external server; your documents are never sent to any third-party service; your conversation history is stored in your own database on your own machine; the AI computation happens on your own processor; and the setup works with no internet connection after the initial model downloads. This is a genuine, verifiable privacy guarantee — not a policy you have to trust, but a technical architecture you can inspect. In a world where most AI services have complex data policies with important asterisks, local AI’s privacy model is simple: if you have not connected to the internet, your data has not left your machine.