How to Use AnythingLLM with Ollama: Complete Setup Guide

AnythingLLM is one of the most polished all-in-one local AI applications available. Unlike Flowise or Dify, which require configuration and technical knowledge to set up properly, AnythingLLM is designed to work well out of the box: install it, connect it to Ollama, upload some documents, and you have a working private AI assistant with document Q&A, multi-user support, and a clean chat interface — no node-based pipeline building required. This guide covers installation, connecting to Ollama, building a knowledge base, and the features that make AnythingLLM worth using.

What AnythingLLM Offers

AnythingLLM packages several capabilities into a single application: a chat interface backed by any LLM you configure, a document management system that handles chunking and embedding automatically, workspace-based organisation (separate knowledge bases for different projects), an agent mode that can use tools like web search and code execution, multi-user support with role-based access, and an API for integration with other tools. The design philosophy is “batteries included” — you shouldn’t need to understand vector databases, embedding models, or retrieval strategies to get a working RAG system. AnythingLLM handles all of that behind a clean UI.

Installation

AnythingLLM has desktop app installers for Windows, macOS, and Linux, which is the easiest path for personal use. Download from anythingllm.com/desktop — pick the right installer for your OS and run it. The app bundles everything it needs; no separate database or node runtime required.

For server deployment or Docker:

docker pull mintplexlabs/anythingllm
docker run -d -p 3001:3001   -v anythingllm_storage:/app/server/storage   mintplexlabs/anythingllm

Access at http://localhost:3001. The first run walks you through a setup wizard that’s genuinely straightforward — much simpler than configuring Flowise or Dify from scratch.

Connecting AnythingLLM to Ollama

In the AnythingLLM settings (gear icon), navigate to AI Providers. Select Ollama as the LLM provider. Enter your Ollama base URL (http://localhost:11434) and click the model picker — AnythingLLM fetches your available models directly from Ollama and shows them in a dropdown. Select your preferred model and save.

Do the same for the Embedding Provider section: select Ollama, set the URL, and choose an embedding model. nomic-embed-text or mxbai-embed-large are both excellent options. That’s the full configuration for a basic setup — no YAML, no environment variables, no node graphs.

Creating Workspaces and Adding Documents

AnythingLLM organises content into Workspaces — each is a separate knowledge base with its own document set, LLM settings, and chat history. Create a workspace for each distinct project or topic area: one for your team’s internal documentation, one for a specific research project, one for a codebase you’re exploring. Keeping workspaces focused produces better retrieval than dumping everything into a single large workspace.

To add documents, click the upload icon in a workspace. AnythingLLM accepts PDFs, Word documents, text files, Markdown, CSV, and YouTube video transcripts by URL. After uploading, it automatically chunks the documents and generates embeddings using your configured Ollama embedding model. The process takes seconds to a few minutes depending on document size. Once embedded, documents are immediately available for RAG-based queries.

The document management UI shows each uploaded file, when it was added, and its chunk count. You can remove documents individually, re-embed after updating content, and move documents between workspaces as your organisation evolves.

Chat and Q&A Modes

Each AnythingLLM workspace has two chat modes switchable from the UI:

Query mode: The model only answers from the documents in the workspace. If the answer isn’t in your uploaded documents, it says so. This is the right mode for a knowledge base assistant where you specifically want answers grounded in your content — no hallucination from the model’s training data.

Chat mode: The model uses your documents as context but also draws on its own knowledge. More like a general assistant that’s aware of your documents. Better for exploratory conversations where you want to go beyond the document content.

Switching between modes is a one-click toggle per workspace. Using query mode as the default and switching to chat mode when needed is the recommended pattern — it gives you reliable document-grounded answers by default without permanently losing the model’s broader knowledge.

Agent Mode

AnythingLLM’s agent mode gives the model access to tools: web search (via DuckDuckGo by default, configurable), a code interpreter, URL fetching, file system access (configurable), and SQL query execution against connected databases. Enable agents in workspace settings and start a chat with @agent to activate agent mode for that message.

Practical uses for agent mode: asking the model to search the web and summarise current information on a topic, running a Python code snippet to verify an answer, or querying a connected database alongside document context. The agent uses a ReAct-style loop — it decides which tool to use, executes it, incorporates the result, and continues until it can answer the original question. With Ollama as the backend, all the reasoning happens locally; only the tool calls themselves (web search, URL fetches) hit external services.

Agent quality depends heavily on your Ollama model choice. Models with strong instruction following and tool use training — Llama 3.1 8B, Qwen 2.5 7B, Mistral Small 3 — work better in agent mode than models that weren’t specifically trained for tool use. The agent will still run with any model but may struggle to use tools correctly on complex multi-step tasks with weaker models.

Multi-User Setup

AnythingLLM supports multiple user accounts with different permission levels — admin, manager, and default user. This makes it practical for small team deployments where you want everyone to have access to the AI assistant and shared knowledge bases without each person needing to run their own Ollama setup. To enable multi-user mode, go to Settings → Multi-User Mode and set it up. Users sign in with credentials you create in the admin panel.

The economics of a shared deployment are compelling: one machine with Ollama and AnythingLLM can serve a team of 5–10 people for casual use (not high-concurrency). The Ollama inference is shared — when two people chat simultaneously, requests queue — but for a team doing occasional document Q&A rather than real-time high-volume use, this is fine in practice. The shared knowledge bases mean everyone benefits from documents uploaded by any team member, creating a genuinely collaborative private AI knowledge tool.

The API and Embedding Options

AnythingLLM exposes a REST API that lets you query workspaces programmatically — useful for integrating it into other applications or scripts. The API supports chatting with a workspace, uploading documents, listing workspaces, and managing users. Authentication uses API keys generated in the admin settings.

# Query a workspace via the API
curl -X POST http://localhost:3001/api/v1/workspace/my-workspace/chat   -H "Authorization: Bearer YOUR_API_KEY"   -H "Content-Type: application/json"   -d '{"message": "What does the documentation say about authentication?", "mode": "query"}'

This API surface means AnythingLLM isn’t just a standalone chat app — it can serve as the RAG backend for other tools, scripts, or even a Slack bot or Discord bot that queries your knowledge base. The combination of an easy-to-use UI for document management and an API for programmatic access makes AnythingLLM one of the most practically useful local AI applications available today.

When to Choose AnythingLLM

AnythingLLM is the right choice when ease of setup and use matters more than maximum customisability, when you want multi-user support without significant configuration overhead, or when you specifically want a document knowledge base that non-technical users can manage and query. It’s the easiest path to a working private ChatGPT-style application backed by your own documents and a local Ollama model. The trade-off versus Flowise or Dify is less flexibility — AnythingLLM’s opinionated design makes it very easy to do what it’s designed for, but harder to build custom pipelines that don’t fit its workspace-centric model. For most personal and small team use cases, that trade-off is worth it: the simplicity of getting something excellent working in under an hour beats spending days configuring a more flexible system that ultimately does the same thing.

Performance and Hardware for AnythingLLM

AnythingLLM itself is lightweight — the application and its built-in vector database use a few hundred MB of RAM. The hardware bottleneck is always Ollama inference. A practical setup for personal use: any machine with 8GB RAM running Ollama with a 7B model alongside AnythingLLM. For team deployment, the same applies but with an eye toward inference concurrency — if multiple users will query simultaneously, set OLLAMA_NUM_PARALLEL=2 and make sure the host machine has 16GB+ RAM to support two concurrent 7B model contexts. AnythingLLM’s embedding step (when you upload documents) is also Ollama-backed — uploading a large batch of documents will run many embedding calls sequentially, which takes time but doesn’t require concurrent inference capacity. A 500-page document library typically takes 5–15 minutes to embed with nomic-embed-text on a mid-range machine, after which queries are fast. The incremental embedding (only new or changed documents get re-embedded) means subsequent uploads are quick once your initial library is indexed.

Keeping Your Knowledge Base Fresh

One of AnythingLLM’s practical advantages is how it handles document updates. When you upload a new version of a document, it re-embeds just that document rather than reprocessing the entire workspace. This makes maintaining an up-to-date knowledge base manageable even for fast-changing content like internal documentation, policy documents, or research notes. The workspace structure also helps: keeping a single workspace per topic or project area means a document update only triggers re-embedding within that workspace, not across your entire AnythingLLM installation. For large document libraries that change frequently, it’s worth setting a periodic review schedule — checking monthly that your knowledge base reflects your current content rather than letting stale documents accumulate and degrade retrieval quality over time. AnythingLLM’s document management UI makes this audit straightforward: you can see when each document was last updated and remove outdated versions with a single click.

Troubleshooting Common AnythingLLM + Ollama Issues

The most common issue is the model dropdown showing empty after entering the Ollama URL — this means AnythingLLM can’t reach your Ollama instance. Check that Ollama is running (curl http://localhost:11434), confirm the URL in AnythingLLM settings matches exactly, and if running the Docker version of AnythingLLM, remember to use http://host.docker.internal:11434 rather than localhost. The second common issue is slow chat responses — this is normal if your model is large relative to your hardware, but if it’s slower than expected, check whether the model is loaded in Ollama with ollama ps and whether other processes are competing for GPU memory. The third issue is poor retrieval quality — documents are uploaded but the model gives vague or incorrect answers. Try reducing your workspace document set (too many documents in one workspace degrades retrieval precision), switch from chat mode to query mode (forces the model to use documents rather than its own knowledge), and verify your embedding model is correctly configured — if embeddings were generated with a different model than what’s currently configured, retrieval quality suffers significantly.

Fixing the embedding model mismatch requires deleting and re-uploading your documents to regenerate embeddings with the correct model — a one-time cost that’s worth doing if your current retrieval quality is poor.