How to Install Ollama on Ubuntu 24.04 (Complete Guide)

Ubuntu 24.04 is one of the best environments for running Ollama — the installation is clean, GPU support via CUDA works reliably, and you can run it headlessly on a server or as a persistent service without any drama. Whether you’re setting it up on a local workstation, a home server, or a cloud VM, this guide covers everything from the initial install to getting GPU acceleration working and running Ollama as a system service.

This guide assumes you’re on Ubuntu 24.04 LTS (Noble Numbat). Most steps also work on Ubuntu 22.04 and Debian-based distros, but 24.04 is the smoothest experience right now.

Installing Ollama

Ollama provides a one-line install script that handles everything — downloading the binary, setting up the systemd service, and adding the ollama command to your PATH. Run this in your terminal:

curl -fsSL https://ollama.com/install.sh | sh

The script detects your system architecture (x86_64 or ARM), downloads the appropriate binary, installs it to /usr/local/bin/ollama, and creates a systemd service that starts automatically on boot. You’ll see output as it runs — if it detects an NVIDIA GPU, it’ll also install the CUDA libraries it needs.

Once it finishes, verify the installation:

ollama --version

And check that the service is running:

systemctl status ollama

You should see active (running). If it’s not running, start it manually:

sudo systemctl start ollama
sudo systemctl enable ollama  # ensures it starts on boot

Running Your First Model

With Ollama running, pull and run a model in one command:

ollama run llama3.2

This downloads the Llama 3.2 3B model (~2GB) and drops you into an interactive chat session. Type your prompt at the >>> prompt, hit Enter, and the response streams back. Type /bye to exit.

If you want to pull a model without running it immediately:

ollama pull mistral
ollama pull phi4
ollama pull qwen2.5:7b

List everything you’ve downloaded:

ollama list

The full model library is at ollama.com/library. Each model page shows available sizes and disk requirements — useful for planning around your storage.

Enabling NVIDIA GPU Acceleration

If you have an NVIDIA GPU, Ollama on Ubuntu is where GPU acceleration works best. The install script tries to set up CUDA automatically, but let’s verify it’s working.

First, confirm your driver and CUDA are in order:

nvidia-smi

If this shows your GPU model, driver version, and CUDA version, you’re in good shape. Ollama requires CUDA 11.3 or later. If nvidia-smi isn’t found, you need to install the NVIDIA driver first:

sudo apt update
sudo apt install -y nvidia-driver-535
sudo reboot

After rebooting, run nvidia-smi again to confirm. Then restart Ollama:

sudo systemctl restart ollama

Run a model with the --verbose flag to confirm GPU layers are being used:

ollama run llama3.2 --verbose

Look for gpu_layers in the output — a non-zero value means your GPU is active. With a capable GPU, inference speed on a 7B model jumps from a few tokens per second (CPU) to 40–80+ tokens per second, which makes the difference between painful and actually useful.

Configuring Ollama with Environment Variables

Ollama’s behaviour is controlled through environment variables. On Ubuntu with systemd, the right way to set these is via a systemd override — not in .bashrc, because the service runs as its own user and won’t pick up shell environment changes.

Create a systemd override file:

sudo systemctl edit ollama

This opens an editor. Add your variables inside the [Service] block:

[Service]
Environment="OLLAMA_MODELS=/data/ollama/models"
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=10m"

Save and exit, then reload and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Here’s what each variable does and when you’d use it:

OLLAMA_MODELS — changes where models are stored. Default is /usr/share/ollama/.ollama/models. If you have a separate data drive with more space, point this there.

OLLAMA_HOST — controls the listen address. Default is 127.0.0.1:11434 (localhost only). Set to 0.0.0.0:11434 to accept connections from other devices on your network — useful for connecting Open WebUI or other clients running on different machines.

OLLAMA_KEEP_ALIVE — how long Ollama keeps a model in memory after the last request. Default is 5 minutes. Set to -1 to keep it loaded indefinitely, or 0 to unload immediately after each request.

OLLAMA_NUM_PARALLEL — number of parallel requests Ollama handles simultaneously. Default is 1. Increase this if you’re serving multiple users or running parallel API calls.

Opening Ollama to Your Local Network

Setting OLLAMA_HOST=0.0.0.0:11434 makes Ollama listen on all interfaces. But you should also make sure your firewall allows it. Ubuntu uses ufw by default:

sudo ufw allow 11434/tcp
sudo ufw status

If ufw isn’t enabled, you can skip this. If it is enabled and you don’t open the port, connections from other machines will time out silently — ufw drops packets without returning an error, which makes it a surprisingly common gotcha.

Test from another machine on your network once this is set up:

curl http://[your-ubuntu-ip]:11434

You should get back Ollama is running. If not, double-check the service is running (systemctl status ollama) and that ufw has the port open.

Using the Ollama API from Python

Ollama exposes a REST API at http://localhost:11434. There’s also an official Python library that wraps it cleanly:

pip install ollama

import ollama

# Simple generation
response = ollama.generate(model='llama3.2', prompt='Explain what a transformer is in ML.')
print(response['response'])

# Chat with message history
messages = [
    {'role': 'user', 'content': 'What is gradient descent?'}
]
response = ollama.chat(model='llama3.2', messages=messages)
print(response['message']['content'])

The library handles streaming too — use stream=True and iterate over the response to print tokens as they arrive, rather than waiting for the full output. This is how tools like Open WebUI and Continue get that real-time typing effect.

Ollama also exposes an OpenAI-compatible endpoint at /v1/chat/completions, so you can point any OpenAI SDK client at http://localhost:11434/v1 and it’ll work without code changes — just set any string as the API key.

Running Ollama Without sudo and Managing Permissions

The install script creates an ollama user and group that the service runs under. If you want to interact with the Ollama socket or models directory without sudo, add your user to the ollama group:

sudo usermod -aG ollama $USER
newgrp ollama

Or log out and back in for the group change to take full effect. This matters more for advanced setups where you’re managing model files directly, not for typical day-to-day use.

Useful Commands and Maintenance

# Check service logs (useful for debugging)
journalctl -u ollama -f

# List downloaded models
ollama list

# Remove a model you no longer need
ollama rm llama3.2

# Show model details including context length and parameters
ollama show mistral

# Check what's currently loaded in memory
ollama ps

# Update Ollama itself (re-run the install script)
curl -fsSL https://ollama.com/install.sh | sh

Updating Ollama is the same command as installing it — the script detects an existing install and upgrades in place, preserving your models and configuration. Worth running occasionally as new versions bring performance improvements and expanded model support.

The journalctl -u ollama -f command is your best friend for debugging. If a model isn’t loading, GPU detection is failing, or API calls are returning unexpected errors, the service logs almost always tell you exactly what’s wrong.

What to Set Up Next

With Ollama running on Ubuntu, the most common next step is adding Open WebUI — a Docker-based web interface that gives you a full chat UI connected to your local Ollama instance. On Ubuntu, Docker installation is straightforward and Open WebUI is a single docker run command.

If you’re planning to use Ollama for API calls from applications or scripts, the Python library above is the easiest entry point. From there, connecting it to LangChain or LlamaIndex for RAG pipelines is well-documented and works reliably with the local Ollama backend.

For server setups where you want Ollama accessible to multiple users or services, consider pairing it with Nginx as a reverse proxy — you get cleaner URL routing, optional HTTPS via Let’s Encrypt, and better control over who can reach the API. That setup is worth a dedicated guide, but the short version is: proxy pass port 80/443 to localhost:11434 and add basic auth if you need it.

Troubleshooting Common Issues

GPU not detected after install. The install script installs CUDA libraries, but if your NVIDIA driver wasn’t already present, it may not have everything it needs. Run nvidia-smi — if it fails, install the driver first (sudo apt install nvidia-driver-535), reboot, then reinstall Ollama. The reinstall is safe and won’t delete your models.

Service fails to start. Check the logs with journalctl -u ollama -n 50. Port conflicts are the most common cause — something else may be on 11434. Use ss -tlnp | grep 11434 to check. Change Ollama’s port via OLLAMA_HOST in the systemd override if needed.

Permission denied on model files. If you moved models manually to a new directory, the ollama service user needs read access. Run sudo chown -R ollama:ollama /your/models/path and restart the service.

Out of memory when running large models. This usually means the model plus system overhead exceeds your RAM. Try a smaller quantized version (ollama pull llama3.1:8b-q4_0) or reduce other processes consuming memory. If you’re on a server, check that swap is configured — Ollama can use swap space as a last resort, though performance will suffer.

Slow inference despite having a GPU. Confirm GPU layers are active with ollama run modelname --verbose. If gpu_layers is 0 or low, your CUDA setup may be incomplete. Also check nvidia-smi during inference to see if GPU utilisation is actually rising — if it stays at 0%, CUDA isn’t being used regardless of what Ollama reports.

Ubuntu’s combination of solid NVIDIA driver support, systemd for service management, and a well-documented ecosystem makes it the most reliable platform for running Ollama in production. Whether you’re running it on a gaming rig, a home server, or a cloud VM, the setup above should get you to a stable, GPU-accelerated Ollama instance that starts automatically, logs cleanly, and is easy to maintain over time.