Once you move beyond basic Ollama usage, a set of advanced server configuration options unlocks meaningfully better performance and more flexible deployment. This guide covers the four most impactful advanced topics: profiling your GPU during inference to identify bottlenecks, running two models simultaneously for different tasks, understanding how Ollama caches models between requests, and configuring NUMA on multi-socket CPU servers. Each of these is a standalone topic — skip to whichever is relevant to your setup.
Profiling GPU Usage During Ollama Inference
Understanding what your GPU is actually doing during inference helps diagnose performance problems and confirm that hardware changes or configuration tweaks have the expected effect. Three tools cover the main platforms:
NVIDIA (nvidia-smi): The most detailed profiling available. Run alongside a generation:
# Stream GPU metrics every second
nvidia-smi dmon -s ucm -d 1
# Columns: sm% (compute), mem% (memory BW), enc%, dec%, fb (VRAM used MB)
# Snapshot every 0.5s with key fields
watch -n 0.5 "nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.used,memory.free,temperature.gpu --format=csv,noheader"What to look for: sm% (GPU shader/compute utilisation) should be near 100% during active token generation. If sm% is consistently below 70%, you are memory-bandwidth bound rather than compute bound — the GPU is waiting for data from VRAM rather than actively computing. This is normal for small models (3B-7B) on high-end GPUs where memory bandwidth is the bottleneck. mem% (memory controller utilisation) being near 100% while sm% is lower confirms this. The implication: a faster GPU will not help much — a GPU with higher memory bandwidth (like NVIDIA H-series vs A-series) or running a larger model that better utilises compute would help more.
NVIDIA Nsight Systems provides deeper profiling — kernel-level timing, memory transfer analysis — but requires the Nsight SDK. Useful for research but overkill for typical user debugging.
Apple Silicon (powermetrics):
sudo powermetrics --samplers gpu_power,cpu_power -i 1000 | grep -E "GPU|ANE|CPU"Shows GPU power draw and utilisation. During active inference, GPU active residency should be high. Apple’s ANE (Neural Engine) is not used by Ollama’s Metal backend — all inference goes through the GPU, so ANE% will remain low.
Linux AMD (radeontop):
radeontop -d - -l 1 # pipe to stdout, 1 second intervalShows GPU engine load and VRAM usage. Equivalent interpretation to nvidia-smi: high VRAM bus utilisation with moderate shader load indicates memory bandwidth bottleneck.
Running Two Models Simultaneously
By default Ollama loads one model at a time — switching models evicts the previous one from GPU memory. The OLLAMA_MAX_LOADED_MODELS environment variable changes this behaviour:
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_NUM_PARALLEL=2
# Restart Ollama after settingWith these set, Ollama will keep up to two models loaded simultaneously. This enables a practical dual-model workflow: a fast small model (3B) for quick interactive queries and a larger model (7B+) for tasks requiring more capability, both staying warm in memory. You switch between them by changing the model name in your request — no loading delay after the first request to each.
Memory requirements: both models’ weights occupy VRAM simultaneously. A 3B Q4 model (~2GB) plus a 7B Q4 model (~4.5GB) requires about 7GB VRAM plus KV cache overhead. On a 12GB GPU this is comfortable. On an 8GB GPU you may need to use aggressive quantizations to fit both. Check with ollama ps during a dual-model session to see actual memory consumption. If Ollama runs out of VRAM, it silently offloads layers to CPU RAM, degrading performance for the model that does not fit. Monitor ollama ps output to see the PROCESSOR column — it shows the GPU/CPU split for each loaded model.
Figure 1 — Dual-Model Setup: Memory Requirements by Config
Practical Dual-Model Workflows
The most useful dual-model pattern is pairing a fast coding or chat model with a dedicated embedding model. Embeddings need to be generated repeatedly for RAG workflows and do not need to share GPU resources with the chat model. Running them simultaneously means your chat model stays warm while the embedding model handles document indexing in the background:
import ollama
import concurrent.futures
# Both models stay loaded simultaneously
def embed_documents(texts):
results = []
for text in texts:
r = ollama.embeddings(model="nomic-embed-text", prompt=text)
results.append(r["embedding"])
return results
def chat_response(message):
r = ollama.chat(
model="llama3.2",
messages=[{"role": "user", "content": message}]
)
return r["message"]["content"]
# Run both simultaneously without model-switching overhead
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as ex:
chat_future = ex.submit(chat_response, "Explain neural networks")
embed_future = ex.submit(embed_documents, ["doc1 text", "doc2 text"])
print(chat_future.result())
embeddings = embed_future.result()Another useful pattern: a general chat model (llama3.1) plus a coding specialist (qwen2.5-coder). Route coding questions to the specialist and general questions to the chat model from your application, with both staying loaded for instant response.
Understanding Ollama Model Caching
Ollama keeps models loaded in GPU memory for a configurable period after the last request — this is the keep-alive mechanism. Understanding it is important for managing GPU memory and ensuring models are available without load latency.
OLLAMA_KEEP_ALIVE controls how long a model stays loaded after the last request. The default is 5 minutes. Options:
export OLLAMA_KEEP_ALIVE=5m # default: unload 5 minutes after last use
export OLLAMA_KEEP_ALIVE=1h # keep loaded for 1 hour
export OLLAMA_KEEP_ALIVE=-1 # keep loaded indefinitely (until Ollama restarts)
export OLLAMA_KEEP_ALIVE=0 # unload immediately after each requestSetting keep-alive to -1 is useful for your primary daily-use model — it stays in GPU memory permanently, meaning every request gets instant response without the 1-5 second model loading delay. The cost is that VRAM is committed to that model until Ollama restarts. On a machine with sufficient VRAM for your primary model, this is usually the right setting. Setting keep-alive to 0 is useful on memory-constrained machines where you run different models for different tasks and cannot afford to keep multiple loaded — each request loads the model fresh and releases memory immediately after.
You can also control keep-alive per-request via the API:
import ollama
# Keep loaded for 30 minutes after this request
ollama.generate(model="llama3.2", prompt="Hello", keep_alive="30m")
# Unload immediately after this request
ollama.generate(model="llama3.2", prompt="Hello", keep_alive=0)Per-request keep-alive overrides the global OLLAMA_KEEP_ALIVE for that specific request. This is useful in batch processing pipelines where you want to unload the model after completing a batch job rather than waiting for the global timeout.
KV Cache and Context Caching
Beyond model weight caching, Ollama caches the KV (key-value) state of the attention layers during a conversation. This is why subsequent messages in the same conversation are faster than the first: the model does not re-process earlier messages from scratch but instead extends the existing KV cache with new tokens. This KV cache lives in GPU memory and grows with conversation length — each additional token adds a small amount of KV state that must be stored and accessed during future generation steps.
When you start a new conversation (a fresh API call without conversation history), Ollama processes the system prompt from scratch. If you use the same system prompt across many requests — for example, a long system prompt defining your assistant’s behaviour — Ollama cannot automatically cache it across separate API calls in the current architecture. This means every new conversation pays the system prompt prefill cost. For very long system prompts (2000+ tokens), this prefill step can take several seconds. The practical workaround: keep system prompts concise for interactive use, or structure your application to maintain conversation history across what the user perceives as separate sessions, reusing the KV cache. The keep_alive mechanism keeps the model and its current KV state loaded — this means within the keep-alive window, a new conversation that shares the same system prompt will re-use some cached computation, though the exact behaviour depends on the Ollama version.
NUMA Configuration for Multi-Socket Servers
NUMA (Non-Uniform Memory Access) is relevant for systems with multiple CPU sockets — typically server hardware or high-end workstations with dual-socket motherboards. On NUMA systems, each CPU socket has its own local memory that it accesses faster than the remote memory attached to the other socket. When an application allocates memory across NUMA nodes inefficiently, it pays a performance penalty on remote memory accesses. Ollama’s llama.cpp backend has NUMA support that can improve inference performance on multi-socket systems by keeping model data local to the CPU and memory that are doing the work.
# Check if your system has multiple NUMA nodes
numactl --hardware
# If output shows "available: 2 nodes", NUMA config is relevant
# Set NUMA strategy via environment variable
export OLLAMA_NUMA=true # or specific strategy if supported
# Alternatively, run Ollama with numactl to pin to specific node
numactl --cpunodebind=0 --membind=0 ollama serveThe numactl --cpunodebind=0 --membind=0 approach pins Ollama to NUMA node 0 (the first CPU socket and its memory). This is the simplest NUMA optimisation and avoids cross-node memory accesses entirely by constraining Ollama to a single node. The cost is that you cannot use more than half your system’s CPU threads and memory. For models that fit within a single NUMA node’s memory, this often outperforms letting the OS schedule across nodes freely. For models that require more memory than one NUMA node provides, cross-node access is unavoidable — in this case, ensuring the model weights are evenly distributed across nodes is better than concentrating them in one.
NUMA optimisation is only relevant on multi-socket systems. Single-socket machines — including all laptops and most desktop workstations — have a single NUMA node and these settings have no effect. Check with numactl --hardware first; if it shows “available: 1 nodes”, NUMA configuration will not help you.
GPU Layer Control: Partial Offloading
The num_gpu parameter controls how many transformer layers run on GPU versus CPU. Setting it explicitly gives fine-grained control over the GPU/CPU split when a model slightly exceeds your available VRAM:
import ollama
# Full GPU (default)
ollama.generate(model="llama3.1:8b", prompt="Hello", options={"num_gpu": 99})
# CPU only
ollama.generate(model="llama3.1:8b", prompt="Hello", options={"num_gpu": 0})
# Partial: first 25 layers on GPU, rest on CPU
# Good when model slightly exceeds VRAM
ollama.generate(model="llama3.1:8b", prompt="Hello", options={"num_gpu": 25})Llama 3.1 8B has 32 transformer layers. Setting num_gpu=25 puts the first 25 on GPU and the remaining 7 on CPU. This gives most of the speed benefit of GPU inference while fitting within a constrained VRAM budget. The optimal split is the highest num_gpu that does not cause out-of-memory errors — experiment by increasing from 0 upward until you hit VRAM limits, then step back one. Monitor with nvidia-smi fb (framebuffer memory) to see VRAM usage as you increase layers. Partial offloading is most useful when a model exceeds your VRAM by a modest amount (10-30%); if the model is much larger than your VRAM, CPU inference is more predictable than highly partial GPU offloading.