llama.cpp vs Ollama: Differences, Speed and Python Bindings

by

llama.cpp and Ollama are not alternatives to each other — Ollama wraps llama.cpp. But developers often face a choice between using Ollama’s API and using llama.cpp directly through its Python bindings. This guide covers the real differences in speed, flexibility, and developer experience, and when the llama-cpp-python library is the better choice over the Ollama Python library.

The Architectural Relationship

Ollama is a model management server that uses llama.cpp as its inference engine. When you call the Ollama API, Ollama loads your model into llama.cpp, routes your request to the inference engine, and returns the result. The llama.cpp inference code is identical in both cases — the same C++ code runs the same matrix multiplications on the same GPU. Any performance difference between Ollama and direct llama.cpp is overhead from Ollama’s server layer, not from inference itself. In practice this overhead is small (typically under 5% latency on a single request) and negligible for most applications.

llama-cpp-python is a different layer: it is a Python binding to llama.cpp that lets you call llama.cpp inference directly from Python without running an HTTP server. The Python process loads the model and runs inference in-process, rather than making HTTP requests to a running server. This eliminates the server hop entirely and gives you direct access to llama.cpp’s C API from Python.

Installing llama-cpp-python

# macOS with Metal GPU (Apple Silicon or AMD)
CMAKE_ARGS="-DGGML_METAL=ON" pip install llama-cpp-python

# Linux with NVIDIA CUDA
CMAKE_ARGS="-DGGML_CUDA=ON" pip install llama-cpp-python

# CPU only (any platform)
pip install llama-cpp-python

# Verify installation
python -c "from llama_cpp import Llama; print('llama-cpp-python ready')"

The CMAKE_ARGS environment variable tells the build system which GPU backend to compile in. The package compiles llama.cpp from source during pip install, which takes 2-5 minutes. If you have a pre-built binary available for your platform, you can install from it to skip compilation — check the llama-cpp-python GitHub releases for platform-specific wheels.

Basic Usage: llama-cpp-python

from llama_cpp import Llama

# Load model (GPU layers: -1 = all on GPU)
llm = Llama(
    model_path="./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
    n_ctx=4096,       # context window
    n_gpu_layers=-1,  # -1 = all layers on GPU
    verbose=False
)

# Chat completion (OpenAI-compatible interface)
response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what GGUF format is."}
    ],
    max_tokens=500,
    temperature=0.2
)
print(response["choices"][0]["message"]["content"])

# Simple text completion
output = llm(
    "The capital of Australia is",
    max_tokens=50,
    stop=["\n"],
    echo=False
)
print(output["choices"][0]["text"])

The Llama class loads the model into memory when instantiated — this takes 2-10 seconds depending on model size. Keep the instance alive and reuse it across calls rather than creating a new instance per request. The create_chat_completion method mirrors the OpenAI API structure, making it easy to adapt code written for the OpenAI Python library.

Streaming with llama-cpp-python

from llama_cpp import Llama

llm = Llama(model_path="./models/model.gguf", n_gpu_layers=-1, n_ctx=4096)

# Stream tokens as they generate
for chunk in llm.create_chat_completion(
    messages=[{"role": "user", "content": "Write a haiku about autumn."}],
    max_tokens=100,
    stream=True
):
    delta = chunk["choices"][0].get("delta", {})
    if "content" in delta:
        print(delta["content"], end="", flush=True)
print()  # newline at end

Figure 1 — Ollama vs llama-cpp-python: Choosing the Right Interface

FactorOllama Python libraryllama-cpp-pythonSetuppip install ollama (server runs separately)pip install + compile (~3 min)ArchitectureHTTP client to Ollama serverDirect in-process inferenceModel managementAutomatic (ollama pull)Manual GGUF filesOverhead per request~5ms HTTP + serverMinimal (in-process)C-level API accessNoYes (logits, embeddings, state)Best forApps, scripts, API integrationResearch, custom inference, no server

Speed Comparison: Ollama vs llama-cpp-python

For a single request, llama-cpp-python has a slight latency advantage over Ollama because it eliminates the HTTP round-trip and server serialisation overhead. In practice for interactive use (one request at a time, responses taking seconds), the difference is imperceptible — both feel the same. The difference becomes relevant in high-frequency automated scenarios: batch processing many short requests, or building a pipeline where you make hundreds of inference calls per minute. In these cases, the in-process llama-cpp-python approach saves 5-20ms per request (the HTTP overhead), which compounds across thousands of requests. For applications that need maximum throughput, llama-cpp-python eliminates the network layer. For everything else, the convenience of Ollama’s model management and persistent server is worth the minimal overhead.

Advanced llama-cpp-python: Logits and Embeddings

The primary reason to use llama-cpp-python over Ollama’s API is access to capabilities that the Ollama API does not expose: raw logits (the probability distribution over the vocabulary before sampling), model state (the KV cache), and low-level embedding extraction. These are useful for research applications, custom sampling strategies, and building tools that need to inspect or manipulate the inference process:

from llama_cpp import Llama
import numpy as np

llm = Llama(
    model_path="./models/model.gguf",
    n_gpu_layers=-1,
    n_ctx=2048,
    logits_all=True  # enable logit access
)

# Get embeddings
llm_embed = Llama(
    model_path="./models/model.gguf",
    embedding=True,  # embedding mode
    n_gpu_layers=-1
)
embedding = llm_embed.create_embedding("Hello world")["data"][0]["embedding"]
print(f"Embedding dimensions: {len(embedding)}")

# Custom generation with logit inspection
tokens = llm.tokenize(b"The answer is")
llm.eval(tokens)
logits = np.array(llm.eval_logits[-1])  # last token logits
top5 = np.argsort(logits)[-5:][::-1]
print("Top 5 next tokens:", [llm.detokenize([t]).decode() for t in top5])

Most applications do not need this level of access. If you are building a standard chat application, RAG system, or batch processing pipeline, the Ollama library is the better choice. If you are implementing custom sampling algorithms, building a token-level analysis tool, or doing research that requires inspecting model internals, llama-cpp-python gives you the access you need.

Grammar-Constrained Generation

One capability that llama-cpp-python provides and Ollama’s API does not (directly): grammar-constrained generation, which forces the model to output text that conforms to a specified grammar. This is powerful for structured output without relying on the model to format correctly:

from llama_cpp import Llama, LlamaGrammar

llm = Llama(model_path="./models/model.gguf", n_gpu_layers=-1)

# Force JSON output via grammar constraint
json_grammar = LlamaGrammar.from_string('''root ::= object
object ::= "{" ws (pair ("," ws pair)*)? "}" ws
pair ::= key ":" ws value
key ::= "\"" [a-zA-Z_]+ "\""
value ::= string | number | "true" | "false" | "null"
string ::= "\"" [a-zA-Z0-9 _.-]* "\""
number ::= [0-9]+
ws ::= [ \t\n]*''')

result = llm(
    "Extract name and age from: John Smith is 34 years old.\n\nJSON:",
    max_tokens=100,
    grammar=json_grammar
)
print(result["choices"][0]["text"])

Grammar constraints guarantee the output format at the inference level — the model cannot generate tokens that violate the grammar. This is more reliable than prompting the model to output JSON and parsing the result, particularly for smaller models that sometimes fail to follow format instructions consistently.

When to Use Which

The decision is straightforward: use the Ollama Python library for applications and scripts where you want good model management, clean API compatibility, and the convenience of the Ollama ecosystem. Use llama-cpp-python when you need to embed inference directly in a Python process without a running server, when you need raw logit access or grammar constraints, when you are deploying in an environment where running a separate server process is inconvenient, or when you want the absolute minimum latency for high-frequency inference calls. Both libraries are well-maintained and stable. Starting with Ollama and switching to llama-cpp-python only when a specific requirement demands it is the practical approach — most applications never need what llama-cpp-python provides beyond what Ollama already offers.

Using llama-cpp-python as an OpenAI Drop-In

llama-cpp-python also ships a server mode that provides an OpenAI-compatible HTTP API, similar to llama-server and Ollama. This is useful when you want the in-process control of llama-cpp-python during development but the HTTP API compatibility for production or for tools that require a network endpoint:

# Start llama-cpp-python's built-in server
python -m llama_cpp.server \
  --model ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --n_gpu_layers -1 \
  --n_ctx 4096 \
  --port 8000

# Now accessible at http://localhost:8000/v1/chat/completions
# compatible with any OpenAI client

Between Ollama, llama-server (the llama.cpp binary), and llama-cpp-python’s server mode, you have three ways to run an OpenAI-compatible local API. The differences: Ollama manages models automatically and keeps a server warm between restarts; llama-server is the lowest overhead direct binary; llama-cpp-python’s server gives you Python-level customisation of the server behaviour. For most cases, Ollama is the clear winner for API serving. llama-cpp-python’s server is primarily useful when you need to modify server behaviour at the Python level, such as adding custom middleware, logging, authentication, or pre/post-processing of requests.

Migration from Ollama to llama-cpp-python

If you have working code using the Ollama Python library and want to switch to llama-cpp-python for a specific use case, the migration is straightforward because both expose an OpenAI-compatible interface:

# Original Ollama code
import ollama
response = ollama.chat(
    model="llama3.1",
    messages=[{"role": "user", "content": "Hello"}]
)
text = response["message"]["content"]

# Equivalent llama-cpp-python code
from llama_cpp import Llama
llm = Llama(model_path="./llama3.1.gguf", n_gpu_layers=-1)
response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello"}]
)
text = response["choices"][0]["message"]["content"]

The main differences: the Llama object must be instantiated with the model path rather than a model name, there is no model management layer (you specify the file directly), and the response format matches the OpenAI API structure rather than Ollama’s slightly different format. For most application logic, the change is limited to the model loading and response parsing lines — the rest of the code remains identical.

Performance Tuning llama-cpp-python

A few settings that meaningfully affect llama-cpp-python performance beyond the GPU layers flag: n_batch controls how many tokens are processed in parallel during prompt evaluation — higher values (512-2048) speed up processing of long prompts at the cost of more peak memory. n_threads sets the number of CPU threads for operations not offloaded to GPU — on Apple Silicon, setting this to the number of performance cores (typically 6-12 depending on chip) rather than total logical cores gives better CPU inference performance. use_mmap=True (the default) allows the OS to memory-map the model file rather than loading it entirely into RAM, which reduces startup time and allows the OS to manage model paging — useful on machines where the model is close to the memory limit. These flags go in the Llama constructor: Llama(model_path=..., n_gpu_layers=-1, n_batch=512, n_threads=8, use_mmap=True).

Summary: The Inference Stack in Full

Understanding the full local LLM inference stack helps clarify where llama.cpp and Ollama fit. At the bottom is the hardware: GPU with Metal (Mac) or CUDA (NVIDIA), or CPU. Above that is the inference kernel layer: llama.cpp’s C++ code that implements the actual transformer computations, optimised for each hardware backend. Above that is the model and quantization layer: the GGUF file containing the model weights in a specific quantization format, loaded and used by llama.cpp. Above that are the interface layers: llama-cpp-python (Python bindings to the C layer), llama-server (HTTP server binary), Ollama (managed server with model library and API), and MLX (Apple’s alternative implementation of the same computations). Above these are the application libraries: the Ollama Python library, the llama-cpp-python high-level API, and any OpenAI-compatible client. At the top are user-facing applications: Open WebUI, LM Studio, VS Code extensions, custom scripts. Each layer adds convenience and removes control relative to the layer below. Choosing where to work in this stack is a practical decision based on what you need to do: most users should be at the application or library layer; researchers and developers building novel applications may need to go lower.

Leave a Comment