Two of the most practical aspects of llama.cpp that trip up new users: understanding how context length works and how to get the best out of it, and setting up llama-server as a persistent OpenAI-compatible API endpoint. This guide covers both thoroughly — the theory, the commands, and the trade-offs.
How Context Length Works in llama.cpp
Context length (often called context window or n_ctx) defines how many tokens the model can process at once — your system prompt, the full conversation history, and the response being generated all count toward this limit. Models have a maximum supported context length baked into their architecture (typically 4096, 8192, or 128K for modern models), but llama.cpp defaults to a much lower value (512 in older versions, 2048 in recent ones) to conserve memory. This default is almost always too low for practical use.
Memory cost of context: the KV (key-value) cache that holds the attention state grows linearly with context length. For a 7B Q4_K_M model, each 1024 tokens of context adds roughly 200-400MB of memory, depending on model architecture and the number of attention layers. Increasing from 2048 to 8192 context on a 7B model might add 1-2GB to memory usage. This is usually affordable — the model weights themselves use 4-5GB, and going from 2048 to 8192 context might take total memory from 5GB to 6.5GB, well within a 12GB GPU.
Setting Context Length: The Right Values
# llama-cli: set context window
./build/bin/llama-cli \
-m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-c 8192 \
-ngl 99 \
--chat-template llama3 \
-i
# For document analysis (needs long context)
./build/bin/llama-cli \
-m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-c 32768 \
-ngl 99 \
--flash-attn \
-iPractical context sizes to use: 4096 for most interactive chat (covers most conversations without excessive memory use), 8192 for technical sessions with longer exchanges or moderate document context, 16384 for document analysis and longer code sessions, 32768 for working with substantial documents, 65536+ for very long document work (requires ample GPU memory and Flash Attention). Never set context higher than you actually need — it slows generation and wastes memory. If your use case is short Q&A, 2048-4096 is more efficient than 128K.
Flash Attention: Essential for Long Context
Flash Attention restructures how attention computation is performed to reduce the KV cache memory usage and speed up generation at long context lengths. Enable it whenever you use context lengths above 8192:
# Enable Flash Attention (recommended for contexts > 8192)
./build/bin/llama-cli \
-m ./models/model.gguf \
-c 32768 \
-ngl 99 \
--flash-attn \
-i
# Build with Flash Attention support:
cmake -B build -DGGML_METAL=ON -DGGML_FLASH_ATTN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(sysctl -n hw.logicalcpu)At 32K context, Flash Attention typically saves 30-50% of KV cache memory and improves generation speed by 15-30%. At 4K context, the benefit is smaller (5-10%) but still positive with no downside. Enable it by default for all inference if your build supports it.
Context Overflow: What Happens When You Hit the Limit
When a conversation fills the context window, llama.cpp must handle the overflow. The default behaviour is to stop accepting new tokens once the context is full — the server returns an error or the CLI stops responding. To handle long conversations gracefully, use the context shift flag:
# Enable context shifting (drops oldest tokens when full)
./build/bin/llama-cli \
-m ./models/model.gguf \
-c 8192 \
-ngl 99 \
--ctx-size-reserved 512 \
-i
# Or in the server:
./build/bin/llama-server \
-m ./models/model.gguf \
-c 8192 \
-ngl 99 \
--ctx-shiftContext shifting drops the oldest tokens (early conversation history) when the context fills up, allowing the conversation to continue at the cost of the model “forgetting” the beginning. This is often preferable to a hard stop, especially for long interactive sessions. Ollama implements this automatically with its keep_alive and context management behaviour.
Figure 1 — Context Length vs Memory Usage (7B Q4_K_M, without Flash Attention)
Setting Up llama-server as a Persistent API
For developers who want a stable OpenAI-compatible endpoint running in the background, llama-server provides exactly this. Unlike llama-cli which is interactive, llama-server accepts HTTP requests and can serve multiple clients:
# Start a persistent server
./build/bin/llama-server \
-m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-c 8192 \
-ngl 99 \
--flash-attn \
--host 127.0.0.1 \
--port 8080 \
--ctx-size 8192 \
--parallel 1 \
--chat-template llama3
# Test the endpoint
curl -s http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 50}' \
| python3 -m json.toolKey server flags: --host 127.0.0.1 restricts to localhost only (use 0.0.0.0 to allow network access); --parallel N allows N simultaneous requests; --chat-template applies the correct format for your model; --flash-attn enables Flash Attention for the server.
Using the Server from Python
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="llama-cpp" # any non-empty string
)
# Non-streaming
response = client.chat.completions.create(
model="llama", # model name doesn't matter for llama-server
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain attention mechanisms briefly."}
],
max_tokens=300,
temperature=0.3
)
print(response.choices[0].message.content)
# Streaming
for chunk in client.chat.completions.create(
model="llama",
messages=[{"role": "user", "content": "Count from 1 to 10."}],
max_tokens=100,
stream=True
):
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)The OpenAI Python library works directly with llama-server because it exposes a fully compatible API. Switch your code between llama-server, Ollama, and the real OpenAI API by changing only the base_url parameter.
Running llama-server as a System Service
For a Mac that functions as a local AI server, set up llama-server to start automatically at login using a LaunchAgent:
cat > ~/Library/LaunchAgents/com.llamacpp.server.plist << EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.llamacpp.server</string>
<key>ProgramArguments</key>
<array>
<string>/Users/you/llama.cpp/build/bin/llama-server</string>
<string>-m</string>
<string>/Users/you/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf</string>
<string>-c</string><string>8192</string>
<string>-ngl</string><string>99</string>
<string>--flash-attn</string>
<string>--host</string><string>127.0.0.1</string>
<string>--port</string><string>8080</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
</dict>
</plist>
EOF
# Load the service
launchctl load ~/Library/LaunchAgents/com.llamacpp.server.plist
# Check status
launchctl list | grep llamacppThe KeepAlive key ensures the service restarts automatically if it crashes. Substitute the paths with your actual llama.cpp build location and model path. On Linux, use a systemd user service with the same server command.
Performance Tuning the Server
Several server-specific settings affect throughput and latency. --parallel N enables N concurrent inference requests — each parallel slot needs its own KV cache allocation, so memory usage increases linearly. --cont-batching enables continuous batching, which allows new requests to start processing while other requests are mid-generation, improving throughput for multiple concurrent users at the cost of slightly higher per-request latency. --mlock locks the model in RAM, preventing the OS from paging it out during periods of inactivity — useful on machines running llama-server alongside other memory-hungry applications. --threads-http N sets the number of HTTP server threads for handling connections. For a single-user personal server, the defaults (parallel=1, no continuous batching) are appropriate. For a server handling multiple users, increase parallel and enable continuous batching while monitoring GPU memory to ensure you stay within limits.
Context Length in Ollama vs Direct llama.cpp
When using Ollama rather than direct llama.cpp, context length behaves similarly but is managed differently. Ollama defaults to 2048 tokens context for all models regardless of what they support. This is conservative — most modern models support 8K to 128K context — and almost always worth increasing. Set it via the num_ctx parameter in your API call, in your Modelfile, or via the OLLAMA_MAX_CONTEXT_LENGTH environment variable. The underlying llama.cpp inference is identical; Ollama just adds the management layer on top. Flash Attention in Ollama is enabled via the OLLAMA_FLASH_ATTENTION=1 environment variable and applies to all models on that server. The memory formula is the same as direct llama.cpp: model weights plus KV cache, where KV cache scales with context length and is reduced 30-50% with Flash Attention enabled. If you are hitting memory limits with long context in Ollama, enabling Flash Attention is usually the first thing to try before reducing context length or switching to a smaller model.
Measuring Context Utilisation
To understand how much of your context window is actually being used during a session, llama-server exposes a health and metrics endpoint. In direct llama.cpp, the verbose output during generation shows context fill percentage. For Ollama, the ollama ps command shows the context size in use for loaded models. Monitoring context utilisation helps identify when you need a larger window (you are hitting the limit and conversation quality is degrading) versus when you have headroom to reduce the context size for faster inference. A well-sized context window — large enough that you never hit the limit on typical interactions, small enough that you are not wasting GPU memory on unused KV cache — is worth the one-time effort to calibrate for your specific use case.
Rope Scaling: Extending Beyond Training Context
Some models support RoPE (Rotary Position Embedding) scaling, which allows inference at context lengths beyond what the model was trained on. This is an advanced technique that can extend a model trained on 8K context to work at 16K or 32K, at some quality cost. llama.cpp supports this via the --rope-scale and --rope-freq-base flags. Ollama passes these through when configured in a Modelfile. The quality of RoPE-extended inference varies significantly by model and the specific extension factor — modest extensions (2x) often work well, while aggressive extensions (8x+) degrade quality substantially. The safest approach is to use models that were trained at the context length you need, rather than extending shorter-context models via RoPE scaling. Llama 3.1 and similar modern models were trained at 128K context, making RoPE scaling largely unnecessary for most practical purposes.
Practical Context Length Recommendations
The right context length depends on what you actually do with the model. For an interactive coding assistant in a VS Code extension, 8192 tokens covers the file context and conversation history needed for most sessions. For a document Q&A RAG pipeline, the context window only needs to hold the retrieved chunks and the answer — typically 4096 is sufficient if your chunks are well-sized. For reading an entire book chapter, 32768 may be needed. For working with a full codebase in a single context window (a common aspiration), 128K is the practical minimum and most 7B models slow noticeably at this length. The pattern: set context to what your actual use case needs, not the maximum the model supports. If you are not sure, start at 8192, watch whether you hit the limit during typical sessions, and increase if you consistently do. Default to enabling Flash Attention always — there is no downside, and the memory saving at long context is substantial.
Monitoring Context Usage During a Session
llama-server's built-in web interface (accessible at http://localhost:8080 when running) shows real-time context utilisation. The API also exposes it: a call to http://localhost:8080/slots returns the current context fill for each active slot. For command-line monitoring, llama-cli's verbose mode prints the number of tokens in context after each generation. Track this over a typical work session to understand your actual context needs — most users are surprised how little context they actually consume on typical tasks, which means they can use a shorter context window and get faster inference than they assumed they needed. The opposite is also true for users doing long document work: they often hit context limits they did not realise were there, causing degraded output quality that they attributed to the model rather than context overflow.
Quick Reference: Key Context and Server Flags
The most important flags consolidated for easy reference. For llama-cli and llama-server, both accept these options. Context: -c N or --ctx-size N sets the context window. Flash Attention: --flash-attn enables it (add DGGML_FLASH_ATTN=ON at build time). GPU layers: -ngl 99 for all layers on GPU. Context shift: --ctx-shift enables graceful handling of context overflow. Chat template: --chat-template llama3 (or mistral, chatml, etc.) for correct formatting. For the server specifically: --host and --port set the network endpoint; --parallel N enables concurrent requests; --cont-batching enables continuous batching for multi-user efficiency; --mlock prevents model paging. These flags, combined thoughtfully based on your specific hardware and use case, give you a well-tuned llama.cpp inference setup that matches or exceeds what higher-level tools provide for most applications.