ExLlamaV2 is an inference engine specifically optimised for NVIDIA GPUs running GPTQ-quantized models. If you have an NVIDIA GPU and want the fastest possible local LLM inference, ExLlamaV2 deserves serious consideration alongside llama.cpp. This guide compares the two engines on what actually matters for practical use: speed, model compatibility, ease of use, and ecosystem integration.
What ExLlamaV2 Is
ExLlamaV2 (by turboderp on GitHub) is a high-performance inference library for NVIDIA GPUs that uses custom CUDA kernels optimised specifically for transformer inference with GPTQ quantization. GPTQ (Generative Pre-Trained Transformer Quantization) is an alternative quantization method to GGUF’s k-quants approach. Where GGUF uses block-wise quantization applied uniformly, GPTQ uses second-order information (the Hessian of the loss) to find the optimal quantization for each weight — producing better quality at the same bit width, particularly at aggressive quantizations (3-4 bit). The ExLlamaV2 CUDA kernels are hand-optimised for speed on NVIDIA hardware, often achieving 20-30% faster generation than llama.cpp’s CUDA kernels for the same model size and quantization level.
Where ExLlamaV2 Wins
Raw generation speed on NVIDIA GPUs. ExLlamaV2 is consistently faster than llama.cpp for models that fit entirely in VRAM. For a 7B GPTQ model on an RTX 4090, ExLlamaV2 typically generates at 120-160 tokens/second versus llama.cpp’s 85-110 t/s — a 30-50% speed advantage. This gap narrows but remains meaningful on lower-end GPUs. If maximum tokens-per-second is your primary concern and you have an NVIDIA GPU, ExLlamaV2 is worth the additional setup complexity.
GPTQ quantization quality at 4-bit. GPTQ 4-bit models consistently show better output quality than GGUF Q4_K_M at equivalent size. The second-order quantization approach preserves more information about important weights, resulting in better perplexity scores and noticeably better outputs on complex reasoning tasks. For users where output quality at a given quantization level matters more than setup simplicity, GPTQ with ExLlamaV2 is worth considering.
EXL2 format. ExLlamaV2 introduced the EXL2 quantization format, which allows mixed-precision quantization — different layers of the model use different bit widths, with more bits allocated to layers that have more impact on quality. This produces better quality at a given average bit width than either GGUF or standard GPTQ. EXL2 models at 4-bit average are often competitive with GGUF Q5_K_M or Q6_K in quality at Q4 file size.
Where llama.cpp Wins
Platform support. llama.cpp runs on NVIDIA (CUDA), AMD (ROCm/HIP), Apple Silicon (Metal), Intel (SYCL), and CPU. ExLlamaV2 only runs on NVIDIA GPUs. If you are not on NVIDIA, llama.cpp is your only option between the two.
Ecosystem breadth. Ollama, LM Studio, Jan AI, and most local AI frontends use llama.cpp as their backend. The GGUF format is the universal standard for model distribution. ExLlamaV2 is supported by some frontends (Oobabooga natively, a few others) but not the mainstream tools. If you want to use standard tools and frontends, llama.cpp is the default.
Model availability. The GGUF model library is vastly larger than the GPTQ/EXL2 library. Virtually every model has a GGUF version; GPTQ and EXL2 versions exist for popular models but not all of them. For niche, fine-tuned, or newly released models, llama.cpp’s GGUF format likely has coverage before EXL2 does.
Large model support. ExLlamaV2 requires the model to fit entirely in VRAM — it does not support CPU offloading for oversized models. llama.cpp handles partial CPU offloading gracefully with -ngl, allowing models that slightly exceed VRAM to run at reduced speed. For users on 8-12GB VRAM GPUs who want to run 13B+ models, llama.cpp’s CPU offloading provides an option ExLlamaV2 does not.
Figure 1 — ExLlamaV2 vs llama.cpp Head-to-Head
Installing ExLlamaV2
# Requirements: Python 3.9+, CUDA 11.8+, NVIDIA GPU
# Install PyTorch with CUDA first
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
# Install ExLlamaV2
pip install exllamav2
# Verify
python -c "import exllamav2; print(exllamav2.__version__)"Running Inference with ExLlamaV2
from exllamav2 import ExLlamaV2, ExLlamaV2Cache, ExLlamaV2Config, ExLlamaV2Tokenizer
from exllamav2.generator import ExLlamaV2DynamicGenerator, ExLlamaV2Sampler
# Load model (EXL2 format)
config = ExLlamaV2Config("./models/Meta-Llama-3.1-8B-Instruct-exl2")
model = ExLlamaV2(config)
cache = ExLlamaV2Cache(model, lazy=True)
model.load_autosplit(cache)
tokenizer = ExLlamaV2Tokenizer(config)
# Set up generator
generator = ExLlamaV2DynamicGenerator(model=model, cache=cache, tokenizer=tokenizer)
sampler = ExLlamaV2Sampler.Settings.greedy() # or .default() for sampling
# Generate
prompt = "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nExplain GPTQ quantization.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
output = generator.generate(prompt=prompt, max_new_tokens=500, gen_settings=sampler)
print(output)TabbyAPI: ExLlamaV2 with an OpenAI-Compatible Server
Using ExLlamaV2 directly requires working with its Python API. TabbyAPI is a popular project that wraps ExLlamaV2 in an OpenAI-compatible HTTP server, making it accessible to the same tools that work with Ollama and llama-server:
git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI
pip install -r requirements.txt
# Configure model path in config.yml, then start:
python main.py
# API available at http://localhost:5000/v1With TabbyAPI running, any OpenAI-compatible tool can use ExLlamaV2 inference simply by pointing its base URL at localhost:5000 instead of localhost:11434 (Ollama) or the OpenAI API. This makes ExLlamaV2 accessible via Open WebUI, LangChain, and any other tool in the ecosystem.
Finding EXL2 and GPTQ Models
EXL2 and GPTQ models are hosted on Hugging Face, primarily by the turboderp account (creator of ExLlamaV2) for EXL2, and by TheBloke (legacy) and LoneStriker for GPTQ. The naming convention for EXL2: model name followed by the average bits per weight, such as Meta-Llama-3.1-8B-Instruct-5.0bpw-exl2 for 5.0 bits per weight. Higher bpw means better quality and larger file size. The EXL2 format allows fractional bit widths (3.5bpw, 4.25bpw, 5.0bpw) which is more flexible than GGUF’s fixed quantization tiers. For a practical starting point, 4.0bpw EXL2 is roughly comparable to Q4_K_M GGUF in quality, while 5.0bpw EXL2 competes with Q5_K_M GGUF and is often slightly better due to the GPTQ quantization approach.
The Decision: Which to Use
The choice between ExLlamaV2 and llama.cpp reduces to a simple decision tree. If you are not on an NVIDIA GPU: llama.cpp (ExLlamaV2 does not run). If you primarily use mainstream frontends like Ollama, Open WebUI, LM Studio: llama.cpp (better ecosystem integration). If you are on NVIDIA and want the fastest possible raw generation speed and are comfortable with additional setup: ExLlamaV2 via TabbyAPI. If you care about model availability and want access to the widest range of models including niche fine-tunes: llama.cpp and GGUF format. If you specifically need the best quality at 4-bit quantization and are on NVIDIA: ExLlamaV2 with EXL2 format. For most users, llama.cpp via Ollama is the right choice because it works immediately, integrates with everything, and its quality is excellent. ExLlamaV2 is a worthwhile exploration for NVIDIA users who want to push performance further once Ollama is working well — not a replacement for the mainstream stack, but a compelling alternative layer for power users.
Speculative Decoding and Batching in ExLlamaV2
ExLlamaV2 supports speculative decoding — a technique where a small draft model generates candidate tokens that are then verified by the main model in parallel, achieving effective generation speeds higher than the main model alone. With a well-matched draft model (typically a 1-3B model paired with a 7-13B main model), speculative decoding can increase generation speed by 1.5-2.5x on NVIDIA GPUs where the memory bandwidth allows. llama.cpp also supports speculative decoding (the –draft-model flag) but the implementation in ExLlamaV2 is more mature and better optimised. Dynamic batching in ExLlamaV2 allows multiple prompts to be batched together efficiently on the GPU, which is particularly useful for batch processing workloads where you are generating responses for many inputs. These advanced features make ExLlamaV2 a serious option for production inference workloads on NVIDIA hardware — not just for individual users but for teams running local AI servers that need to maximise throughput from a fixed GPU budget.
Practical Benchmark: 7B on RTX 4090
To make the speed comparison concrete: on an RTX 4090 with a 7B model, llama.cpp with CUDA achieves approximately 90-110 tokens per second for generation (eval), while ExLlamaV2 with EXL2 at equivalent quality achieves 130-160 tokens per second — a 40-50% speed advantage that is genuinely noticeable in interactive use. At this generation speed, ExLlamaV2 produces responses faster than most people can read them, which may or may not matter depending on whether you are doing interactive chat (where the speed difference is less relevant once it exceeds comfortable reading pace) or batch processing (where throughput directly affects how long jobs take). The prompt processing speed (how fast the model processes your input before starting to generate) is also higher in ExLlamaV2, which matters for RAG workloads that regularly process long retrieved documents before generating. For users on RTX 3080-4090 class hardware who run local LLMs heavily throughout the day, the ExLlamaV2 speed advantage compounds meaningfully over many sessions.
Getting Started: The Recommended Path
If you are new to local LLM inference on NVIDIA and are evaluating both options, the recommended starting sequence is: get Ollama working first (five minutes, handles everything automatically), run models through it for a few days to understand your actual use patterns, then benchmark ExLlamaV2 via TabbyAPI with an EXL2 version of the same model to compare the raw speed difference on your specific hardware. If the speed improvement matters for how you actually use the model, adopt ExLlamaV2 for your primary inference. If Ollama’s speed is sufficient for your use case, the simplicity and ecosystem integration of staying with Ollama is worth more than the marginal speed gain. Most users who try both end up keeping Ollama for its convenience and using ExLlamaV2 via TabbyAPI for specific high-throughput tasks. The two are not mutually exclusive — you can run both simultaneously on different ports and route different workloads to each.