Mixtral 8x7B was a landmark release when it came out — a Mixture of Experts model that delivers performance competitive with much larger models by selectively routing each token through just two of its eight expert networks at inference time. This means you get the quality of a large model at the computational cost of a smaller one. In practice, it runs faster than you’d expect for something with 46.7 billion total parameters, and the quality on reasoning and language tasks is excellent.
This guide covers pulling and running Mixtral 8x7B with Ollama, what hardware you actually need, and how to get the most from it.
Hardware Requirements: What You Actually Need
Mixtral 8x7B is where hardware requirements get serious. The model has 46.7B total parameters across its 8 experts — but at any point during inference, only about 13B of those are active (two experts of ~7B each). This active parameter count is what matters for inference speed; the full 46.7B all need to fit in memory though.
In Q4 quantization (the default Ollama pull), the model weighs in at approximately 26GB. Here’s what that means for your hardware:
NVIDIA GPU with 24GB VRAM (RTX 4090, 3090): The model fits entirely in VRAM with a small margin. This is the sweet spot — expect 20–35 tokens per second, which is fast enough for comfortable interactive use. The RTX 4090 is genuinely the best consumer GPU option for Mixtral 8x7B.
Apple Silicon with 32GB+ unified memory (M4 Pro 48GB, M3 Max, M4 Max, or any Ultra): Runs well on unified memory. Expect 15–25 tokens per second depending on chip and memory bandwidth. The M4 Max with 48GB or 64GB unified memory handles it comfortably.
CPU with 32GB+ RAM: Technically works, but slow — expect 2–5 tokens per second. Viable for batch processing where you’re not waiting for interactive responses. On a fast 16-core CPU you might push 4–6 tokens per second with the Q4 model.
GPU with less than 24GB VRAM + CPU RAM: Partial offloading splits the model between VRAM and system RAM. On an RTX 4080 16GB with 32GB system RAM, you’d get around 10–18 tokens per second — slower than full GPU, meaningfully faster than CPU-only.
Pulling Mixtral 8x7B with Ollama
ollama pull mixtralThis downloads the default Q4_K_M quantized version — approximately 26GB. Plan for a significant download time on most connections. Once downloaded, it loads in about 30–45 seconds on machines with 24GB+ VRAM or 32GB+ RAM.
To pull a specific quantization:
ollama pull mixtral:8x7b-instruct-v0.1-q4_K_M # default, recommended
ollama pull mixtral:8x7b-instruct-v0.1-q6_K # higher quality, ~33GB
ollama pull mixtral:8x7b-instruct-v0.1-q2_K # smallest, ~15GB, quality dropsThe Q2 variant at 15GB is worth knowing about if you have a 16GB GPU — it’s the only way to run Mixtral 8x7B fully in VRAM on that hardware. Quality is noticeably reduced versus Q4 on complex tasks, but for straightforward queries it’s still better than most 7B models.
Running Mixtral 8x7B
ollama run mixtralThe first load is slow — the model is large and takes time to move into memory. Subsequent uses within the keep-alive window are instant since the model stays loaded. The --verbose flag is useful on first run to confirm GPU layers are being used:
ollama run mixtral --verboseLook for gpu_layers in the output. For full GPU operation on a 24GB card, you should see all layers on GPU. If you see partial layers, Ollama is splitting between GPU and CPU RAM — you’ll get slower inference but the model will still run.
Figure 1 — Mixtral 8x7B: Hardware Options and Expected Speed
How Mixture of Experts Works (and Why It Matters)
Understanding how Mixtral’s architecture works helps you use it more effectively and set realistic expectations. A standard dense model like Llama 3.1 8B uses all 8B parameters for every token it generates. Mixtral 8x7B has eight separate “expert” networks, each roughly 7B parameters — but a router selects only two experts for each token, using around 13B parameters at any given moment.
This has two practical consequences. First, inference speed: because only a fraction of the model’s total parameters are active at a time, Mixtral generates tokens at roughly the speed you’d expect from a 13B model, despite having the knowledge capacity of a much larger model. Second, memory: the router needs all 46.7B parameters available to choose from, so the entire model must be in memory — you can’t just load 13B worth and get the full benefit.
The quality advantage comes from specialisation. Different experts develop strength in different domains during training — some excel at code, others at reasoning, others at language. The router learns to dispatch tokens to the relevant experts, giving you specialist-quality outputs without the user needing to know which expert is handling which task.
What Mixtral 8x7B Is Good At
Mixtral’s strengths reflect its training and architecture. It’s particularly strong on tasks that benefit from broad knowledge and strong language modelling:
Long-form writing and reasoning. Complex analytical tasks, essay-length responses, and multi-step reasoning chains are where the quality advantage over smaller models shows most clearly. The 32K context window (increase via num_ctx) supports long conversations and documents.
Multilingual tasks. Mistral AI trained Mixtral with strong multilingual data, and it handles French, German, Spanish, Italian, and Portuguese particularly well — better than most open-source English-focused models at the same effective parameter count.
Code generation. Not a dedicated coding model, but capable across Python, JavaScript, and common languages. For mixed tasks (explain a concept, then implement it), the conversational quality alongside the code is strong.
Instruction following. The instruct fine-tune follows complex multi-part instructions reliably. System prompts work well and the model stays on task through long conversations.
Where it falls short compared to newer models: Mixtral 8x7B is now several years old, and models like Qwen 2.5 72B and Llama 3.1 70B have surpassed it on most benchmarks. If your hardware can handle a 70B model, you’ll generally get better results from a more recent model. Mixtral’s advantage is running at 70B-class speeds on hardware that would make a dense 70B model impractical.
Using Mixtral via the API
import ollama
response = ollama.chat(
model='mixtral',
messages=[
{
'role': 'system',
'content': 'You are an expert technical writer. Give thorough, well-structured answers.'
},
{
'role': 'user',
'content': 'Explain the difference between transformer attention mechanisms and state space models.'
}
],
options={'num_ctx': 8192}
)
print(response['message']['content'])Mixtral responds well to specific system prompts — the instruct fine-tune is good at role adherence. For technical tasks, setting a clear expert role in the system prompt consistently improves output quality.
Mixtral 8x22B: The Bigger Sibling
Mistral AI also released Mixtral 8x22B — a larger MoE model with 8 experts of approximately 22B parameters each. In Q4 quantization it weighs around 80GB, putting it out of reach for most consumer hardware. But if you have a multi-GPU setup or a Mac with 128GB+ unified memory (M4 Ultra, M3 Ultra), it’s worth knowing about:
ollama pull mixtral:8x22bThe quality improvement over 8x7B is significant on complex tasks, and it’s available under the Apache 2.0 license which allows commercial use without restrictions. For most users, 8x7B is the practical choice — but 8x22B is there if your hardware supports it.
Figure 2 — Mixtral 8x7B vs Comparable Models
Is Mixtral 8x7B Still Worth Running in 2026?
This is a fair question given that Mixtral 8x7B was released in late 2023 and the landscape has changed considerably. Honest answer: for most use cases on hardware that can handle it, Mistral Small 3 (24B dense) or Qwen 2.5 14B are better choices today — higher MMLU scores, better instruction following, stronger multilingual performance, and easier hardware requirements. If your machine has a 24GB GPU or 32GB+ unified memory, you’re better served by Mistral Small 3 at 14GB than Mixtral 8x7B at 26GB.
Where Mixtral 8x7B still has an edge is the specific case where you want near-70B quality at 13B active parameter inference speed — if you have 24GB VRAM and want the fastest possible large-model experience, Mixtral’s MoE architecture delivers tokens faster than a dense model of comparable quality. It’s also worth running if you have specific benchmarks that favoured it, or if you’re comparing architectures and want to understand MoE in practice.
For most people arriving at this guide: try Mistral Small 3 or Qwen 2.5 14B first. If those feel too small for your tasks and your hardware can handle Mixtral’s 26GB footprint, it’s worth the pull. The model still performs well and remains a strong choice for multilingual and long-form tasks — it’s just no longer the default recommendation for the 24GB VRAM tier that it was two years ago.
Optimising Mixtral for Your Hardware
A few configuration tweaks improve the Mixtral experience on specific hardware setups. On machines where the model is splitting between GPU and system RAM, increasing num_gpu gradually can help: ollama run mixtral --parameter num_gpu 56 — Mixtral has more transformer layers than a standard dense model, so the right value varies by hardware; start at the automatic value shown in ollama ps and increase incrementally. If you’re getting good GPU layers but slow tokens per second, check whether memory bandwidth is the bottleneck — on systems where VRAM bandwidth is saturated, reducing context length helps: --parameter num_ctx 2048 significantly reduces KV cache pressure during long generations. For Apple Silicon users, Mixtral benefits more than most models from enabling Flash Attention, which reduces memory usage during long-context inference: set OLLAMA_FLASH_ATTENTION=1 in your environment before starting Ollama. The bandwidth advantage of Apple Silicon’s unified memory architecture makes Mixtral run particularly well on M-series chips, and Flash Attention pushes that advantage further on long sessions.