How to Run Qwen 2.5 72B Locally with Ollama

by

Qwen 2.5 72B is Alibaba’s largest open-weight model and one of the most capable you can run locally. At 72 billion parameters it’s in the same tier as Llama 3.1 70B and Mixtral 8x22B, but it benchmarks ahead of both on several tasks — particularly coding, mathematics, and multilingual work. If your hardware can handle 40+ GB of memory, it’s worth serious consideration as your primary large local model.

Hardware Requirements

In Q4 quantization, Qwen 2.5 72B weighs approximately 43–47GB depending on the specific variant. Here’s what you realistically need:

Apple Silicon with 64GB+ unified memory (M4 Max 64GB, M3 Max 96GB, M4 Ultra, M3 Ultra): The most practical consumer setup. The high memory bandwidth of Apple Silicon makes this surprisingly fast — 15–25 tokens per second on M4 Max, 20–30+ on Ultra configurations. The unified memory means the full model loads without VRAM constraints.

NVIDIA with 48GB+ VRAM (dual RTX 3090 NVLink, RTX 6000 Ada, A6000): Full GPU inference at 20–35 tokens per second. Dual 3090s with NVLink at 48GB VRAM total is just enough for the Q4 model with a reasonable context length.

NVIDIA 24GB GPU + 32GB+ system RAM: Partial offloading — roughly 25–30 layers on GPU, rest on CPU RAM. Expect 6–12 tokens per second. Usable but slow for interactive work; better for batch processing.

CPU only with 64GB RAM: 1–3 tokens per second. Viable for overnight batch jobs, impractical for interactive use.

Pulling Qwen 2.5 72B with Ollama

ollama pull qwen2.5:72b

Downloads the default Q4_K_M quantized version — approximately 47GB. Plan for a significant download. For a smaller memory footprint at some quality cost:

ollama pull qwen2.5:72b-instruct-q2_K   # ~26GB, larger quality drop
ollama pull qwen2.5:72b-instruct-q4_K_M # ~47GB, recommended
ollama pull qwen2.5:72b-instruct-q6_K   # ~60GB, higher quality

The Q2 variant at 26GB fits on hardware that can’t handle 47GB — an RTX 4090 (24GB) with 32GB system RAM can handle it via partial offloading, and the quality, while lower than Q4, is still competitive with many 14–30B dense models.

What Makes Qwen 2.5 72B Stand Out

Coding. Qwen 2.5 was trained with a very large code corpus, and the 72B version is one of the strongest locally-runnable models for complex coding tasks. It handles multi-file reasoning, architecture discussions, algorithm design, and code review at a level that rivals what you’d get from GPT-4o mini on coding benchmarks. For teams that want strong local coding assistance without a dedicated coding model, it covers both general and coding tasks well.

Mathematics. The MATH benchmark scores for Qwen 2.5 72B are notably high — it solves competition-level math problems that trip up many larger models. If you use a local LLM for quantitative reasoning, financial modelling, or step-by-step mathematical work, this is one of the best options.

Multilingual. Qwen’s Chinese-language capability is unmatched among open-weight models — it was trained extensively on high-quality Chinese text and produces native-quality output. European languages are also strong. If your use case involves any non-English content, Qwen 2.5 72B is significantly ahead of Llama and Mistral alternatives.

Long context. With 128K context window support and good performance across long inputs, it handles long-document tasks well. The quality of attention over long contexts is better than many models nominally supporting the same window size.

Figure 1 — Qwen 2.5 72B: Hardware Options and Speed

Hardware Memory Mode Speed (t/s) Verdict M4 Ultra / M3 Ultra128–192GB unifiedFull Metal GPU25–35Best M4 Max / M3 Max (64GB)64GB unifiedFull Metal GPU15–25Excellent Dual RTX 3090 NVLink48GB VRAMFull GPU20–30Great RTX 4090 + 32GB RAM24GB + 32GB splitPartial offload6–12Slow CPU only (64GB RAM)64GB system RAMCPU only1–3Batch only Q4_K_M quantization (~47GB). Actual speed varies by CPU/GPU generation and available memory bandwidth.

Running Qwen 2.5 72B

ollama run qwen2.5:72b

First load takes 45–90 seconds as the model moves into memory. Once loaded, generation begins quickly. Run with --verbose to confirm GPU layers are active:

ollama run qwen2.5:72b --verbose

Look for gpu_layers in the output — a high number (close to the model’s total layers) means most inference is running on GPU. On Apple Silicon, all layers should be on Metal GPU automatically.

Increase context length for document work:

ollama run qwen2.5:72b --parameter num_ctx 32768

Remember that larger context requires significantly more memory on top of the base model size. On 64GB unified memory, a 32K context with Qwen 2.5 72B Q4 is close to the limit — keep an eye on memory pressure in Activity Monitor if you’re on a Mac.

Qwen 2.5 72B vs Llama 3.1 70B: Which to Choose?

These two models sit in the same hardware tier and are the main comparison for anyone with 48GB+ VRAM or 64GB+ unified memory. The honest assessment:

Qwen 2.5 72B wins on: coding tasks (meaningfully better on HumanEval and MBPP), mathematics (higher MATH benchmark scores), Chinese and multilingual content (significantly better), structured output and instruction following precision.

Llama 3.1 70B wins on: English-language reasoning tasks where it slightly edges Qwen, broader community tooling and integration support, slightly more conservative outputs that are less likely to be verbose.

For most technical users — developers, data scientists, researchers — Qwen 2.5 72B is the better choice. The coding and math advantages are real and meaningful in practice. For general-purpose English-language chat and writing tasks, either model performs excellently and the choice comes down to preference. The multilingual advantage of Qwen is decisive for any non-English use case.

Figure 2 — Qwen 2.5 72B vs Llama 3.1 70B: Benchmark Comparison

Benchmark Qwen 2.5 72B Llama 3.1 70B MMLU (general knowledge)86.1%82.6% HumanEval (coding)86.6%80.5% MATH (mathematics)83.1%68.0% Multilingual (C-Eval, others)Significantly strongerGood (EN-focused)

Using Qwen 2.5 72B via the API

import ollama

response = ollama.chat(
    model='qwen2.5:72b',
    messages=[
        {
            'role': 'system',
            'content': 'You are an expert software engineer. Provide detailed, production-quality code.'
        },
        {
            'role': 'user',
            'content': 'Write a Python class that implements a thread-safe LRU cache with TTL expiration.'
        }
    ],
    options={
        'num_ctx': 16384,
        'temperature': 0.2  # lower for more precise code output
    }
)
print(response['message']['content'])

Qwen 2.5 72B’s coding output is noticeably more detailed and production-aware than smaller models — it considers edge cases, adds appropriate error handling, and includes docstrings without being asked. For code review, architecture discussion, and complex algorithmic problems, lowering temperature to 0.1–0.3 produces more consistent and correct outputs than the default.

The Qwen 2.5 Family: Other Sizes Worth Knowing

The Qwen 2.5 family covers a wide range, and it’s worth knowing where the 72B sits relative to its siblings. The 7B and 14B models are excellent for their size and run on much more accessible hardware — the 7B on any 8GB machine, the 14B on 16GB. The 32B (about 20GB in Q4) is an underappreciated sweet spot: better than the 14B on complex tasks, fits on a 24GB GPU, and runs at a comfortable 20–35 tokens per second on mid-range hardware. If you find the 72B’s hardware requirements prohibitive, the 32B is worth trying before dropping all the way to 14B — it captures much of the quality improvement from the larger model at roughly half the hardware cost. And if coding is your primary use case, the dedicated Qwen 2.5 Coder 7B and 32B variants are specifically tuned for code and benchmarks higher than the general 72B model on pure coding tasks, while fitting on significantly smaller hardware. Having one large general model (72B if your hardware supports it, 32B otherwise) alongside a fast coding specialist (Qwen 2.5 Coder 7B) is an efficient local model setup that covers most use cases without constantly loading and unloading models.

Optimising for Your Hardware

A few configuration tweaks make a meaningful difference for Qwen 2.5 72B specifically. Enable Flash Attention before starting Ollama — it reduces memory usage at long context lengths and speeds up generation by 10–20% on compatible hardware: set OLLAMA_FLASH_ATTENTION=1 as an environment variable. On Apple Silicon, Flash Attention via Metal is particularly effective. Set OLLAMA_KEEP_ALIVE to a long duration (30 minutes or more) when you’re actively using Qwen 2.5 72B — the 45–90 second load time is significant enough that reloading between queries adds up quickly during a working session. For partial GPU offloading on NVIDIA hardware, tune num_gpu carefully: start at the automatic value Ollama chooses, then increment and check ollama ps to confirm the model still loads without VRAM overflow. On systems right at the memory threshold, reducing context length (num_ctx 8192 instead of 32768) frees enough memory to load a few more GPU layers, which can improve tokens per second more than the context reduction costs in usability. Monitor GPU utilisation with nvidia-smi dmon during inference — if it stays below 80%, you have headroom to increase GPU layers; if it’s at 100% with memory pressure, you’re at the limit and should reduce context or consider a lighter quantization.

Is Qwen 2.5 72B Worth the Hardware Investment?

The honest answer: if you have 64GB+ unified memory or 48GB+ VRAM and you use a local LLM regularly for technical work — coding, mathematics, analysis, multilingual tasks — yes, it’s worth it. The quality gap between 72B-class models and 14B models is real and noticeable on complex tasks, not just on benchmarks. The gap between Qwen 2.5 72B and Llama 3.1 70B is smaller but meaningful on the specific tasks Qwen was optimised for. If you’re buying hardware specifically to run local LLMs, the M4 Max MacBook Pro with 64GB unified memory is the most practical consumer machine for Qwen 2.5 72B — it runs the model at 15–25 tokens per second, which is fast enough for interactive use, in a laptop form factor with no additional hardware required. For desktop users, a high-RAM system with an RTX 4090 (24GB) and 64GB system RAM gives partial offloading performance around 8–12 tokens per second — slower, but workable for non-interactive workflows. The model rewards the hardware investment with outputs that, on many technical tasks, are difficult to distinguish from what you’d get from a frontier API model.

Leave a Comment