Apple MLX-LM makes it straightforward to run popular open-weights models directly on Apple Silicon Macs — Llama, Mistral, Qwen, Gemma, Phi, and many others. This guide covers the specific setup for the most commonly used models, performance expectations on different Mac configurations, and how to get the best results from each model family.
Finding MLX Models on Hugging Face
MLX models are hosted on Hugging Face under the mlx-community organisation. The naming convention is consistent: the base model name followed by the quantization level. mlx-community/Meta-Llama-3.1-8B-Instruct-4bit is the Llama 3.1 8B instruction-tuned model at 4-bit quantization. Browse huggingface.co/mlx-community for the full list, which includes hundreds of models. The most important models for daily use are all available, typically within days of the original release on Hugging Face.
Running Llama with MLX
The Llama 3.x family is among the most capable and widely used models in MLX format. The 3.1 series introduced 128K context support; the 3.2 series added vision capabilities and efficient smaller models (1B, 3B):
from mlx_lm import load, generate
# Llama 3.2 3B — fast, capable, great for interactive use on any M-series Mac
model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
# Llama 3.1 8B — strong general capability, recommended default
model, tokenizer = load("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")
# Llama 3.1 70B — best quality, needs 48GB+ unified memory
model, tokenizer = load("mlx-community/Meta-Llama-3.1-70B-Instruct-4bit")
# Generate with chat template
def llama_chat(model, tokenizer, message: str, system: str = None) -> str:
messages = []
if system:
messages.append({"role": "system", "content": system})
messages.append({"role": "user", "content": message})
prompt = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
return generate(model, tokenizer, prompt=prompt, max_tokens=1000)
print(llama_chat(model, tokenizer, "What is the capital of Australia?"))Performance on M4 Pro 24GB: Llama 3.2 3B at ~120-140 t/s, Llama 3.1 8B at ~55-70 t/s. On M4 Max 128GB: 8B at ~100-120 t/s, 70B at ~20-28 t/s.
Running Mistral with MLX
The Mistral family includes strong base models and the Mixtral MoE (mixture of experts) variants. Mixtral 8x7B uses 8 expert networks and routes each token through 2 of them, giving quality close to a 47B model at inference cost closer to a 13B model:
from mlx_lm import load, generate
# Mistral 7B — fast and capable
model, tokenizer = load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")
# Mistral Small 3 (24B) — significant quality jump over 7B
model, tokenizer = load("mlx-community/Mistral-Small-3-4bit")
# Mixtral 8x7B — quality of ~47B at ~13B inference cost
# Needs ~28GB unified memory for 4-bit
model, tokenizer = load("mlx-community/Mixtral-8x7B-Instruct-v0.1-4bit")
# Mistral uses [INST] template for older models
def mistral_chat(model, tokenizer, message: str) -> str:
if hasattr(tokenizer, "apply_chat_template") and tokenizer.chat_template:
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": message}],
tokenize=False, add_generation_prompt=True
)
else:
prompt = f"[INST] {message} [/INST]"
return generate(model, tokenizer, prompt=prompt, max_tokens=800)Mistral Small 3 (24B) is the practical sweet spot for users who want noticeably better quality than 7B but cannot run 70B — it fits in 16GB unified memory at 4-bit and generates at 40-55 t/s on M4 Pro, delivering quality close to 70B models on most tasks.
Running Qwen with MLX
The Qwen 2.5 family from Alibaba is among the most capable open-weights models, with strong performance on multilingual tasks, coding, and mathematics. The Qwen 2.5 Coder models are particularly strong for code generation:
from mlx_lm import load, generate
# Qwen 2.5 7B — strong general model with good multilingual support
model, tokenizer = load("mlx-community/Qwen2.5-7B-Instruct-4bit")
# Qwen 2.5 Coder 7B — best local coding model at this size
model, tokenizer = load("mlx-community/Qwen2.5-Coder-7B-Instruct-4bit")
# Qwen 2.5 72B — near-frontier quality, needs 48GB+ unified memory
model, tokenizer = load("mlx-community/Qwen2.5-72B-Instruct-4bit")
# Qwen models use ChatML format
def qwen_chat(model, tokenizer, message: str, system: str = "You are a helpful assistant.") -> str:
messages = [
{"role": "system", "content": system},
{"role": "user", "content": message}
]
prompt = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
return generate(model, tokenizer, prompt=prompt, max_tokens=1000)Figure 1 — MLX Model Performance by Mac Configuration
Running Phi and Gemma with MLX
Microsoft’s Phi-4 (14B) and Google’s Gemma models are both available in MLX format and worth knowing about:
from mlx_lm import load, generate
# Phi-4 — Microsoft's strong 14B model, excellent on reasoning and math
model, tokenizer = load("mlx-community/phi-4-4bit")
# Gemma 2 9B — Google's model, good quality-to-size ratio
model, tokenizer = load("mlx-community/gemma-2-9b-it-4bit")
# Gemma 2 27B — strong 27B model, fits in 16GB unified memory
model, tokenizer = load("mlx-community/gemma-2-27b-it-4bit")Phi-4 is particularly strong on analytical tasks, step-by-step reasoning, and mathematical problems — notable for its quality relative to its 14B size. Gemma 2 27B is one of the best models at the sub-30B tier, fitting comfortably in 16GB unified memory while delivering quality noticeably better than 7B models. Both follow the standard MLX chat interface with apply_chat_template.
Context Length Configuration
Most MLX models support 128K context in their base versions, but MLX-LM defaults to shorter context for memory efficiency. For long document work or extended conversations, increase the context window:
from mlx_lm import load, generate
# Load with extended context
model, tokenizer = load(
"mlx-community/Meta-Llama-3.1-8B-Instruct-4bit",
model_config={"max_position_embeddings": 32768} # extend to 32K
)
# Or use the generate max_tokens to control output length only
response = generate(
model, tokenizer,
prompt=prompt,
max_tokens=2000, # max output tokens
verbose=False
)Increasing context window consumes additional unified memory for the KV cache — at 32K context, add roughly 2-4GB per 7B model size. Monitor memory usage with Activity Monitor (GPU memory) to ensure you are not pushing into swap territory, which degrades performance significantly.
Benchmarking Your Setup
MLX-LM prints tokens per second during generation when verbose=True. For a more rigorous benchmark comparing multiple models on your specific hardware:
import time
from mlx_lm import load, generate
def benchmark_model(model_name: str, prompt: str, runs: int = 3) -> float:
model, tokenizer = load(model_name)
formatted = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=False, add_generation_prompt=True
)
rates = []
for _ in range(runs):
start = time.time()
# Count tokens to get accurate t/s
output = generate(model, tokenizer, prompt=formatted,
max_tokens=200, verbose=False)
elapsed = time.time() - start
# Approximate: 200 tokens / elapsed seconds
rates.append(200 / elapsed)
return sum(rates) / len(rates)
models = [
"mlx-community/Llama-3.2-3B-Instruct-4bit",
"mlx-community/Meta-Llama-3.1-8B-Instruct-4bit",
]
for m in models:
rate = benchmark_model(m, "Explain quantum computing.")
print(f"{m.split('/')[1]}: {rate:.0f} t/s")Recommended Starting Configurations
For users new to MLX on Mac, a practical starting progression: begin with Llama 3.2 3B on any M-series Mac to confirm MLX is working and get familiar with the interface. Then load Llama 3.1 8B as your primary model — it is the recommended default for general use, offering good quality at practical generation speed on any M-series Mac with 16GB+. If you have 24GB+ unified memory, test Mistral Small 3 (24B) for tasks where quality matters more than speed — the jump in output quality over 8B models is significant on analytical, creative, and complex tasks. For coding, switch to Qwen 2.5 Coder 7B as your primary model when the task is code-focused. Once you have settled on a model or two that covers your main use cases, the MLX setup is largely maintenance-free — models update occasionally on mlx-community and running a fresh load() picks up the latest version automatically.
Choosing Between Models: What Each Excels At
The model families available through MLX have different strengths that make them better choices for specific tasks, and understanding these differences saves you from running the wrong model for your use case. Llama 3.1 is the strongest general-purpose model family — versatile across writing, coding, reasoning, and conversation, with good instruction following and the ability to follow complex multi-step instructions reliably. It is the safe default when you are not sure which model to use. Mistral models are particularly strong on European languages and produce clean, concise prose — good for writing assistance where you want responses that do not ramble. The Mixtral MoE architecture gives quality close to a 47B model at the memory and speed cost of a 13B model, making it excellent value for users who want the best quality possible within a constrained memory budget. Qwen 2.5 models lead on multilingual tasks (particularly Chinese, Japanese, Korean), mathematics, and coding — if your work spans multiple languages or requires strong mathematical reasoning, Qwen 2.5 consistently outperforms same-size Llama models on these specific tasks. Phi-4 is optimised for reasoning-heavy tasks — logical problems, step-by-step analysis, structured thinking — where its training on high-quality reasoning data gives it an edge over models trained on broader but less curated corpora. Gemma 2 from Google is well-balanced and reliable, with strong safety tuning making it a good choice when you want outputs that are consistently appropriate for a wide range of contexts.
Managing Multiple Models
Unlike Ollama, which manages model storage and switching automatically, MLX downloads models to the Hugging Face cache directory (~/.cache/huggingface) and you load them explicitly in your Python code. Models are large files — 5-40GB each — and disk space management becomes important if you are experimenting with many models. The huggingface_hub library provides tools for managing the cache:
# List cached models
python -c "from huggingface_hub import scan_cache_dir; print(scan_cache_dir())"
# Delete a specific cached model
huggingface-cli delete-cacheA practical approach: keep only the two or three models you use regularly in the cache, and re-download others when needed. Ollama’s model management is significantly more convenient for users who frequently switch between many models — another reason to use Ollama alongside MLX rather than MLX exclusively.
MLX vs Ollama Quality: Are They the Same?
For the same base model at the same quantization level, MLX and Ollama produce effectively identical output quality — both run the same model weights with similar quantization. The quantization methods differ slightly (MLX uses Apple’s quantization format, Ollama uses GGUF), which can introduce minor differences in outputs at the bit level, but these are not perceptible in practice. If you load Llama 3.1 8B in both MLX and Ollama and run the same prompt, the outputs will be similar in quality, style, and accuracy — just not bit-for-bit identical. The choice between MLX and Ollama for a given task should be based on workflow requirements (fine-tuning, API compatibility, model availability) rather than output quality, which is effectively equivalent at the same model scale.
Common Issues and Solutions
A few issues come up regularly when getting started with MLX on Mac. If you see out-of-memory errors when loading a model, the model does not fit in your available unified memory — try a smaller quantization (4-bit instead of 8-bit) or a smaller model. If generation is very slow (under 5 t/s), MLX may be falling back to CPU — check that you are on Apple Silicon (not Intel Mac) and that your macOS is up to date. If the model produces unexpected output, verify you are applying the correct chat template — each model family has specific formatting requirements, and the apply_chat_template method from the tokenizer handles this automatically when the model’s tokenizer config includes a template. If a model you want is not in mlx-community, use mlx_lm.convert to convert it from the original Hugging Face weights — the conversion process typically takes 10-30 minutes and the result is compatible with all MLX-LM functions.
The Three-Line MLX Quick Start
For those who want to verify everything is working before reading any further: install faster-whisper with pip install mlx mlx-lm, open a Python file, paste the three lines from mlx_lm import load, generate, model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit"), and print(generate(model, tokenizer, prompt="Hello, what can you do?", max_tokens=100, verbose=True)), and run it. You should see the model download on first run, then tokens generate at 100+ t/s on any modern M-series Mac. That is the complete MLX proof-of-concept — everything else in this guide builds on those three lines.