MLX Advanced Guide: Quantization, Model List and vs llama.cpp

This guide covers the more advanced aspects of Apple MLX: how to quantize your own models, where to find the best community models, how MLX compares to llama.cpp in real-world performance, and practical Python workflows for building MLX-based applications. If you are new to MLX, the getting started guide earlier in this series covers the basics — this guide assumes MLX is already installed and you have run your first model.

MLX vs llama.cpp: Real Performance Comparison

Both MLX and llama.cpp (which Ollama uses via its Metal backend) accelerate inference on Apple Silicon using Metal GPU kernels, and in practice their performance is close. The differences are nuanced and model-dependent. MLX tends to perform better on Llama family models and recently released architectures where Apple has invested specific kernel optimisations. llama.cpp (via Ollama) tends to perform better on some older architectures and benefits from the broader community of GPU kernel contributors targeting the GGUF format. In benchmarks across dozens of models, the two frameworks typically land within 5-15% of each other on the same model and quantization, with neither consistently ahead across all conditions.

What the difference is not: a fundamental architectural advantage of one over the other. Both use the same underlying Metal GPU; the differences reflect kernel implementation details that change with each release. If you benchmark a specific model and find one framework noticeably faster, use that one for that model. If you haven’t benchmarked, either works — pick based on workflow requirements (Ollama for API compatibility and model management, MLX for fine-tuning and Python-level access).

from mlx_lm import load, generate
import time

def tps_benchmark(model_name: str, prompt: str = "Explain neural networks.", tokens: int = 200) -> float:
    model, tokenizer = load(model_name)
    formatted = tokenizer.apply_chat_template(
        [{"role": "user", "content": prompt}],
        tokenize=False, add_generation_prompt=True
    )
    # Warmup
    generate(model, tokenizer, prompt=formatted, max_tokens=50, verbose=False)
    # Benchmark
    start = time.time()
    generate(model, tokenizer, prompt=formatted, max_tokens=tokens, verbose=False)
    elapsed = time.time() - start
    tps = tokens / elapsed
    print(f"{model_name.split('/')[1]}: {tps:.1f} t/s")
    return tps

tps_benchmark("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")

Quantizing Your Own Models with MLX

If a model you want is not available in MLX format on Hugging Face, you can convert and quantize it yourself from the original Hugging Face weights. This takes 10-30 minutes depending on model size and produces a fully compatible MLX model:

# Convert and quantize a HuggingFace model to MLX 4-bit
python -m mlx_lm.convert \
  --hf-path meta-llama/Meta-Llama-3.1-8B-Instruct \
  --mlx-path ./llama-3.1-8b-mlx-4bit \
  --quantize \
  --q-bits 4

# Convert to 8-bit (higher quality, more memory)
python -m mlx_lm.convert \
  --hf-path your-org/your-model \
  --mlx-path ./your-model-mlx-8bit \
  --quantize \
  --q-bits 8

# Convert without quantization (bfloat16, full precision)
python -m mlx_lm.convert \
  --hf-path your-org/your-model \
  --mlx-path ./your-model-mlx-bf16

The conversion requires enough RAM to load the full model in bfloat16 (roughly 2x the quantized size). A 7B model needs ~14GB RAM during conversion; a 70B model needs ~140GB. On a Mac with 24GB or less unified memory, you can only convert models up to ~8B parameters. For larger models, either use a Mac with more unified memory or download pre-converted models from mlx-community.

Uploading to Hugging Face for Team Sharing

# Install huggingface_hub
pip install huggingface_hub

# Login to Hugging Face
huggingface-cli login

# Upload your converted model
huggingface-cli upload your-username/your-model-mlx-4bit ./your-model-mlx-4bit

# Or from Python:
from huggingface_hub import upload_folder
upload_folder(
    folder_path="./your-model-mlx-4bit",
    repo_id="your-username/your-model-mlx-4bit",
    repo_type="model"
)

Once uploaded, any team member with access can load the model directly: model, tokenizer = load("your-username/your-model-mlx-4bit"). Set the repository to private for proprietary fine-tuned models.

Finding the Best MLX Community Models

The mlx-community organisation on Hugging Face is the primary hub for MLX-format models. At the time of writing it hosts over 800 models. Navigating to huggingface.co/mlx-community and sorting by most downloads gives a practical ranking of the most useful and tested models. A few reliable patterns for finding what you need. For the latest version of a major model family, search the organisation for the model family name (Llama, Mistral, Qwen, Phi, Gemma). The most recently uploaded and most downloaded versions are usually the best starting points. For models that have just been released, check the original model’s Hugging Face page — the mlx-community conversion typically appears within a few days. For specialised fine-tunes (coding models, multilingual models, instruction fine-tunes), the model card description usually includes a note about the specialisation. If a model you want is not yet available, the mlx-community Discord is an active community where requests often get fulfilled quickly, or you can run the convert command yourself as described above.

Streaming with MLX

MLX-LM’s verbose=True in the generate() call streams tokens to stdout. For application use where you want to process tokens as they arrive (building a chat UI, real-time display, streaming to an API response), use the lower-level generation API:

import mlx.core as mx
from mlx_lm import load
from mlx_lm.utils import generate_step

model, tokenizer = load("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Tell me a short story about a robot."}],
    tokenize=False, add_generation_prompt=True
)
prompt_tokens = tokenizer.encode(prompt)

# Stream tokens one by one
for token, _ in zip(generate_step(mx.array(prompt_tokens), model), range(300)):
    word = tokenizer.decode([token.item()])
    print(word, end="", flush=True)  # stream to stdout
print()  # final newline

The generate_step function is a Python generator that yields one token at a time. You can intercept each token for custom handling — streaming to a WebSocket, building a streaming HTTP response, or detecting stop conditions mid-generation.

Memory Management in Long MLX Sessions

MLX uses a memory cache that grows during a session as models and intermediate computations are allocated. For long-running scripts that load multiple models or process many requests, the cache can grow large. Force a cache clear between heavy operations:

import mlx.core as mx
import gc

# After a heavy generation or after unloading a model:
mx.metal.clear_cache()  # clear Metal GPU memory cache
gc.collect()            # Python garbage collection

# To release a model from memory:
del model
del tokenizer
mx.metal.clear_cache()
gc.collect()

This pattern is important for scripts that load and process multiple large models sequentially — without explicit cache clearing, you can accumulate GPU memory usage that eventually causes out-of-memory errors even when no models are actively in use. Apple’s unified memory manager will swap to disk before crashing (unlike discrete GPU setups that hard OOM), but swap I/O degrades performance dramatically and is a sign your memory management needs attention.

Building a Simple MLX Application

For developers who want to build a persistent application on top of MLX — a CLI tool, a background service, or an integration with another application — the key is loading the model once and reusing it across requests rather than loading on each call. A minimal pattern for a long-running MLX service:

from mlx_lm import load, generate
import threading

class MLXService:
    def __init__(self, model_name: str):
        print(f"Loading {model_name}...")
        self.model, self.tokenizer = load(model_name)
        self.lock = threading.Lock()  # MLX is not thread-safe
        print("Ready.")

    def generate(self, message: str, max_tokens: int = 500) -> str:
        prompt = self.tokenizer.apply_chat_template(
            [{"role": "user", "content": message}],
            tokenize=False, add_generation_prompt=True
        )
        with self.lock:  # ensure single-threaded inference
            return generate(self.model, self.tokenizer,
                           prompt=prompt, max_tokens=max_tokens, verbose=False)

# Use once, reuse many times
service = MLXService("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")
for question in ["What is Python?", "What is Rust?", "What is Go?"]:
    print(service.generate(question))

MLX for Production vs Development

MLX as used in this guide is a development and research tool — loading models from Python scripts, running fine-tuning, building custom inference pipelines. For production deployment of a local LLM on a Mac server, the choice between MLX and Ollama depends on your requirements. Ollama is better suited to production inference: it handles model lifecycle management automatically, provides a stable API with consistent behaviour across updates, supports concurrent requests via OLLAMA_NUM_PARALLEL, and integrates with monitoring tools via standard HTTP. MLX’s mlx_lm.server provides OpenAI-compatible API access but with less operational maturity — it is designed for development use rather than production serving. The practical recommendation: develop and fine-tune with MLX, serve with Ollama (optionally loading fine-tuned models that have been converted to GGUF format for Ollama compatibility). If you need to serve a model that is only available in MLX format or that uses a custom MLX adapter, mlx_lm.server works for low-traffic deployment, but for anything serving multiple concurrent users, Ollama’s inference server is more production-ready.

The MLX Roadmap and Ecosystem Growth

MLX is a rapidly evolving framework. Apple ships updates regularly, adding support for new model architectures, improving Metal kernel performance, and expanding the MLX-LM feature set. The mlx-community on Hugging Face grows with each new model release. The gap between MLX and CUDA in terms of ecosystem breadth is real but narrowing — most tasks that require CUDA-specific features for research are still on CUDA, but all the practical inference and fine-tuning tasks that most users need work well with MLX today. For Apple Silicon Mac users who want to do serious local AI work beyond just running models, MLX is worth the investment in learning. The combination of fast inference via Metal, practical LoRA fine-tuning, Python-level access for custom pipelines, and a growing community model library makes it the most capable AI development environment available for Mac in 2026.

Practical Quick Reference: Key MLX Commands

A summary of the most useful MLX-LM commands for daily use: pip install mlx mlx-lm to install; python -m mlx_lm.generate --model mlx-community/MODEL-4bit --prompt "your prompt" for quick generation; python -m mlx_lm.lora --model MODEL --train --data ./data --iters 500 for fine-tuning; python -m mlx_lm.convert --hf-path ORG/MODEL --mlx-path ./output --quantize --q-bits 4 to convert and quantize a HuggingFace model; python -m mlx_lm.fuse --model MODEL --adapter-path ./adapters --save-path ./fused to merge LoRA weights into base model; and python -m mlx_lm.server --model MODEL --port 8080 to start an OpenAI-compatible API server. These commands cover the full lifecycle of MLX model work from installation through inference, fine-tuning, conversion, and serving. Bookmark this list and the MLX-LM GitHub repository for the complete parameter documentation as the framework evolves.

Contributing to mlx-community

The mlx-community on Hugging Face is community-maintained — anyone can upload converted models. If you have converted a model that is not already available, uploading it benefits other Mac users who want the same model without running the conversion themselves. The process is straightforward: convert with mlx_lm.convert, test that the converted model generates correctly, create a model card (the README.md in the model repository) with the original model source, quantization details, and any known limitations, then upload via the huggingface-cli. The community norm is to name models consistently following the existing pattern (MODEL-NAME-Xbit) and to credit the original model authors in the model card. Active contributors to mlx-community are a significant reason the ecosystem is as useful as it is — the breadth of available models is the result of hundreds of individual conversion and upload contributions. If you find yourself converting a model that others likely want, consider uploading it rather than keeping it local.