How to Fine-Tune LLMs with MLX on Mac: LoRA Guide

by

One of MLX’s most compelling features is the ability to fine-tune LLMs directly on Apple Silicon — something Ollama does not support. LoRA (Low-Rank Adaptation) fine-tuning lets you adapt a pre-trained model to your specific data, domain, or style using a fraction of the memory and compute required for full fine-tuning. On an M4 Pro or M4 Max Mac, a LoRA training run that would take hours on CPU completes in 15-60 minutes. This guide covers the complete MLX fine-tuning workflow from data preparation to testing your adapted model.

What Fine-Tuning Does and When to Use It

Fine-tuning teaches a model to behave differently than its base training suggests. You are not changing the model’s fundamental capabilities — you are adjusting its default tendencies, terminology usage, response style, and domain knowledge emphasis. The result is a model that more reliably produces the kind of outputs you want for your specific use case without needing long, explicit prompts every time.

Good use cases for fine-tuning: adapting a model to your company’s writing style and terminology so it produces on-brand content without prompting; teaching a model to follow a specific output format consistently (e.g., always returning structured JSON with specific fields); improving performance on a narrow domain where your training data is high quality (medical terminology, legal language, technical documentation for your specific product); creating a specialised assistant that stays focused on its task rather than wandering into unrelated territory. Fine-tuning is not the right tool for: teaching the model new facts it does not know (use RAG for this); fixing fundamental capability gaps (use a larger model); getting better general reasoning (use prompt engineering first — it is faster and usually sufficient).

LoRA vs Full Fine-Tuning

Full fine-tuning updates all model weights — practical only on server hardware with massive GPU clusters. LoRA (Low-Rank Adaptation) adds small trainable matrices to the attention layers and trains only those, keeping the base weights frozen. The trainable parameters are typically 0.1-1% of the total model parameters, making training feasible on consumer hardware. The LoRA adapters are saved as small files (50-200MB) that you load alongside the base model at inference time. QLoRA (quantized LoRA) is LoRA applied to a quantized base model, further reducing memory requirements. MLX-LM supports both LoRA and QLoRA through the same interface.

Preparing Training Data

MLX-LM accepts training data in JSONL format (one JSON object per line). Two formats are supported:

Completion format (for teaching the model to continue text in a specific style):

{"text": "Write a product description for wireless earbuds.

Sonance Pro Wireless earbuds deliver 32 hours of playtime with ANC that adapts to your environment in real time..."}
{"text": "Write a product description for a standing desk.

The ErgoRise Pro transforms your workspace with one-touch height memory and a whisper-quiet motor..."}

Instruction format (for teaching specific instruction-following behaviour):

{"messages": [{"role": "user", "content": "Classify this email as urgent or routine: Meeting rescheduled to 3pm today."}, {"role": "assistant", "content": "urgent"}]}
{"messages": [{"role": "user", "content": "Classify this email as urgent or routine: Monthly newsletter attached."}, {"role": "assistant", "content": "routine"}]}

Organise your data into train.jsonl, valid.jsonl (validation set, ~10% of data), and optionally test.jsonl. A minimum viable dataset for style fine-tuning is 50-100 examples; for reliable results on specific tasks, 200-500 examples is better. Quality matters more than quantity — 100 well-crafted examples outperforms 1000 noisy or inconsistent ones.

mkdir training_data
# Create train.jsonl and valid.jsonl with your examples
# Verify the files are valid JSONL:
python -c "import json; [json.loads(l) for l in open('training_data/train.jsonl')]"
echo "Valid JSONL"

Running LoRA Fine-Tuning

# Basic fine-tuning run
python -m mlx_lm.lora \
  --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
  --train \
  --data ./training_data \
  --iters 500 \
  --batch-size 4 \
  --lora-layers 8 \
  --learning-rate 1e-4 \
  --save-every 100 \
  --adapter-path ./adapters

# For smaller Macs (limited memory), reduce batch size:
# --batch-size 1 --grad-checkpoint
# For more aggressive adaptation:
# --lora-layers 16 --iters 1000

Key parameters: --iters is the number of training steps (not epochs). --batch-size is examples processed per step — reduce if you get memory errors. --lora-layers is how many attention layers get LoRA adapters (higher = more adaptation capacity, more memory). --save-every saves checkpoint adapters periodically so you can test intermediate results. Watch the validation loss in the training output — it should decrease initially and then plateau. If it starts increasing (overfitting), stop training.

Figure 1 — MLX Fine-Tuning: Time and Memory by Mac Config

ConfigModelMemory used500 iters timeFits?M4 Pro 16GBLlama 3.2 3B 4-bit~6 GB~10 minYesM4 Pro 24GBLlama 3.1 8B 4-bit~12 GB~20 minYesM4 Max 64GBLlama 3.1 8B 4-bit~12 GB~12 minYesM4 Max 128GBLlama 3.1 70B 4-bit~55 GB~90 minYes (slow)

Testing Your Fine-Tuned Adapter

# Test with the saved adapter
python -m mlx_lm.lora \
  --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
  --adapter-path ./adapters \
  --prompt "Write a product description for noise-cancelling headphones."
from mlx_lm import load, generate
from mlx_lm.utils import load as load_with_adapter

# Load model with adapter
model, tokenizer = load(
    "mlx-community/Meta-Llama-3.1-8B-Instruct-4bit",
    adapter_path="./adapters"
)

# Generate with the fine-tuned model
response = generate(
    model, tokenizer,
    prompt=tokenizer.apply_chat_template(
        [{"role": "user", "content": "Write a product description for wireless earbuds."}],
        tokenize=False, add_generation_prompt=True
    ),
    max_tokens=300
)
print(response)

Compare the fine-tuned model’s output to the base model on the same prompt. The difference should be noticeable after even 200-500 training iterations if your training data has a consistent pattern. If you do not see a difference, your training data may not have a strong enough signal, or you may need more iterations — increase to 1000-2000 and watch the validation loss.

Fusing Adapters into the Base Model

For production use, you can fuse the LoRA adapters permanently into the model weights rather than loading them separately on each run. This produces a standalone fine-tuned model:

python -m mlx_lm.fuse \
  --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
  --adapter-path ./adapters \
  --save-path ./my-finetuned-model

The fused model can be loaded with the standard load() function without specifying an adapter path. Fused models can also be pushed to Hugging Face for sharing with teams or for use in other tools that support MLX format, such as mlx_lm.server.

Common Training Problems and Fixes

A few issues come up regularly in MLX fine-tuning. Memory errors during training: reduce –batch-size to 1 and add –grad-checkpoint (gradient checkpointing reduces memory at the cost of slightly slower training). If still failing, use a smaller model or quantize more aggressively. Loss not decreasing: check your training data format — malformed JSONL or inconsistent formatting prevents the model from learning the pattern. Print a few examples from your dataset to verify they look correct. Increase –learning-rate slightly (try 2e-4 or 5e-4). Overfitting (training loss decreasing but validation loss increasing): reduce training iterations, add more training data, reduce –lora-layers, or add data augmentation. Fine-tuned model losing general capability: a sign of too much adaptation — reduce iterations and lora-layers. LoRA fine-tuning is designed to preserve general capability while adjusting specific behaviours; heavy adaptation can cause catastrophic forgetting of base model knowledge. Output format ignored: your training data may not have enough examples showing the format you want, or the formatting is inconsistent across examples. Audit a random sample of training examples for consistency.

A Practical Fine-Tuning Workflow

The most effective fine-tuning workflow for a new use case: start by curating 50 high-quality, consistent training examples and train for 200 iterations. Test the adapter on 10 held-out prompts not in the training set and assess quality. If the improvement is in the right direction but not strong enough, add more examples (target 200 total) and train for 500 iterations. Repeat — evaluate, add data, increase iterations — until quality is sufficient or plateaus. Do not chase further improvement by training longer once the validation loss has stabilised; at that point, more data is the only lever that helps. Save checkpoints at each stage (–save-every handles this) so you can revert to an earlier checkpoint if the model starts degrading. The full cycle from zero to a useful fine-tuned adapter typically takes one to three days of iteration time, most of which is data curation rather than training compute. The training itself runs fast on Apple Silicon — getting the data right is the hard part.

Sharing and Deploying Fine-Tuned Models

Once you have a fine-tuned adapter that works well, there are several ways to use it beyond your local development machine. The fused model (base weights merged with adapter) can be uploaded to a private Hugging Face repository for team access, loaded via mlx_lm.server as an API endpoint accessible to other team members on your local network, or converted back to GGUF format for use with Ollama using the gguf conversion tools from the llama.cpp project. The adapter files alone (without base weights) are small enough to share via email or Slack — anyone with the same base model can apply your adapter to get the same fine-tuned behaviour. This lightweight sharing pattern is one of LoRA’s key practical advantages: the base model stays local on each machine and only the small adapter diff is distributed. For internal team tools where you want consistent AI behaviour across multiple people, this adapter-sharing pattern provides a straightforward way to distribute your fine-tuning work without moving large model files.

When Fine-Tuning Beats Prompt Engineering

The honest answer to “when should I fine-tune versus just engineering better prompts?” is: try prompt engineering first, always. A well-crafted system prompt with examples (few-shot prompting) solves most consistency problems without any training. Fine-tuning is worth the investment when you have tried good prompting and still need better results, when you are running the model at high volume and cannot afford long system prompts per request (fine-tuned models need shorter prompts to achieve the same behaviour), when the style or format you need is genuinely difficult to describe in a prompt, or when inference latency is critical and you want the model to follow a pattern without the overhead of long context. Fine-tuning adds maintenance overhead — you now have a model artifact to track, version, and update as the base model updates — so reserve it for cases where the prompt engineering ceiling has genuinely been reached for your specific use case.

Leave a Comment