Windows is a perfectly capable platform for local LLM inference, with access to the full NVIDIA CUDA ecosystem, a choice between native Windows and WSL2 Linux environments, and dedicated GUI tools that make getting started straightforward. This guide covers the complete Windows local AI setup: CUDA drivers, Ollama on Windows, WSL2 as an alternative, and … Read more
ExLlamaV2 is an inference engine specifically optimised for NVIDIA GPUs running GPTQ-quantized models. If you have an NVIDIA GPU and want the fastest possible local LLM inference, ExLlamaV2 deserves serious consideration alongside llama.cpp. This guide compares the two engines on what actually matters for practical use: speed, model compatibility, ease of use, and ecosystem integration. … Read more
Two of the most practical aspects of llama.cpp that trip up new users: understanding how context length works and how to get the best out of it, and setting up llama-server as a persistent OpenAI-compatible API endpoint. This guide covers both thoroughly — the theory, the commands, and the trade-offs. How Context Length Works in … Read more
GGUF quantization format names — Q4_K_M, Q5_K_S, Q8_0, IQ2_XS — are cryptic at first glance but follow a consistent pattern once you understand the system. Choosing the right quantization is one of the most impactful decisions in local LLM deployment: it determines memory usage, inference speed, and output quality. This guide decodes the naming system, … Read more
llama.cpp and Ollama are not alternatives to each other — Ollama wraps llama.cpp. But developers often face a choice between using Ollama’s API and using llama.cpp directly through its Python bindings. This guide covers the real differences in speed, flexibility, and developer experience, and when the llama-cpp-python library is the better choice over the Ollama … Read more
llama.cpp is the C++ inference engine that powers Ollama, MLX-LM (partially), LM Studio, and most other local LLM tools. Running it directly — rather than through a wrapper — gives you access to the latest models and quantizations before they appear in higher-level tools, full control over compilation flags, and the ability to benchmark raw … Read more
Local AI finds genuinely useful applications in some less-discussed professional and personal contexts. This guide covers three specific use cases — education for children, research assistance for journalists, and note-taking support for therapists — where the privacy, cost, and customisability of local LLMs make them particularly well-suited, and where cloud AI alternatives raise concerns worth … Read more
This guide covers the more advanced aspects of Apple MLX: how to quantize your own models, where to find the best community models, how MLX compares to llama.cpp in real-world performance, and practical Python workflows for building MLX-based applications. If you are new to MLX, the getting started guide earlier in this series covers the … Read more
One of MLX’s most compelling features is the ability to fine-tune LLMs directly on Apple Silicon — something Ollama does not support. LoRA (Low-Rank Adaptation) fine-tuning lets you adapt a pre-trained model to your specific data, domain, or style using a fraction of the memory and compute required for full fine-tuning. On an M4 Pro … Read more
Apple MLX-LM makes it straightforward to run popular open-weights models directly on Apple Silicon Macs — Llama, Mistral, Qwen, Gemma, Phi, and many others. This guide covers the specific setup for the most commonly used models, performance expectations on different Mac configurations, and how to get the best results from each model family. Finding MLX … Read more
