On-device LLM inference means running the full model inference pipeline on the hardware where the application runs — no cloud API, no server, no network dependency. For developers building desktop applications, mobile apps, embedded systems, and edge deployments, understanding the full stack of on-device inference — model formats, runtime engines, hardware acceleration, and integration patterns — is increasingly essential. This guide covers the technical foundation of how on-device LLM inference works and the practical decisions required to make it production-ready.
The On-Device Inference Stack
On-device LLM inference involves four layers that each affect performance, compatibility, and developer experience. The model layer: the neural network weights and architecture, stored in a format like GGUF, SafeTensors, or a framework-specific format. The quantisation layer: the precision at which weights are stored (FP16, INT8, INT4), trading quality for memory and speed. The runtime layer: the inference engine that executes the model computations — llama.cpp, ONNX Runtime, MLX, MLC-LLM, or others. The hardware layer: the physical compute resources — CPU, GPU, NPU/Neural Engine — that the runtime targets. Understanding how these layers interact is the key to optimising on-device inference for a specific deployment target.
Model Formats for On-Device Deployment
Several model formats are used for on-device deployment, each suited to different runtimes and hardware targets:
- GGUF — the standard format for llama.cpp and Ollama. Stores model weights in a single binary file with built-in quantisation support. Supported across CPU and GPU on all major platforms. The most portable format — a GGUF file runs on Windows, macOS, Linux, Android, and iOS through llama.cpp without conversion.
- SafeTensors — Hugging Face’s format for storing model weights, used natively by the Transformers library. The standard format for Python-based inference. Does not embed quantisation — quantisation happens at load time.
- MLX format — Apple’s format for models converted for MLX inference on Apple Silicon. Optimised for the unified memory architecture of M-series chips. Best performance on Mac, iPhone, and iPad.
- LiteRT / TFLite — Google’s format for mobile and edge deployment. Required for MediaPipe LLM Inference. Supported on Android and iOS with hardware acceleration through the Neural Processing Unit.
- ONNX — framework-agnostic format supported by ONNX Runtime. Good for enterprise edge infrastructure where hardware diversity and long-term maintainability matter.
Figure 1 — On-device inference: runtime selection by platform
Quantisation: The Central Performance Lever
Quantisation is the most impactful single decision in on-device inference optimisation. It converts model weights from 32-bit or 16-bit floating point to lower-precision integer representations, reducing memory footprint and increasing inference speed at a cost to output quality. The practical options:
- FP16 (16-bit float) — full precision for on-device standards. Best quality, highest memory use. Use when memory is not a constraint and quality matters most.
- Q8 (8-bit integer) — half the memory of FP16 with minimal quality loss on most tasks. The recommended default for laptop and edge server deployments where quality matters and memory allows it.
- Q4 (4-bit integer) — quarter the memory of FP16. Noticeable quality degradation on complex reasoning; acceptable for classification, extraction, and simple Q&A. The standard choice for mobile deployment.
- Q2 / Q3 — very aggressive quantisation, severe quality loss. Only appropriate for the smallest models where even Q4 exceeds hardware limits.
The relationship between quantisation level and quality is not linear — the jump from FP16 to Q8 loses little quality, the jump from Q8 to Q4 loses more, and Q4 to Q2 loses substantially more. Start with Q8 and only move to Q4 when memory constraints require it.
KV Cache and Memory Management
The KV (key-value) cache stores intermediate attention computations for the context processed so far, enabling efficient autoregressive generation without recomputing previous tokens. Its memory footprint grows linearly with context length — for a 1.7B model with a long conversation, the KV cache can consume more memory than the model weights themselves. On memory-constrained devices, managing KV cache size is essential: set a maximum context length appropriate for the device, implement sliding window attention to limit cache growth in long conversations, or use quantised KV cache (Q8 or Q4) to reduce its memory footprint. Most production on-device inference frameworks support KV cache quantisation as a configuration option; enabling it is one of the most effective memory optimisations that does not significantly affect output quality.
Hardware Acceleration Paths
Each hardware platform has specific acceleration capabilities that inference runtimes exploit. Understanding which acceleration path your framework uses on your target hardware explains the performance you observe and guides optimisation decisions. On Apple Silicon, the Neural Engine (ANE) handles matrix operations efficiently and with low power; MLX and Core ML both target the ANE. On NVIDIA GPUs, CUDA enables massively parallel matrix computations — llama.cpp with CUDA, ONNX Runtime with CUDA or TensorRT, and Hugging Face Transformers with CUDA all exploit this. On AMD GPUs, ROCm provides similar capability to CUDA but with less mature tooling. On Qualcomm SoCs (Snapdragon), the Hexagon DSP and Adreno GPU both handle inference; MLC-LLM targets the Adreno GPU while MediaPipe targets the Hexagon NPU. On Intel CPUs, OpenVINO and ONNX Runtime with OpenVINO execution provider apply Intel-specific optimisations. Knowing which acceleration path your framework targets explains why the same model may perform very differently on similar hardware from different manufacturers.
Prefill vs Decode: Two Inference Phases
LLM inference has two distinct phases with different performance characteristics. Prefill processes the full input prompt in a single parallel operation — this is fast because it is highly parallelisable and GPU/NPU hardware excels at it. Decode generates output tokens one at a time, autoregressively — this is memory-bandwidth-bound rather than compute-bound, meaning faster memory bandwidth directly improves decode speed more than raw compute power does. This is why Apple Silicon performs so well for on-device inference despite lower raw FLOPS than comparable NVIDIA GPUs: its unified memory architecture provides extremely high bandwidth between memory and compute, which is the bottleneck for the decode phase that dominates inference time during generation.
Speculative Decoding
Speculative decoding is an advanced technique that uses a small draft model to generate candidate tokens in parallel, which the larger target model then verifies in a single forward pass. This can increase effective decode speed by 2–3x on hardware where the larger model’s decode phase is the bottleneck. Practical on-device use: pair Qwen2.5-0.5B as the draft model with Qwen2.5-3B as the verifier model — both fit in memory together on a modern smartphone, and the 0.5B model generates candidates fast enough that the verification step (shared with 3B inference) produces meaningful speedup on the most capable hardware tier. Speculative decoding is implemented in llama.cpp and is increasingly supported in mobile-focused frameworks; it is worth experimenting with on high-end mobile devices where generation speed is the primary constraint.
Benchmarking Your Specific Setup
Published inference benchmarks rarely match real-world performance in your specific setup because they measure different models, different quantisation levels, different context lengths, and different hardware than your target. Before deploying, measure inference performance in your actual application context: the actual model and quantisation you will use, the actual prompt lengths your application generates, the actual hardware you are targeting, and under the actual thermal conditions of sustained use (which triggers throttling on mobile devices). A simple benchmarking script that runs 100 representative prompts through your inference stack and measures tokens per second, peak memory use, and first-token latency gives you the numbers that actually predict production behaviour.
On-device LLM inference in 2026 is a mature engineering discipline with well-understood tradeoffs and production-ready tooling across every major platform. The decisions that matter most — model size, quantisation level, runtime framework, and hardware acceleration path — are all consequential and interdependent. Getting them right for your specific deployment target requires understanding the full stack from model format through hardware acceleration, and validating the choices with real measurements on real hardware rather than relying on benchmarks from different setups.
Latency Budgets and User Experience Design
On-device LLM inference has different latency characteristics from cloud APIs, and designing user interfaces that work well within those characteristics is as important as the technical optimisation. First-token latency — the time between submitting a prompt and seeing the first token of output — is typically 0.5–2 seconds for a well-optimised setup on capable hardware. This is perceptible but acceptable if the UI clearly signals that processing is happening (a typing indicator, an animated spinner). Streaming tokens from the first token onwards makes the response feel fast even if total generation takes 5–10 seconds — users reading streaming output are engaged rather than waiting. For longer responses, consider breaking generation into sentences with UI updates between them, which allows the user to start reading before generation is complete. These UX patterns are standard in cloud AI applications and apply equally to on-device deployment; the difference is that generation speed is less predictable across the device range you support, so designing for the lower end of your supported hardware range ensures a good experience across all users.
Combining On-Device and Cloud Inference
A hybrid architecture — handling routine queries on-device and routing complex queries to a cloud API — captures the benefits of both approaches. Simple, repetitive queries (greetings, basic classifications, short summarisations) are handled on-device with sub-second latency and no API cost. Complex queries requiring deep reasoning, long context, or broad knowledge are routed to a cloud API when the device is online. The routing logic can be as simple as a query complexity estimate based on length and keyword patterns, or as sophisticated as a small classifier model that evaluates whether the on-device model is likely to handle the query well. This hybrid approach is particularly well-suited to consumer applications that need to work offline but deliver the best possible quality when connected — the on-device model handles the offline experience while the cloud model handles the demanding queries when connectivity is available.