Running a language model directly on a smartphone — no server, no API, no internet connection required — was a niche experiment in 2023 and is a practical deployment option in 2026. The hardware caught up: modern high-end phones have 8–12GB of RAM, powerful NPUs, and the inference frameworks to use them efficiently. This guide covers the tools, models, and integration patterns for shipping an LLM inside an iOS or Android app, including what works today, what the realistic performance looks like, and what the user experience tradeoffs are.
Why Run On-Device
The case for running an LLM on the device rather than calling a cloud API is strongest in four scenarios. Privacy: user conversations, documents, and queries never leave the device — a genuine differentiator for health apps, journaling apps, personal assistants, and any application where users are sensitive about data leaving their phone. Offline capability: the model works without an internet connection, which matters for travel apps, field tools, and any application used in areas with unreliable connectivity. Latency: eliminating the round-trip to a server and back reduces perceived response time, especially for short queries where the round-trip dominates. Cost: no per-query API charges at scale, which can be meaningful for high-engagement consumer apps. The tradeoffs are model quality (on-device models are smaller and less capable than frontier cloud models), battery consumption, and app bundle size (models add 0.5–4GB to the app download).
iOS: The MLX and Core ML Path
Apple Silicon iPhones (A14 and later) and iPads (M1 and later) are the strongest mobile platforms for on-device LLM inference due to their unified memory architecture and Neural Engine. Two frameworks are well-suited for production iOS deployment.
MLX, Apple’s array computation framework built for Apple Silicon, delivers the best performance on iPhones and iPads with Apple chips. The mlx-lm Swift package wraps MLX for iOS development, enabling Hugging Face models converted to MLX format to run directly in an iOS app. SmolLM2-1.7B and Qwen2.5-1.5B both have community-maintained MLX conversions available on Hugging Face. On an iPhone 15 Pro, these models generate at 15–25 tokens per second — fast enough for real-time interactive use. The MLX path requires iOS 16+ and an A14+ chip, excluding older devices.
Core ML is Apple’s on-device ML framework with broader device compatibility, supporting older iPhones through the ANE (Apple Neural Engine). Converting a model to Core ML requires the coremltools Python package and produces a .mlpackage file bundled inside the app. Core ML inference is optimised for the ANE, which is highly power-efficient — an important consideration for extended mobile use. The tradeoff is that Core ML conversion is more involved than MLX and supports a narrower range of model architectures. Google’s Gemma models have official Core ML support, making them the most straightforward choice for the Core ML path.
Figure 1 — iOS vs Android LLM deployment: framework comparison
Android: MLC-LLM and MediaPipe
Android deployment is more fragmented than iOS because of hardware diversity — flagship Snapdragon devices, mid-range MediaTek chips, and older GPUs all need to be handled. Two frameworks cover most Android deployment scenarios.
MLC-LLM (Machine Learning Compilation for LLMs) compiles models for specific Android hardware using TVM (Tensor Virtual Machine), producing highly optimised inference code for Qualcomm Adreno and ARM Mali GPUs. The result is the best performance available on Android for supported model architectures — SmolLM2-1.7B runs at 15–20 tokens per second on a Snapdragon 8 Gen 2 device with MLC-LLM, which is genuinely fast interactive performance. The tradeoff is setup complexity: MLC-LLM requires compiling models for each target device architecture, which is more involved than other frameworks. The MLC-LLM team maintains pre-compiled packages for popular models and common Snapdragon architectures, reducing the compilation burden for these combinations.
Google’s MediaPipe LLM Inference API is the simplest path to Android LLM deployment and the recommended starting point for most developers. It supports Gemma models natively with minimal configuration — download the model weights in LiteRT format, initialise the inference engine in a few lines of Kotlin or Java, and call generate() with a prompt. On a mid-range Android phone it runs SmolLM2-inspired models at 8–12 tokens per second, sufficient for interactive use if the interface streams tokens as they are generated rather than waiting for the full response.
Model Selection for Mobile
The model choice for mobile deployment follows directly from device capabilities. For iOS: SmolLM2-1.7B (MLX format) for iPhone 12–14 with 4–6GB RAM; Qwen2.5-3B (MLX format) for iPhone 15 Pro and iPad Pro with 6–8GB available RAM. For Android: Gemma-2-2B (LiteRT format) for MediaPipe deployment on mid-range devices; SmolLM2-1.7B (MLC compiled) for flagship Snapdragon devices where performance matters most. In all cases, use Q4 quantisation to minimise model size and memory use — the quality difference from Q8 is small on tasks appropriate for mobile deployment (conversational assistance, document Q&A, summarisation) and the memory savings are significant.
App Bundle Size Considerations
The model adds substantially to app download size. SmolLM2-1.7B at Q4 is approximately 1.1GB; Qwen2.5-3B at Q4 is approximately 2GB. This is large for a mobile app but manageable with the right download strategy. Rather than bundling the model inside the app package (which would make the initial download enormous), download it on first launch over Wi-Fi with a clear user explanation — “this app uses a local AI model (1.1GB download) that keeps your data private.” Store the model in the app’s documents directory. On subsequent launches, the model is already present. This on-demand download approach is well-established in gaming apps and is increasingly expected for AI-enabled apps. Both iOS App Store and Google Play support apps that download large assets post-install.
Streaming Tokens for Perceived Speed
Regardless of the actual tokens-per-second rate, streaming generated tokens to the UI as they arrive dramatically improves perceived responsiveness. A model generating at 12 tokens per second that streams output feels much faster than one that waits for the full response before displaying anything — the user sees words appearing within milliseconds of tapping send rather than waiting 5–10 seconds for a complete response. Both MLX and MLC-LLM support async token streaming; implement it from the start rather than as an afterthought. A typing indicator shown during inference (alongside streaming tokens) further signals that the model is working, preventing the user from thinking the app has frozen during generation.
Testing Across Device Tiers
Always test on the oldest and lowest-spec device you intend to support, not just the flagship. A model that runs smoothly at 20 tokens per second on an iPhone 15 Pro may run at 6 tokens per second on an iPhone 12 mini and exceed memory limits on an older device entirely. Establish a minimum supported device specification before committing to a model — the oldest device whose RAM and NPU can handle the chosen model at acceptable performance — and communicate that requirement clearly in the App Store listing. For Android, testing on multiple chipset families (Snapdragon, MediaTek, Exynos) is important because performance varies substantially, and a model or framework that is well-optimised for Snapdragon may underperform on MediaTek devices.
On-device LLM deployment on mobile in 2026 is ready for production apps with the right scope and expectations. The models are capable enough for a meaningful range of use cases, the frameworks are mature enough for shipping, and the privacy and offline arguments are compelling for the right applications. The key constraints — model quality below frontier, battery consumption, bundle size, and device compatibility — are manageable engineering problems rather than fundamental blockers. For applications where user trust and data privacy are competitive advantages, shipping an on-device model is increasingly worth the engineering investment.
Handling Updates and Model Versioning
Once deployed, the on-device model becomes a dependency that needs maintenance. When a better version of your chosen model is released — which happens frequently in the rapidly evolving small model space — updating the on-device model requires downloading a new model file, which involves a significant user-facing event: a multi-gigabyte download notification. Design the update flow thoughtfully. Background updates on Wi-Fi that are ready the next time the app launches are less disruptive than blocking the user with a mandatory update prompt. Keep the previous model version available until the new one is fully downloaded and verified, so a failed download does not leave the user without a working model. Version the model file name in local storage so that incompatible model updates (different tokeniser, different format) are handled cleanly without corrupting the existing model. These operational concerns are straightforward to handle when designed in from the start and surprisingly painful to retrofit after deployment.
Privacy Architecture: What Stays On Device
Running the model on-device does not automatically guarantee privacy — it depends on what your app does with the model inputs and outputs. A genuinely private on-device LLM implementation never sends prompts, conversations, or model outputs to any remote server. If your app logs analytics, crash reports, or any user activity data, ensure these logs do not include any LLM conversation content. If the app uses a remote service for any feature alongside the on-device model, be explicit in your privacy policy about what does and does not leave the device. Users choosing an on-device AI app specifically for privacy reasons will scrutinise this carefully, and the trust benefit of on-device processing is only realised if the privacy architecture is genuinely end-to-end local.