MediaPipe’s LLM Inference API is Google’s solution for running language models directly on Android and iOS devices. It takes a small language model — Gemma 2B, Phi-2, Falcon-RW-1B, or a custom model you convert — and runs inference entirely on-device using the hardware accelerators available on the phone: GPU via WebGL or OpenCL, and CPU as a fallback. No server, no API key, no network required once the model is bundled. This guide covers the complete setup, supported models, performance expectations, and what Google’s recent shift to LiteRT means for new projects.
Important: MediaPipe LLM API Status in 2026
As of mid-2026, Google has placed the MediaPipe LLM Inference API in maintenance-only mode. New feature development is happening in LiteRT (formerly TensorFlow Lite), which is Google’s current recommended path for on-device ML including LLM inference. Models converted with LiteRT Torch are compatible with the LLM Inference API for CPU inference on Android and iOS, so existing implementations continue to work. For new projects starting today, evaluate both MediaPipe LLM Inference (simpler API, broader model support) and LiteRT (Google’s current investment, better long-term support). This guide covers the MediaPipe path because it remains the most widely documented and deployed approach for on-device LLMs on Android, but awareness of the LiteRT transition is important for production planning.
Supported Models
The MediaPipe LLM Inference API supports several pre-converted model formats out of the box. Gemma 3 1B is the primary recommended model — it is available directly from HuggingFace in the .task/.litertlm format ready for the API, requires approximately 1.8GB of storage, and runs on mid-range Android devices from 2022 onwards. Gemma 2B in the .task format is also supported. For custom models, MediaPipe provides a bundler tool that converts models from safetensors or GGUF format into the .task bundle format the API expects. The bundler handles weight conversion, tokeniser packaging, and metadata required for the API. Phi-2 and Falcon-RW-1B are among the models the community has successfully converted, though official support varies.
Figure 1 — MediaPipe LLM Inference API: supported backends
Android Setup
Add the MediaPipe Tasks GenAI dependency to your Android project’s build.gradle. Download a supported model — Gemma 3 1B from HuggingFace is the recommended starting point — and place it in your app’s assets folder or on device storage. In your Android code, create an LlmInference object by specifying the model path and backend preference (GPU is the default). To generate text, call generateResponse() with a prompt string. For streaming output that displays tokens as they generate (which dramatically improves perceived responsiveness), use generateResponseAsync() with a callback that receives each token as it is produced.
The model file is large — Gemma 3 1B is approximately 1.8GB — which means distributing it as part of the APK is not practical. Instead, download the model file on first app launch from your server or from a URL the user provides, cache it on device storage, and check for its existence before initialising the inference engine. Google Play Asset Delivery is the recommended channel for shipping large AI model files to Android users through the Play Store, as it handles delivery optimisation and caching automatically.
iOS Setup
The MediaPipe LLM Inference API for iOS follows the same pattern as Android but uses Swift or Objective-C. Add the MediaPipeTasksGenAI pod to your Podfile, download the model file, and initialise LlmInference with the local file path. The API surface is identical to Android — generateResponse() for blocking generation, generateResponseAsync() for streaming. On iOS, the GPU backend uses Metal for hardware acceleration. Memory constraints are stricter on iOS than Android due to iOS’s tighter memory limit enforcement — test on the lowest-specification device you intend to support before committing to a model size.
Performance Expectations
Performance varies significantly by device. On a 2023 flagship Android phone (Snapdragon 8 Gen 2), Gemma 3 1B generates at approximately 10–20 tokens per second with GPU acceleration enabled — fast enough for interactive use but noticeably slower than desktop inference. On mid-range 2022 devices, expect 5–10 tokens per second. On budget devices with limited GPU compute, the API falls back to CPU inference which may drop to 1–3 tokens per second — slow enough to require streaming output with a visible typing indicator to remain usable. For iOS on iPhone 15 Pro with Metal, expect 15–25 tokens per second for Gemma 3 1B. Always measure on your target device range rather than relying on benchmarks from a single flagship.
Converting Custom Models
To use a model not in the official supported list, MediaPipe provides a Python bundler script. Install the mediapipe package, prepare your model in safetensors format, and run the bundler with your model directory and output path as arguments. The bundler packages weights, tokeniser, and model metadata into a .task file. Not all model architectures are supported — the bundler works with Gemma, Phi, and Falcon variants but may fail on other architectures. Check the official MediaPipe documentation for the current supported model list before attempting conversion, as support has expanded with each MediaPipe release.
Alternatives to MediaPipe for Mobile LLM
MediaPipe is not the only option for on-device LLM inference on mobile. MLC-LLM from the Machine Learning Compilation project supports a wider range of models and architectures, works on both Android and iOS, and is more actively developed for cutting-edge model support. ExecuTorch from Meta provides a path for running PyTorch models on mobile, well-integrated with Llama models. LiteRT (Google’s successor to TensorFlow Lite) is Google’s current investment for mobile ML. For teams starting new projects in 2026, evaluating MLC-LLM and LiteRT alongside MediaPipe is worthwhile — MediaPipe’s maintenance-only status means MLC-LLM in particular may be a better long-term foundation for production mobile AI applications.
Frequently Asked Questions
- Is the MediaPipe LLM Inference API free to use?
- Yes. MediaPipe is an open-source project from Google. The LLM Inference API is free to use with no usage fees. The models themselves (Gemma) are also available under permissive licences for most uses.
- Can MediaPipe LLM Inference run without internet?
- Yes, once the model is downloaded and cached on device. All inference happens locally — no network calls are made during generation. This is the primary advantage over cloud API approaches.
- What is the minimum Android version required?
- Android 10 (API level 29) or higher for GPU backend. CPU inference works on earlier versions. The Gemma 3 1B model specifically requires at least 4GB RAM on the device for reliable operation.
Last updated: October 2026
Handling Model Updates in Production Apps
One underappreciated challenge of shipping a model in a mobile app is updating it. Unlike cloud APIs where you swap the model server-side, a local model lives on the user’s device and requires an explicit update mechanism. For MediaPipe models stored on device, implement a version check on app launch: compare a stored model version number against your server’s current version, and trigger a background download if they differ. Show the user a progress indicator during the download and continue serving the old model until the new one is fully downloaded and verified. Store model files in the app’s documents directory or application support directory (not the cache, which the OS may purge). For critical model improvements, prompt the user explicitly; for routine updates, download silently in the background when on Wi-Fi. This update lifecycle is often more engineering work than the initial inference integration, and planning for it from the start avoids painful refactoring later.
Debugging Common MediaPipe LLM Issues
The most common issues when getting started: model file not found — ensure the path you pass to LlmInference is absolute and the file actually exists at runtime, not just at build time; GPU initialisation failure — some older Android devices have limited OpenCL support, set the backend to CPU as a fallback when GPU initialisation throws; out of memory crash — the model requires more RAM than the device provides, switch to a smaller model or a more aggressively quantised variant; slow first inference — the first generateResponse() call includes model initialisation overhead, which can take 2–5 seconds; subsequent calls are faster. Wrap all LlmInference initialisation and inference calls in try-catch blocks and log errors to your crash reporting tool to diagnose device-specific issues in production.
Designing the Streaming Experience
Keep the inference engine as a single long-lived object instead of recreating it for every request, since loading the model is the most expensive step. Stream tokens to the screen with a typing indicator so the interface feels responsive even on slower phones. Cancel generation when the user leaves the screen, and release the model when the system signals low memory. Handle the first-run download with a clear progress bar and the option to resume, because a multi-gigabyte model file will not always finish in one go.