WebLLM is an open-source project from the MLC (Machine Learning Compilation) team at CMU that runs large language models directly inside the web browser using WebGPU. No server, no API key, no backend — the model weights download to the user’s machine and inference runs entirely in the browser tab using the GPU through the WebGPU API. For web developers who want to add AI features to a web application without running any backend infrastructure, WebLLM is the most capable option available in 2026.
How WebLLM Works
WebLLM uses three browser technologies working together. WebGPU is the modern GPU API that replaced WebGL for compute workloads — it gives JavaScript near-native access to the GPU for general-purpose computation, which is what LLM matrix operations require. WebAssembly handles CPU-side operations that are too complex for JavaScript. The Service Worker API caches downloaded model weights between sessions so the user only downloads the multi-gigabyte model file once. The MLC team pre-compiles popular models (Llama, Phi, Gemma, Qwen, SmolLM) using their TVM-based compilation pipeline into a format WebLLM can execute efficiently through WebGPU.
The resulting experience: a user visits your web app for the first time and is prompted to download the model (shown as a progress bar). After the download completes and on all subsequent visits, the AI features work instantly without any network requests during inference. The model runs at 10–30 tokens per second on a modern GPU (discrete or integrated), which is fast enough for real-time streaming output in the browser.
WebGPU Browser Support
WebGPU is not universally supported, which is the most important practical constraint for WebLLM. As of late 2026, WebGPU is fully supported in Chrome 113+ and Edge 113+ on Windows, macOS, and Linux. Firefox has support behind a flag but not enabled by default in stable releases. Safari added WebGPU support in Safari 18 (iOS 18, macOS Sequoia), covering recent iPhones, iPads, and Macs. Android Chrome supports WebGPU on recent devices with Vulkan-capable GPUs. The practical implication: WebLLM works for the majority of users on modern devices using Chrome, but you need a fallback strategy for Firefox users and older devices. Check webgpu.io/implementation-status for the current state of browser support, which continues to improve.
Figure 1 — WebLLM architecture: browser-native inference
Getting Started with WebLLM
Add WebLLM to a web project via npm or a CDN script tag. The API is designed to be familiar to anyone who has used the OpenAI JavaScript SDK — it exposes a chat completions interface compatible with the OpenAI API format, making it easy to swap between cloud and browser inference. The core workflow: create an MLCEngine instance, call a prebuilt model by ID, wait for it to initialise (which downloads and caches the model on first use), then call chat.completions.create() with your messages in the standard format.
WebLLM supports streaming through async generators, which is essential for good UX — streaming tokens to the UI as they are generated feels responsive even at 15 tokens per second, whereas waiting for the full response before displaying it feels slow. The streaming API follows the same pattern as streaming in the OpenAI SDK, making it straightforward to implement a smooth typing effect for generated responses.
Available Models and Sizes
WebLLM maintains a registry of pre-compiled models available through its CDN. As of 2026, the registry includes variants of Llama 3.2, Phi-4-mini, Gemma 2, Qwen 2.5, SmolLM2, Mistral, and others across quantisation levels from Q4 to FP16. The practical size choices for web deployment:
- SmolLM2-1.7B-Instruct-q4f16_1 — around 1GB download, fastest inference, suitable for mobile browsers and lower-end GPUs. The best starting point for applications requiring broad device support.
- Phi-4-mini-instruct-q4f16_1 — around 2GB download, noticeably stronger reasoning than SmolLM2 at similar inference speed on capable hardware. The best quality-to-size option for desktop Chrome.
- Llama-3.2-3B-Instruct-q4f16_1 — around 2GB download, strong general capability, widely tested with WebLLM. Good choice for applications where model familiarity and community support matter.
Avoid larger models (7B+) for production web deployment — download sizes become prohibitive for typical users and inference speed on integrated GPUs is too slow for interactive use.
Handling the Download and Initialisation
The first-run model download is the biggest UX challenge in WebLLM applications. A 1–2GB download takes 1–5 minutes on a typical broadband connection, which is too long to ask users to wait silently. Design the download experience carefully: show a prominent progress bar with bytes downloaded and estimated time remaining, explain what is being downloaded and why (“downloading the AI model — this only happens once”), and allow users to continue using other parts of the app during the download if possible. Once the model is cached by the Service Worker, subsequent loads take 1–3 seconds for initialisation — fast enough to feel immediate. For applications where the download experience is critical, offering a cloud API fallback while the local model downloads gives users immediate access while the local model prepares.
Use Cases Well-Suited to WebLLM
WebLLM is particularly well-suited to use cases where browser-native inference offers clear advantages over a backend API. Privacy-sensitive document tools: a contract analyser, a journal assistant, a medical symptom checker where users are uncomfortable sending content to a server. Offline-capable web apps: Progressive Web Apps that need to work without internet connectivity. Developer tools embedded in IDEs or code editors that run in browser contexts (VS Code’s web version, CodeSandbox, StackBlitz). Interactive demos and educational tools where deploying backend infrastructure would add complexity without benefit. Games and creative tools where AI generation is a feature that should work without any server dependency.
Performance Expectations by Hardware
WebLLM performance varies substantially by GPU. On a recent discrete GPU (NVIDIA RTX 3060+, AMD RX 6600+), SmolLM2-1.7B generates at 25–40 tokens per second — genuinely fast interactive performance. On integrated GPUs (Intel Iris Xe, AMD integrated in Ryzen), expect 8–15 tokens per second — usable for interactive chat with streaming. On older integrated graphics or devices without WebGPU support, WebLLM falls back to CPU via WebAssembly at 2–5 tokens per second — slow but functional for short responses. Testing your target use case on representative hardware is important before committing to WebLLM for production — the experience on a developer’s RTX 4090 will not be representative of the experience for users on integrated graphics.
OpenAI-Compatible API
WebLLM’s OpenAI-compatible API is one of its most useful features for developers already using the OpenAI SDK. The same message format, the same streaming pattern, and the same function calling interface work with WebLLM as with the OpenAI API. This means a web app built with the OpenAI SDK can be adapted to use WebLLM by swapping the API client initialisation — the rest of the application code remains unchanged. For teams building web apps that use the OpenAI API but want to offer an optional privacy-preserving local mode, WebLLM’s API compatibility makes that option low-engineering-effort to add once the basic integration is working.
WebLLM represents a genuine capability milestone: production-quality LLM inference running entirely in the browser with no server infrastructure required. Its WebGPU dependency limits device support but that support continues to expand rapidly. For web applications where privacy, offline capability, or infrastructure simplicity are priorities, WebLLM is increasingly the right architectural choice rather than an experimental alternative.
WebLLM for Function Calling and Structured Output
WebLLM supports function calling and structured JSON output for models that were trained with these capabilities — Llama 3.2 and Phi-4-mini both support tool calling through WebLLM’s API. This enables browser-based agent workflows where the model can call JavaScript functions defined in the browser: search local IndexedDB, read from browser storage, call a web API, update the DOM. A browser-native agent that uses the local filesystem API, reads local documents, and calls web services entirely client-side is a genuinely new architectural pattern that WebLLM makes possible. The function calling API follows the OpenAI tool calling format, so developers familiar with that pattern can implement browser-native function calling without learning a new API surface.
Contributing Models to the WebLLM Registry
The WebLLM model registry is community-maintained, and teams who want to deploy a custom fine-tuned model through WebLLM can contribute their model to the registry after converting it through the MLC compilation pipeline. The MLC team provides conversion scripts and documentation for the compilation process, which takes the original model weights and produces the optimised WebGPU kernels and model parameters that WebLLM needs. For teams building applications around a proprietary fine-tuned model, this compilation step is a one-time investment that produces a model deployable through WebLLM with the same performance characteristics as the pre-built registry models. Custom models can also be self-hosted rather than contributed to the public registry, using WebLLM’s API with a custom model URL pointing to your own CDN or storage bucket.
WebLLM vs Server-Side API: The Decision Framework
Choosing between WebLLM and a server-side API involves weighing several factors: privacy (WebLLM wins — data stays in the browser), model quality (API wins — frontier models far exceed what runs in a browser), cost at scale (WebLLM wins — no per-query cost after the one-time download), reliability (tie — both can fail, for different reasons), and first-run experience (API wins — no multi-gigabyte download). For most production web applications where quality matters, a hybrid approach works best: use a server API as the default for high-quality responses, with WebLLM as an optional privacy-preserving mode for users who opt into it. This gives users a choice rather than imposing either the quality ceiling of browser models or the privacy concerns of server APIs.