How to Run Gemma 3 Locally with Ollama

Gemma 3 is Google’s best open-weight model family to date, and it’s a significant step up from Gemma 2. The standout version is the 27B model, which competes with much larger open-source models on reasoning and instruction following. But the whole family is worth knowing — from the 1B model that runs on almost anything to the 27B that needs a bit more hardware. Ollama makes pulling and running any of them a single command.

This guide covers the Gemma 3 model sizes, which to choose for your hardware, and how to get the most out of it locally.

Pulling Gemma 3 with Ollama

Gemma 3 is in the Ollama library under gemma3:

ollama run gemma3

This pulls the default size — currently the 4B model at Q4 quantization (~3.3GB). To pull a specific size:

ollama pull gemma3:1b     # 815MB — runs on almost anything
ollama pull gemma3:4b     # 3.3GB — excellent all-rounder
ollama pull gemma3:12b    # 8.1GB — strong reasoning
ollama pull gemma3:27b    # 17GB  — near-frontier quality

The 4B is the right starting point for most people. It runs comfortably on 8GB RAM, generates at a fast interactive speed, and the quality is genuinely impressive for its size. Google’s training efficiency improvements over Gemma 2 are visible in practice.

Gemma 3’s Multimodal Capability

One thing that makes Gemma 3 stand out from most other locally-runnable models: the 4B, 12B, and 27B variants are multimodal — they can understand images as well as text. You can send an image alongside your prompt and ask questions about it.

From the Ollama CLI:

ollama run gemma3 "Describe what's in this image" --image /path/to/image.jpg

Or via the API:

import ollama
import base64

with open('photo.jpg', 'rb') as f:
    image_data = base64.b64encode(f.read()).decode()

response = ollama.chat(
    model='gemma3:4b',
    messages=[{
        'role': 'user',
        'content': 'What is in this image? Describe it in detail.',
        'images': [image_data]
    }]
)
print(response['message']['content'])

This multimodal capability at the 4B scale is something very few other local models offer. LLaVA is the main alternative, but Gemma 3’s visual understanding is generally stronger and the text quality is higher. If you need local image analysis — document parsing, photo description, chart reading — Gemma 3 is one of the best options available right now.

Choosing the Right Gemma 3 Size

The model family covers a wide range of hardware requirements:

Gemma 3 1B: Less than 1GB in Q4. Runs on anything — 4GB RAM machines, old laptops, even Raspberry Pi 5. Useful for simple classification tasks, short Q&A, and edge deployment scenarios. Not competitive with larger models on complex tasks but fast and remarkably capable for 1B parameters.

Gemma 3 4B (default): The sweet spot for most people. 8GB RAM or 4GB VRAM is enough. Fast inference, multimodal, and strong on instruction following and general reasoning. This is the model you’d recommend to someone who wants to try Gemma 3 without worrying about hardware.

Gemma 3 12B: Needs 12GB RAM or 8GB VRAM (Q4). A significant step up in reasoning quality and output length. Handles complex multi-step tasks, longer document analysis, and more nuanced instruction following than the 4B. Worth it if your hardware supports it.

Gemma 3 27B: The flagship — 17GB in Q4, needs 20GB+ RAM or 24GB VRAM for comfortable full-GPU running. Competes with Llama 3.1 70B on several benchmarks despite being less than half the size. If you have an RTX 4090 or a Mac with 32GB+ unified memory and want the highest quality open-weight model you can run locally, this is a serious contender.

What Gemma 3 Is Good At

Instruction following is Gemma 3’s strongest suit. Google trained it with a heavy focus on following complex, multi-part instructions reliably — it’s one of the more “obedient” models in the local model ecosystem. If you have structured prompts with specific requirements (format, length, style, constraints), it tends to respect all of them rather than selectively applying them.

The 128K context window across all sizes is also genuinely useful. Most local models default to 2048 or 4096 tokens in Ollama; Gemma 3 can handle much more, though you’ll need to increase num_ctx explicitly:

ollama run gemma3 --parameter num_ctx 32768

Be aware that large context lengths require proportionally more RAM. For the 4B model, a 32K context needs about 4–5GB extra on top of the base model memory. For the 27B model with large context you’ll need a well-specced machine.

Code generation is solid without being exceptional. The 4B handles Python, JavaScript, and standard algorithms well. The 12B and 27B are more competitive with dedicated coding models. If code is your primary use case, Qwen 2.5 Coder still has a narrow edge — but Gemma 3’s advantage is doing code and image understanding and general Q&A all in one model.

Multilingual capability is better than most models its size. Google trained across many languages, and European and East Asian languages both come out well. Not as strong as Qwen 2.5’s multilingual coverage, but meaningfully better than most English-focused open-source alternatives.

Gemma 3 vs Other Models: Where It Fits

Comparing Gemma 3 to the other models in this guide helps clarify when to reach for it:

Versus Llama 3.2 3B / Llama 3.1 8B: Gemma 3 4B is competitive or better on most benchmarks and adds image understanding. If you’re already happy with Llama 3.2 for text tasks, Gemma 3 4B is worth adding to your toolkit specifically for multimodal use.

Versus Phi-4 14B: Phi-4 is stronger on mathematical reasoning. Gemma 3 12B is more versatile — better on open-ended tasks, image understanding, and long context. Different strengths, worth having both if your hardware allows.

Versus Qwen 2.5 Coder: For code-specific tasks, Qwen 2.5 Coder wins. For mixed tasks that include code alongside other content, Gemma 3 is more balanced.

Versus Llama 3.1 70B: Gemma 3 27B competes on many benchmarks at less than half the VRAM requirement. If you’ve been locked out of 70B-class quality by hardware, the 27B is your path in.

Practical Tips for Running Gemma 3

A few things worth knowing once you have it running. Gemma 3 uses a different tokenizer than Llama-family models, and its context handling is slightly different — the effective usable context at high quality is somewhat shorter than the maximum 128K suggests. For most practical purposes, keeping context under 32K tokens gives the best quality-to-performance ratio.

For image tasks, JPEG and PNG are both supported. Larger images are resized internally, so you don’t need to resize before sending. The 4B model handles image understanding well for everyday tasks — identifying objects, reading text in images, describing scenes — though complex visual reasoning (counting specific items, understanding diagrams with fine detail) is stronger in the 12B.

Gemma 3 responds well to explicit formatting instructions. Asking it to “respond in bullet points” or “use markdown headers” or “be concise, under 200 words” produces reliable results. The instruction following training shows clearly on structured output requests.

To use it with Open WebUI or any OpenAI-compatible client, the model name is just gemma3 or gemma3:4b, and it connects via the standard Ollama OpenAI-compatible endpoint at /v1/chat/completions. Multimodal requests work through the same endpoint using the standard image content format.

Gemma 3 represents a meaningful shift for Google’s open-weight model strategy — the 4B model is genuinely one of the best things you can run in that size class, and the multimodal capability at that footprint is something competitors haven’t matched yet. If you haven’t tried it, it’s worth the 3GB download.

Using Gemma 3 for Document Q&A

One of the best local use cases for Gemma 3 is chatting with documents — pasting in a PDF, report, or article and asking specific questions about it. With the 128K context window and good instruction following, you can feed in surprisingly long documents and ask nuanced questions. The workflow is simple: set a high num_ctx, paste your document in the system or user message, then ask your questions. For example, a 20-page PDF converted to text is roughly 8–12K tokens — well within the 4B model’s comfortable range with num_ctx 16384. The model’s tendency toward structured, accurate responses makes it well-suited to this: it cites specific parts of the document, distinguishes what’s in the text from what it’s inferring, and handles follow-up questions about the same document coherently across a multi-turn conversation. Combine it with a simple Python script to extract PDF text (pypdf2 or pdfplumber work well) and you have a capable local document Q&A system that costs nothing per query and doesn’t send your documents to any external service.

Gemma 3 on Apple Silicon

Apple Silicon is a particularly good platform for Gemma 3. The unified memory model means all four sizes run without VRAM constraints on well-specced Macs — a Mac with 24GB unified memory handles the 12B model comfortably, and the 27B runs on 32GB+ configurations. Metal GPU acceleration is automatic through Ollama, and the memory bandwidth on M3/M4 chips makes inference fast enough for interactive use even on the larger models. On an M4 Pro MacBook Pro with 24GB, the 12B model runs at around 35–45 tokens per second, which is comfortable for real use. The 4B is fast enough to feel near-instant — 60–80 tokens per second on the same chip. If you’re on Apple Silicon and haven’t tried Gemma 3 yet, it’s one of the models worth keeping permanently downloaded alongside whatever else you have installed. The combination of multimodal capability, strong instruction following, and good multilingual coverage makes it versatile enough to handle a wide range of tasks without switching models.

Troubleshooting Common Issues with Gemma 3

The most common issue people run into is the model loading but running slowly. If you’re seeing 2–5 tokens per second on the 4B model and expect faster, first check whether GPU acceleration is active — run ollama ps and look for GPU utilisation. If it shows CPU-only, check your GPU setup (see the CUDA or Metal sections of this guide for your platform). The second common issue is running out of memory when setting a large context. Increasing num_ctx to 32768 or higher requires significantly more RAM than the base model footprint — if you get out-of-memory errors, reduce num_ctx back to 8192 and increase gradually. On the multimodal side, if image inputs aren’t being processed correctly, check that you’re using a model variant that supports vision (gemma3:4b, 12b, or 27b — not 1b) and that images are being passed as base64-encoded strings in the correct format for the API you’re using. The Ollama Python library handles this automatically; if you’re using raw HTTP requests, make sure the image data is base64-encoded and passed in the images array of the request body, not as a URL.

Combining Gemma 3 with Open WebUI for a Full Local Setup

Gemma 3’s multimodal capabilities are easiest to use through a chat UI rather than the CLI, and Open WebUI handles image uploads natively. With Open WebUI connected to your Ollama instance, you can drag and drop images into the chat window and ask questions about them exactly as you would with a cloud multimodal model — except everything runs locally. The setup is the same as with any other Ollama model: install Open WebUI via Docker, point it at your Ollama host, and select gemma3 from the model dropdown. The image upload button appears automatically when you select a vision-capable model. For document workflows, you can pair this with Open WebUI’s document upload feature, which chunks and indexes uploaded files for RAG-style retrieval — giving you a complete local document assistant that combines Gemma 3’s strong instruction following with retrieval over your own document library. It’s one of the most capable local AI setups you can build without any cloud dependencies, and the hardware requirement for the 4B version is low enough that it runs well on a mid-range machine most people already own.

What to Try Next After Gemma 3

Once Gemma 3 is running, the natural next experiments depend on what you’re using it for. For coding tasks, pull Qwen 2.5 Coder alongside it and compare outputs on the same problems — having both lets you route code-specific tasks to the specialist while using Gemma 3 for everything else. For heavier reasoning, try the 12B or 27B variants if your hardware supports them. For pipeline and automation use, the Ollama Python library makes it straightforward to build scripts that use Gemma 3 for text and image understanding in sequence — process an image to extract text, then run that extracted text through another prompt for classification or summarisation, all running locally. Google has committed to the Gemma series as an ongoing effort, and future versions are likely to push the quality-per-parameter ratio further. For now, Gemma 3 sits among the top choices at every size class it offers — a well-rounded model family that covers more use cases than most alternatives of comparable size.

Running Gemma 3 via the Python API

For scripted use, the Ollama Python library gives you clean access to all Gemma 3 features. Install it with pip install ollama and use it like this:

import ollama

# Text only
response = ollama.chat(
    model='gemma3:4b',
    messages=[{
        'role': 'system',
        'content': 'You are a helpful, concise assistant.'
    }, {
        'role': 'user',
        'content': 'Explain the difference between supervised and unsupervised learning in plain English.'
    }]
)
print(response['message']['content'])

# Streaming for real-time output
stream = ollama.chat(
    model='gemma3:4b',
    messages=[{'role': 'user', 'content': 'Write a short explanation of neural networks.'}],
    stream=True
)
for chunk in stream:
    print(chunk['message']['content'], end='', flush=True)

Gemma 3 also works with the OpenAI-compatible endpoint at /v1/chat/completions, making it a drop-in for any tool that supports custom OpenAI base URLs. Point your client at http://localhost:11434/v1, set any string as the API key, and use gemma3 as the model name. For vision tasks through the API, use the standard multi-modal message format with base64-encoded images — the Ollama library handles the encoding automatically when you pass file paths.

Gemma 3 is one of those models that rewards experimentation — pulling the 4B and spending an afternoon putting it through different task types will reveal where it fits best in your specific workflow, and the results are usually more impressive than the parameter count would suggest. Start with the 4B, try it on a few real tasks you care about, and upgrade to 12B or 27B if you want more depth on the tasks where it’s falling short. With Ollama managing the download and serving, the whole process from zero to running is under 10 minutes for the 4B model — a low enough investment to make the experiment worthwhile even if you end up preferring a different model for your primary use.

How Gemma 3 Compares to Gemma 2

If you’ve used Gemma 2 before, Gemma 3 is a meaningful upgrade across the board. The instruction following is more reliable, the 4B model’s quality jumped noticeably (it now competes where Gemma 2 9B used to sit), the context window is extended to 128K (Gemma 2 topped out at 8K), and the addition of native multimodal capability is entirely new. The 27B variant in particular represents a significant leap — Gemma 2 27B was good but not exceptional; Gemma 3 27B sits comfortably at near-frontier quality. If Gemma 2 is in your current model toolkit, replacing it with Gemma 3 is a straightforward upgrade with no workflow changes required, just a new model pull.