How to Run Llama 3.3 70B Locally with Ollama

Llama 3.3 70B is the largest model most people can realistically run on consumer hardware — and it’s genuinely impressive. Meta’s instruction-tuned 70B model sits in a category by itself among open-source models: strong enough to compete with frontier API models on many tasks, but running entirely on your machine. The catch is that you need serious hardware. This guide covers what that hardware looks like, how to get the model running with Ollama, and what to expect in practice.

If you’re wondering whether your machine can handle it — that’s the first thing we’ll sort out.

Hardware Requirements: The Honest Picture

Llama 3.3 70B in Q4 quantization weighs in at around 43GB. That has to fit somewhere — either in VRAM, system RAM, or split across both. Here’s what actually works:

NVIDIA GPU with 48GB+ VRAM (RTX 6000 Ada, A6000, dual 3090s with NVLink): Full GPU inference. Fast — 20–35 tokens per second. This is the ideal setup but requires expensive hardware.

NVIDIA 24GB GPU (RTX 4090, 3090, 4080): Partial GPU offloading. The model splits between VRAM and system RAM. You’ll need at least 48GB total system RAM. Inference runs at 5–12 tokens per second depending on how many layers fit in VRAM — slower than full GPU but faster than CPU-only.

Apple Silicon with 64GB+ unified memory (M4 Ultra, M3 Ultra, M2 Ultra, M4 Max 64GB): The best consumer option. Unified memory means the whole model fits in fast memory without CPU/GPU splits. Expect 15–25 tokens per second — slower than a top NVIDIA setup but consistent and comfortable for real use.

CPU only with 64GB+ RAM: Works, but slow. On a modern 16-core CPU you’ll see 1–3 tokens per second with the Q4 70B model. Usable for batch tasks where you’re not waiting for the output, but not for interactive use.

48–64GB system RAM + mid-range GPU: The model loads mostly into RAM with some layers on GPU. Speeds vary significantly — anywhere from 3–10 tokens per second depending on your GPU and the layer split Ollama chooses.

Pulling Llama 3.3 70B with Ollama

Once you’ve confirmed your hardware can handle it, pulling the model is straightforward:

ollama pull llama3.3:70b

This downloads the default Q4_K_M quantized version — about 43GB. The download takes a while on most connections. You can also pull specific quantization variants if you want to trade quality for speed or memory:

ollama pull llama3.3:70b-instruct-q2_K    # 26GB, much faster, quality loss
ollama pull llama3.3:70b-instruct-q4_K_M  # 43GB, recommended
ollama pull llama3.3:70b-instruct-q8_0    # 74GB, highest quality, needs huge RAM

Q4_K_M is the sweet spot for most setups — minimal quality loss versus full precision, manageable size. Q2 is only worth considering if you’re genuinely constrained by memory and have no alternative. Q8 requires 80GB+ of fast memory to run at a reasonable speed.

Once downloaded, run it:

ollama run llama3.3:70b

The first load takes 30–60 seconds as it reads the model into memory. Subsequent loads from the same session are faster since Ollama keeps it warm.

Maximising GPU Layer Offloading

When the 70B model is too large to fit entirely in VRAM, Ollama automatically splits it — some transformer layers run on GPU, the rest on CPU. More layers on GPU means faster inference. Ollama picks the split automatically, but you can be explicit with the num_gpu parameter:

# Force all layers to GPU (fails if VRAM is insufficient)
ollama run llama3.3:70b --parameter num_gpu 999

# Offload a specific number of layers to GPU (tune this to your VRAM)
ollama run llama3.3:70b --parameter num_gpu 40

To find the sweet spot, start with Ollama’s automatic choice and run ollama ps in another terminal to see how many GPU layers are active. Then increase num_gpu incrementally and watch whether the model still loads successfully (not running out of VRAM). On an RTX 4090 with 24GB, you can typically offload around 40–45 of the 80 transformer layers, keeping the rest in CPU RAM.

Watch GPU utilisation during inference with:

nvidia-smi dmon -s u   # Linux/Windows with NVIDIA
# On Mac: Activity Monitor → GPU History tab

What the 70B Model Is Actually Good For

Running a 70B model locally isn’t about bragging rights — it’s about having access to a qualitatively different level of reasoning without API costs or data leaving your machine. The 70B model handles things that smaller models consistently struggle with:

Long document analysis. With a 128K context window (set via num_ctx), you can feed in a full research paper, a long contract, or a large codebase and ask nuanced questions across the whole thing. Smaller models lose coherence over long contexts; the 70B model maintains it much better.

Complex multi-step reasoning. Tasks that require holding multiple constraints in mind simultaneously — complex logic problems, system design questions, multi-variable analysis — benefit significantly from model size. The 70B’s reasoning is noticeably more careful and complete than a 7B or 13B model on these.

Nuanced writing. The output quality for long-form writing — technical documentation, detailed analysis, structured reports — is closer to what you’d expect from a frontier API model. The language is more precise and the arguments are more coherent than smaller models.

Fewer hallucinations on factual tasks. While no LLM is hallucination-free, the 70B model has notably better calibration on what it knows and doesn’t know, and is more likely to express appropriate uncertainty rather than confidently generating wrong information.

Setting Context Length and Memory Usage

The 70B model supports up to 128K tokens of context. By default, Ollama sets num_ctx to 2048 — tiny for what this model can handle. Increase it based on your use case, keeping in mind that larger context requires proportionally more RAM:

# For document analysis (needs ~80GB+ RAM for full 70B + large context)
ollama run llama3.3:70b --parameter num_ctx 32768

# For conversational use (more manageable)
ollama run llama3.3:70b --parameter num_ctx 8192

A rough rule: each 1K tokens of context adds about 1MB of memory overhead for the 70B model. A 32K context costs around 32GB extra on top of the base model memory requirement — plan accordingly.

Is It Worth Running on Your Hardware?

The honest answer depends on what you’re trying to do and what hardware you have. If you’re using an M4 Max Mac or a high-end NVIDIA workstation with 48GB+ VRAM and you need a private, offline model for serious work — document analysis, research, complex writing — Llama 3.3 70B is worth every GB of storage. The quality gap over a 13B model on demanding tasks is real and meaningful.

If you’re on an RTX 4090 with 32GB system RAM and want interactive use, the partial offloading at 8–12 tokens per second is workable but not comfortable for back-and-forth conversation. In that scenario, a well-quantized 13B model running fully on GPU often delivers a better experience even if it’s technically less capable — because it’s actually fast enough to feel responsive.

For batch tasks — processing documents overnight, generating content offline, summarising large files — CPU-only at 2 tokens per second is perfectly acceptable. You’re not waiting for it; you’re letting it work while you do something else. In that context, the 70B model’s quality advantage is worth the slower speed.

The 70B is a remarkable achievement in open-source AI. Running something this capable on hardware you own, with no API costs and complete privacy, was simply not possible two years ago. Whether it makes sense for you comes down to your hardware and your specific use case — but if you have the RAM for it, it’s worth trying at least once to see what local AI at this scale actually feels like.

Running 70B via the API

Once Llama 3.3 70B is loaded, the Ollama REST API is the same as any other model — just use llama3.3:70b as the model name. The main practical difference from smaller models is that you’ll want to account for the load time in any application that needs to load the model fresh. Setting OLLAMA_KEEP_ALIVE=-1 keeps the model permanently loaded in memory, so subsequent API calls respond immediately without waiting 30–60 seconds for the model to load each time. This obviously consumes the full ~43GB of memory continuously, so only use it if you’re actively running workloads against the 70B model and have memory to spare. If you’re mixing 70B and smaller models in the same session, a shorter keep-alive (say, 10–15 minutes) is more practical — it unloads the 70B when you switch to a smaller model and frees that memory back to the system.

Troubleshooting: When the 70B Won’t Load

The most common failure is running out of memory. If Ollama hangs during loading or returns a memory error, you don’t have enough RAM + VRAM for the model. Close all other applications first and try again — on a machine with exactly the minimum memory, freeing 4–8GB can make the difference. If you’re using partial GPU offloading and the model loads but inference is extremely slow (under 1 token per second), try reducing num_gpu to put fewer layers on GPU — counter-intuitively, reducing GPU layers can sometimes improve speed if VRAM thrashing is causing the bottleneck. Finally, if the model downloads but won’t run, check that you have enough free disk space for the model plus working space — a full disk will cause cryptic failures that don’t immediately point to storage as the cause.