How to Run Command R+ Locally with Ollama

by

Command R+ is Cohere’s flagship open-weight model and one of the most capable models you can run locally for RAG (retrieval-augmented generation) and tool use. It was specifically trained with these workflows in mind — it handles long documents better than most models its size, supports structured tool calling natively, and performs well in multi-turn conversation over long contexts. At 104B parameters, it’s large, but the Q4 quantized version runs on machines that can fit it in memory.

This guide covers hardware requirements, how to pull and run Command R+ with Ollama, and the specific scenarios where it has a genuine advantage over alternatives.

Hardware Requirements

Command R+ is 104B parameters — larger than Llama 3.1 70B. In Q4 quantization it weighs approximately 59GB. Here’s what that means for your hardware:

Mac with 64GB+ unified memory (M4 Ultra, M3 Ultra, M4 Max 64GB, M3 Max 96GB): Runs well, though you’ll need to close other applications to free up memory. Expect 8–15 tokens per second depending on chip and available bandwidth.

Mac with 128GB unified memory (M4 Ultra, M3 Ultra): Comfortable — plenty of headroom. 12–20 tokens per second.

Multi-GPU NVIDIA with 64GB+ VRAM (e.g. dual RTX 3090 NVLink = 48GB — tight, dual RTX 6000 Ada = 96GB): Full GPU inference at 15–25 tokens per second on 96GB setups.

CPU with 64GB+ RAM: Runs, but very slowly — 0.5–2 tokens per second. Only practical for batch processing with extended wait times.

This puts Command R+ out of reach for most mid-range hardware. If you have a 24GB GPU or 32GB RAM machine, Mistral Small 3 (24B) or Llama 3.1 70B via partial offloading are better fits. Command R+ is for users with high-memory setups who specifically need its RAG and tool-use strengths.

Pulling Command R+ with Ollama

ollama pull command-r-plus

This downloads the Q4 quantized version — approximately 59GB. A significant download; plan accordingly. Once downloaded, loading takes 45–90 seconds on most hardware.

Cohere also released the smaller Command R (35B) which is more accessible:

ollama pull command-r

Command R in Q4 is about 20GB — manageable on a 24GB GPU or 32GB unified memory Mac. It has most of Command R+’s strengths in RAG and tool use, at lower quality. If your hardware can’t handle the full 104B, Command R is worth trying first.

What Command R+ Does Exceptionally Well

RAG pipelines. Cohere trained Command R+ specifically for retrieval-augmented generation. It handles the “grounded generation” pattern — generate a response based on provided documents, cite which documents you used — exceptionally well. It’s less likely to hallucinate content beyond the provided context than most models, and it formats citations cleanly when prompted to do so.

Long document reasoning. With a 128K context window and strong performance across it, Command R+ is one of the better models for tasks that require understanding and synthesising across a very long document. The quality doesn’t degrade as quickly near the end of long contexts as with many other models.

Multi-step tool use. It was trained specifically with function calling and agent-style workflows in mind. For building local AI agents that call tools, search, and reason over multiple steps, Command R+ is reliably better than general-purpose models at planning and executing multi-step sequences without getting confused mid-task.

Structured output. Asking it to produce JSON, tables, or other structured formats produces clean, consistent output. Useful in extraction and data pipeline scenarios where predictable output structure matters more than creative prose.

Figure 1 — Command R+ vs Alternatives: Where Each Excels

Task Command R+ Llama 3.1 70B Mistral Small 3 Phi-4 RAG / grounded generation★★★★★★★★★★★★★★★ Multi-step tool use / agents★★★★★★★★★★★★★★★ Long document analysis★★★★★★★★★★★★★★★ Math / logical reasoning★★★★★★★★★★★★★★★★ Hardware requirement64GB+ RAM48GB+ RAM16GB RAM12GB RAM

Running Command R+ for RAG

Command R+ has a specific RAG mode where you provide documents and it generates grounded responses. Here’s how to use it with the Ollama API:

import ollama

# Documents to ground the response in
documents = [
    "Quarterly revenue increased 23% year-over-year to $4.2B. Operating expenses grew 18% to $3.1B.",
    "The company expanded into three new markets: Brazil, India, and South Korea, contributing 8% of total revenue.",
    "Product margins improved from 42% to 47% due to supply chain optimisations implemented in Q2."
]

doc_context = "

".join([f"Document {i+1}:
{doc}" for i, doc in enumerate(documents)])

response = ollama.chat(
    model='command-r-plus',
    messages=[
        {
            'role': 'system',
            'content': f"""You are a financial analyst. Answer questions based only on the provided documents.
Always cite which document supports each claim.

Documents:
{doc_context}"""
        },
        {
            'role': 'user',
            'content': 'What drove revenue growth this quarter and how did margins change?'
        }
    ],
    options={'num_ctx': 32768}
)
print(response['message']['content'])

Command R+ naturally includes references to the documents it’s drawing from without needing elaborate prompt engineering to force citation behaviour. This is one of its distinguishing features versus models that weren’t specifically trained for grounded generation.

Tool Use with Command R+

Command R+ supports structured tool calling via the Ollama API:

import ollama
import json

tools = [{
    'type': 'function',
    'function': {
        'name': 'search_database',
        'description': 'Search the product database for items matching a query',
        'parameters': {
            'type': 'object',
            'properties': {
                'query': {'type': 'string', 'description': 'Search query'},
                'category': {'type': 'string', 'description': 'Product category filter'},
                'max_results': {'type': 'integer', 'description': 'Maximum results to return'}
            },
            'required': ['query']
        }
    }
}]

response = ollama.chat(
    model='command-r-plus',
    messages=[{'role': 'user', 'content': 'Find me all laptops under $1000 in the electronics category'}],
    tools=tools
)

if response['message'].get('tool_calls'):
    for tool_call in response['message']['tool_calls']:
        print(f"Tool: {tool_call['function']['name']}")
        print(f"Args: {json.dumps(tool_call['function']['arguments'], indent=2)}")

The reliability of tool call formatting — producing syntactically correct tool invocations on the first try without mangling the JSON — is noticeably better than general-purpose models on complex multi-argument tools. For production agent pipelines where tool call failures cause error cascades, this reliability is the practical reason to use Command R+ despite the hardware requirements.

Figure 2 — Command R and Command R+: Quick Comparison

Factor Command R (35B) Command R+ (104B) Size (Q4)~20 GB~59 GB Min RAM24 GB64 GB RAG qualityVery goodBest-in-class local Best for24–32GB hardware, most RAG needs64GB+ hardware, max RAG quality

Command R+ vs Llama 3.1 70B for RAG

If you have hardware that can run both Command R+ and Llama 3.1 70B, which should you use? For pure RAG use cases — providing documents and asking the model to answer from them — Command R+ is the better choice. It stays more faithfully grounded in the provided documents, produces cleaner citations, and is less likely to blend in hallucinated knowledge from its training data when the answer should come solely from the provided context. For general-purpose tasks beyond RAG — open-ended reasoning, creative work, coding, conversation — Llama 3.1 70B is broadly competitive and sometimes better. The practical approach: use Command R+ when your workflow is document-grounded retrieval and generation, use Llama 3.1 70B for everything else. Having both in your Ollama model library and selecting based on task type is the right strategy if your hardware supports both. The 128K context window they both offer is large enough that either can handle substantial documents — the difference is in how faithfully Command R+ uses that context versus how creatively Llama 3.1 fills gaps.

Optimising Command R+ for Your Hardware

Given the model’s size, a few hardware-specific optimisations make a meaningful difference. Enable Flash Attention before starting Ollama — it reduces KV cache memory significantly at the long context lengths Command R+ is typically used with: export OLLAMA_FLASH_ATTENTION=1. Set a reasonable context length for your use case rather than the maximum — for most RAG workflows, 16384 to 32768 tokens covers your document plus conversation without the memory overhead of 128K: ollama run command-r-plus --parameter num_ctx 32768. On Apple Silicon, Command R+ benefits strongly from the unified memory bandwidth — it loads faster and runs with better consistency than on CPU-only setups with the same RAM amount, because Apple Silicon’s memory bandwidth is significantly higher than typical DDR5 system RAM. If you’re on a Mac with 64GB unified memory, Command R+ is genuinely usable for interactive work rather than just batch processing. Keep OLLAMA_KEEP_ALIVE high when using Command R+ in active sessions — the 45–90 second reload time is significant enough that loading it repeatedly between queries is frustrating. Setting OLLAMA_KEEP_ALIVE to 30 minutes or more during a working session avoids repeated waits while still freeing memory when you’re done.

Getting Started: A Simple RAG Test

The fastest way to validate Command R+ is working correctly for your use case is a grounded generation test. Create a short test document, paste it into a system prompt, and ask a question that can only be answered from the document. If the model answers correctly and attributes the answer to your document rather than making up an answer, the RAG capability is working. If it ignores the document and generates from its own knowledge, check your system prompt structure — Command R+ needs explicit instruction to ground responses in provided documents rather than drawing on its own knowledge. The prompt pattern “Answer based only on the provided documents. If the answer is not in the documents, say so.” is reliable across most RAG use cases and is the baseline to start from before adding more sophisticated RAG prompt engineering for your specific domain.

Who Command R+ Is For

Command R+ is a specialist model for a specific class of use cases, not a general-purpose upgrade over smaller models. It makes sense to run it if you have the hardware and you’re building RAG pipelines, document Q&A systems, or multi-step agent workflows where grounded generation and reliable tool calling are the core requirements. It does not make sense to run it for general chat, creative writing, or coding tasks where Llama 3.1 70B, Qwen 2.5 14B, or Phi-4 offer comparable or better results on more accessible hardware. The 59GB footprint is a significant investment — make sure you’ve validated that Command R’s specific strengths are actually the bottleneck in your use case before committing to hardware that can run the 104B version. Command R (35B, ~20GB) is almost always the right starting point: it covers the same use cases, fits on 24GB hardware, and the quality difference between R and R+ is meaningful but not decisive for most applications. Upgrade to R+ when you’ve confirmed that R’s outputs aren’t meeting your quality bar for grounding and citation accuracy.

Command R+ in Production

For production RAG deployments where Command R+ is serving multiple users or processing high query volumes, the key bottleneck is inference speed — at 8–15 tokens per second on consumer hardware, it’s significantly slower than 7B models. If your use case requires sub-second response times or high throughput, Command R+ running locally is probably not the right architecture. It excels in lower-throughput, higher-quality scenarios: document research assistants, enterprise knowledge base Q&A, compliance checking against policy documents, or any application where a human waits 30–60 seconds for a high-quality grounded answer rather than expecting instant responses. The Ollama API handles concurrent requests (set OLLAMA_NUM_PARALLEL=2 if you have the memory headroom) and the model’s long context window means each request can handle substantial context without multiple round trips — both useful properties for production RAG architectures where query complexity varies widely across users.

Leave a Comment