How to Use Ollama with CrewAI and AutoGen: Local AI Agents Guide

AI agent frameworks — systems where multiple LLM-backed agents collaborate on tasks, use tools, and produce coordinated outputs — have matured significantly, and both CrewAI and AutoGen work well with Ollama as the local inference backend. Running agents locally means zero API costs for long multi-step agent runs, no rate limits when agents make many LLM calls in sequence, and all data staying on your machine. This guide covers setting up both frameworks with Ollama and building practical agents you can actually use.

CrewAI with Ollama

CrewAI organises AI agents into “crews” — groups of specialised agents that collaborate toward a shared goal. Each agent has a role, goal, and backstory that shapes how it approaches tasks. Agents are assigned tasks, can delegate to each other, and produce outputs that feed into subsequent tasks. The framework handles the orchestration; your job is to define the agents and what they should accomplish.

pip install crewai crewai-tools

Configuring CrewAI to Use Ollama

CrewAI uses the OpenAI-compatible endpoint. Set the environment variables before running your crew:

export OPENAI_API_BASE="http://localhost:11434/v1"
export OPENAI_MODEL_NAME="llama3.1"
export OPENAI_API_KEY="ollama"

Or set them in Python before importing CrewAI:

import os
os.environ["OPENAI_API_BASE"] = "http://localhost:11434/v1"
os.environ["OPENAI_MODEL_NAME"] = "llama3.1"
os.environ["OPENAI_API_KEY"] = "ollama"

Alternatively, configure the LLM directly on each agent for more control:

from crewai import Agent, Task, Crew
from langchain_openai import ChatOpenAI

ollama_llm = ChatOpenAI(
    model="ollama/llama3.1",
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

A Practical CrewAI Example: Research and Writing Crew

from crewai import Agent, Task, Crew, Process
import os

os.environ["OPENAI_API_BASE"] = "http://localhost:11434/v1"
os.environ["OPENAI_MODEL_NAME"] = "llama3.1"
os.environ["OPENAI_API_KEY"] = "ollama"

# Define agents
researcher = Agent(
    role="Senior Research Analyst",
    goal="Find and synthesise the most relevant information on a given topic",
    backstory="""You are an expert researcher who excels at finding key insights,
    identifying patterns, and presenting information clearly and concisely.""",
    verbose=True,
    allow_delegation=False
)

writer = Agent(
    role="Technical Writer",
    goal="Write clear, engaging content based on research provided",
    backstory="""You are a skilled technical writer who transforms complex research
    into accessible, well-structured documents that readers find genuinely useful.""",
    verbose=True,
    allow_delegation=False
)

# Define tasks
research_task = Task(
    description="""Research the current state of local LLM inference in 2026.
    Focus on: key models available, hardware requirements, main use cases,
    and how the ecosystem has evolved. Provide a structured research summary.""",
    expected_output="A structured research summary with key findings, organised by theme",
    agent=researcher
)

writing_task = Task(
    description="""Using the research provided, write a 500-word blog post
    introduction about local LLM inference for a developer audience.
    Make it engaging, practical, and focused on real-world benefits.""",
    expected_output="A 500-word blog post introduction, ready to publish",
    agent=writer,
    context=[research_task]
)

# Assemble and run the crew
crew = Crew(
    agents=[researcher, writer],
    tasks=[research_task, writing_task],
    process=Process.sequential,
    verbose=True
)

result = crew.kickoff()
print(result)

The sequential process runs research_task first, then passes its output as context to writing_task. The researcher agent analyses the topic, the writer agent produces the final content. Each agent’s reasoning is printed (verbose=True) so you can follow what’s happening.

Model Choice for CrewAI Agents

Agentic tasks are more demanding than single-turn chat — agents need to follow complex instructions, reason about what to do next, format outputs correctly for downstream tasks, and avoid getting stuck in loops. This means model choice matters more than for simple chat applications. Llama 3.1 8B is the minimum practical model for simple agent tasks. For crews with multiple agents doing complex reasoning, Qwen 2.5 14B or Mistral Small 3 24B produce significantly more reliable agent behaviour — fewer hallucinated tool calls, better task decomposition, and more consistent output formatting. If your hardware supports it, running 70B-class models for agent tasks produces much more reliable autonomous behaviour, though the slower inference makes long agent runs take considerably more time. The trade-off is explicit: faster smaller models for simpler agent tasks, slower larger models for tasks requiring careful multi-step reasoning.

AutoGen with Ollama

AutoGen is Microsoft’s agent framework, with a different philosophy from CrewAI. Where CrewAI defines agents with roles and crews them together, AutoGen focuses on conversational agents that collaborate through dialogue — agents talk to each other, and complex tasks emerge from their conversation. It’s particularly strong for tasks that benefit from iterative back-and-forth rather than a fixed pipeline of sequential tasks.

pip install pyautogen

Configuring AutoGen for Ollama

import autogen

config_list = [
    {
        "model": "llama3.1",
        "api_key": "ollama",
        "base_url": "http://localhost:11434/v1",
        "api_type": "openai"
    }
]

llm_config = {
    "config_list": config_list,
    "timeout": 120,
    "temperature": 0.1,
}

# Create agents
assistant = autogen.AssistantAgent(
    name="Assistant",
    llm_config=llm_config,
    system_message="You are a helpful AI assistant. Be concise and precise."
)

user_proxy = autogen.UserProxyAgent(
    name="User",
    human_input_mode="NEVER",    # fully automated
    max_consecutive_auto_reply=5,
    code_execution_config={
        "work_dir": "workspace",
        "use_docker": False
    },
    llm_config=llm_config
)

# Run a conversation
user_proxy.initiate_chat(
    assistant,
    message="Write a Python function to calculate the nth Fibonacci number recursively, then test it with n=10."
)

AutoGen’s UserProxyAgent can execute code that the assistant writes — it runs the code locally in the work_dir, captures the output, and feeds it back to the assistant. This creates a loop where the assistant writes code, the proxy runs it, reports the result, and the assistant can fix bugs or iterate. With human_input_mode="NEVER" this runs fully automatically until the task is complete or max_consecutive_auto_reply is reached.

A Multi-Agent Code Review System with AutoGen

import autogen
import os

config_list = [{
    "model": "qwen2.5-coder:7b",
    "api_key": "ollama",
    "base_url": "http://localhost:11434/v1",
}]
llm_config = {"config_list": config_list, "timeout": 120}

# Coding agent writes code
coder = autogen.AssistantAgent(
    name="Coder",
    llm_config=llm_config,
    system_message="""You are an expert Python developer.
    Write clean, efficient, well-documented code.
    Always include type hints and docstrings."""
)

# Reviewer checks the code
reviewer = autogen.AssistantAgent(
    name="Reviewer",
    llm_config=llm_config,
    system_message="""You are a senior code reviewer.
    Review code for: correctness, edge cases, performance, security, style.
    Be specific about issues and suggest concrete improvements."""
)

# Proxy runs the conversation
proxy = autogen.UserProxyAgent(
    name="Manager",
    human_input_mode="NEVER",
    max_consecutive_auto_reply=10,
    code_execution_config=False,
    default_auto_reply="Continue. Coder, implement the reviewer's suggestions."
)

# Start with a group chat
groupchat = autogen.GroupChat(
    agents=[proxy, coder, reviewer],
    messages=[],
    max_round=8
)
manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)

proxy.initiate_chat(
    manager,
    message="""Task: Write a Python class that implements a thread-safe LRU cache.
    Requirements: max size configurable, O(1) get and put, thread-safe, includes tests."""
)

The GroupChat has the Coder write the initial implementation, the Reviewer critiques it, and the Coder revises based on feedback — a loop that continues for up to 8 rounds. The proxy’s default auto-reply prompts the next agent to continue when no human input is available.

Practical Challenges with Local Agent Frameworks

Running agent frameworks with local models surfaces some challenges that don’t appear with cloud APIs. The most significant is instruction following reliability — agent frameworks depend on models following complex, multi-part instructions precisely, including output format requirements that downstream agents depend on. Smaller local models (7B and below) frequently fail to format outputs as expected, miss instructions in long system prompts, or get stuck in loops repeating similar responses. The practical mitigation: use models of at least 14B parameters for any non-trivial agentic task, be explicit about output formats in your agent system prompts, and set conservative max_consecutive_auto_reply limits to prevent runaway agent loops that consume your hardware resources without producing useful output.

The second challenge is latency — each agent turn is a full LLM inference call, and complex multi-agent workflows with 10–20 turns take minutes even on capable hardware. This is expected and fine for batch tasks, but makes interactive agent use slow. Design agents with the minimum number of turns needed to complete the task, and use smaller models for agents doing simple routing or validation tasks while reserving larger models for agents doing actual reasoning.

The third challenge is determinism — with temperature=0 you get more consistent behaviour, but local models at low temperature sometimes become repetitive or refuse to make decisions. A temperature of 0.1–0.3 usually gives the right balance of consistency and willingness to take initiative for agent tasks.

CrewAI vs AutoGen: Which to Choose

The frameworks have different strengths that map to different use cases. CrewAI’s role-based crew model works well for content and analysis pipelines — research crews, content generation workflows, document processing pipelines where each agent has a specific specialisation and hands off to the next. The pipeline structure is intuitive to design and debug. AutoGen’s conversational model is better for iterative problem-solving — coding tasks where the agent needs to run code, see errors, and fix them; research tasks where the agents debate and refine an answer; any workflow that benefits from back-and-forth refinement rather than a fixed pipeline. AutoGen’s built-in code execution is particularly valuable — the ability to run generated code and incorporate the output into the conversation creates a natural feedback loop that CrewAI requires external tools to achieve. For teams building multiple agent applications, knowing both frameworks and matching the framework to the task type produces better results than defaulting to one for everything.

Keeping Agent Costs Under Control

One of the biggest advantages of running agents locally with Ollama is eliminating the per-token cost of agent workflows, which can be substantial with cloud APIs. A CrewAI crew making 50 LLM calls with a 70B model running locally costs zero in API fees — the same workflow via GPT-4o could cost $5–20 depending on context lengths. This changes how you design agents: with free local inference, you can afford to be generous with context, run multiple iterations, and use verbose agent descriptions without worrying about cost. You can also run experiments freely — trying different crew compositions, testing different models for each agent role, running the same task multiple times to compare outputs — that would be expensive with cloud APIs. The hardware investment amortises quickly for anyone building agent-heavy applications, and the development velocity improvement from zero-cost iteration is significant.

Guardrails and Safety for Local Agents

When agents run autonomously — executing code, making decisions, producing outputs that feed into further automated steps — it’s worth thinking about what guardrails you need. AutoGen’s code execution is sandboxed by default (the work directory) but can access your filesystem, run network requests, and install packages if not constrained. For development, this is usually fine. For production deployments handling sensitive data or with broader permissions, add explicit constraints: use Docker for code execution to isolate from the host filesystem, restrict the working directory’s network access, and set a strict max_consecutive_auto_reply cap so agents can’t run indefinitely. CrewAI is generally safer by default since it doesn’t execute code directly, but any agent that can call external tools (web search, API calls, file operations) needs the same thoughtfulness about what actions you’re permitting. The local execution context means a misconfigured agent deletes a local file or consumes significant CPU is your problem rather than a cloud provider’s — which is both the freedom and the responsibility of running agents locally. Start with conservative permissions and expand them as you gain confidence in your agent’s behaviour on your specific tasks.

Debugging Agent Workflows

Agent debugging is qualitatively different from debugging regular code — the non-determinism and emergent behaviour of multi-agent systems makes traditional debugging approaches less effective. The most useful technique is verbose logging: both CrewAI (verbose=True) and AutoGen log each agent turn’s input and output, which lets you trace exactly where things went wrong. When an agent produces unexpected output, reading the full conversation history usually reveals whether the agent misunderstood its task, received malformed input from a previous step, or simply made a reasoning error. Keeping runs short during development (low max_consecutive_auto_reply, simple tasks) lets you iterate quickly. Save successful runs’ full conversation logs — they’re valuable reference points for understanding what prompt configurations reliably produce good agent behaviour, and for regression testing when you update models or change agent configurations.