How to Use Ollama with CrewAI and AutoGen: Local AI Agents Guide

AI agent frameworks let you build systems where multiple LLM-powered agents collaborate on complex tasks — one agent researches, another writes, a third reviews, and an orchestrator coordinates the whole thing. CrewAI and AutoGen are two of the most popular frameworks for this, and both support Ollama as the local inference backend. Running agents locally means no API costs, no data leaving your machine, and no rate limits — you can run long, complex multi-agent workflows without worrying about the bill. This guide covers how to set up both frameworks with Ollama and build practical agent workflows.

Why Local Agents Matter

Cloud-based agent workflows have a hidden cost beyond API fees: each agent “turn” in a multi-agent conversation is a separate API call. A complex 10-agent workflow with 5 turns each generates 50 API calls, and at GPT-4o pricing that adds up quickly for regular use. With Ollama, those 50 inference calls cost nothing per call — just hardware time. For teams experimenting with agent architectures, this makes the difference between “we can afford to iterate extensively” and “we need to be careful about how many runs we do.” Agents also tend to generate long outputs and consume large contexts, which amplifies API costs. Local inference inverts this concern: longer contexts are a hardware constraint rather than a cost constraint.

CrewAI with Ollama

CrewAI models AI workflows as a crew of agents, each with a specific role, goal, and backstory. Agents are assigned tasks, can use tools, and collaborate to produce a final output. It’s the most approachable agent framework for users new to multi-agent systems — the role-based mental model maps naturally to how teams of people work.

pip install crewai crewai-tools

Configure Ollama as the LLM via environment variables — CrewAI uses the LiteLLM library for model access, which supports Ollama natively:

export OPENAI_API_BASE=http://localhost:11434/v1
export OPENAI_API_KEY=ollama
export OPENAI_MODEL_NAME=ollama/llama3.1:8b

Or configure it directly in Python:

from crewai import Agent, Task, Crew, Process
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="ollama/llama3.1:8b",
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

# Define agents
researcher = Agent(
    role="Research Analyst",
    goal="Find and summarise key information on a given topic",
    backstory="You are an expert researcher who distils complex information into clear summaries.",
    llm=llm,
    verbose=True
)

writer = Agent(
    role="Technical Writer",
    goal="Write clear, engaging content based on research findings",
    backstory="You transform research notes into polished, readable articles.",
    llm=llm,
    verbose=True
)

reviewer = Agent(
    role="Editor",
    goal="Review content for accuracy, clarity and completeness",
    backstory="You are a meticulous editor who catches errors and improves clarity.",
    llm=llm,
    verbose=True
)

# Define tasks
research_task = Task(
    description="Research the key concepts and recent developments in local LLM inference optimisation.",
    expected_output="A detailed research brief covering key techniques, benchmarks, and practical recommendations.",
    agent=researcher
)

writing_task = Task(
    description="Write a 500-word article based on the research brief.",
    expected_output="A well-structured, engaging article ready for publication.",
    agent=writer,
    context=[research_task]
)

review_task = Task(
    description="Review the article and suggest specific improvements.",
    expected_output="A final reviewed version with tracked changes and commentary.",
    agent=reviewer,
    context=[writing_task]
)

# Assemble and run the crew
crew = Crew(
    agents=[researcher, writer, reviewer],
    tasks=[research_task, writing_task, review_task],
    process=Process.sequential,
    verbose=True
)

result = crew.kickoff()
print(result)

This three-agent pipeline — research, write, review — runs entirely locally. Each agent uses the same Ollama model (you can assign different models to different agents by creating multiple ChatOpenAI instances), and the full output of each task feeds into the next as context.

Choosing Models for CrewAI Agent Roles

Not all agents in a crew need the same model. A useful pattern: assign a capable 7–8B model to worker agents doing focused tasks, and a larger model to the orchestrator or reviewer role where broader reasoning matters. CrewAI supports per-agent LLM configuration, so you can give the researcher a fast model (Llama 3.2 3B for simple lookups) and the reviewer a more capable one (Llama 3.1 8B or Qwen 2.5 14B for nuanced feedback). This balances throughput with quality — the bottleneck is usually the reasoning-heavy tasks, not the information-gathering ones.

Tool use in CrewAI works well with models that have been specifically trained for it: Llama 3.1, Qwen 2.5, and Mistral Small 3 are the strongest locally runnable options for agents that need to reliably call tools. If your crew agents need to use web search, file operations, or custom API calls, test tool reliability on a small workflow before scaling — some models are significantly more reliable at tool calling than others, and the difference shows up quickly in agent workflows where a bad tool call can derail the entire task sequence.

AutoGen with Ollama

AutoGen from Microsoft takes a different approach — agents communicate through a conversational back-and-forth rather than executing predefined task sequences. The most powerful pattern is the AssistantAgent + UserProxyAgent pair: the assistant generates responses and code, the user proxy executes code and reports results back, and they iterate until the task is complete. This makes AutoGen particularly strong for tasks involving code generation and execution.

pip install pyautogen

import autogen

# Configure Ollama as the LLM
config_list = [{
    "model": "llama3.1:8b",
    "base_url": "http://localhost:11434/v1",
    "api_key": "ollama",
    "api_type": "openai"
}]

llm_config = {
    "config_list": config_list,
    "temperature": 0.1,
    "timeout": 120
}

# Create assistant agent
assistant = autogen.AssistantAgent(
    name="Assistant",
    llm_config=llm_config,
    system_message="You are a helpful AI assistant. When asked to write code, write clean, well-commented Python."
)

# Create user proxy (executes code locally)
user_proxy = autogen.UserProxyAgent(
    name="User",
    human_input_mode="NEVER",    # or "TERMINATE" to review each step
    max_consecutive_auto_reply=10,
    code_execution_config={
        "work_dir": "agent_workspace",
        "use_docker": False         # True for sandboxed execution
    },
    is_termination_msg=lambda msg: "TERMINATE" in msg.get("content", "")
)

# Start a task
user_proxy.initiate_chat(
    assistant,
    message="Write a Python script that downloads the top 10 trending topics from Hacker News and saves them to a CSV file."
)

AutoGen’s UserProxyAgent actually executes the code the assistant writes — in the agent_workspace directory on your machine. The assistant sees the execution output and can fix errors, refine the code, and iterate. This loop continues until the task completes or the maximum reply count is reached. The code runs locally, the inference runs locally via Ollama, and the whole interaction produces a working script at the end.

AutoGen GroupChat: Multiple Agents Collaborating

AutoGen’s GroupChat lets multiple agents participate in the same conversation, with a manager deciding who speaks next:

import autogen

config_list = [{"model": "llama3.1:8b", "base_url": "http://localhost:11434/v1", "api_key": "ollama", "api_type": "openai"}]
llm_config = {"config_list": config_list, "temperature": 0.1}

coder = autogen.AssistantAgent("Coder", llm_config=llm_config,
    system_message="You write clean Python code. Always include error handling.")

reviewer = autogen.AssistantAgent("Reviewer", llm_config=llm_config,
    system_message="You review code for bugs, security issues, and best practices. Be specific.")

user_proxy = autogen.UserProxyAgent("User", human_input_mode="NEVER",
    code_execution_config={"work_dir": "workspace", "use_docker": False},
    is_termination_msg=lambda m: "TERMINATE" in m.get("content", ""))

groupchat = autogen.GroupChat(
    agents=[user_proxy, coder, reviewer],
    messages=[],
    max_round=12
)
manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)

user_proxy.initiate_chat(manager,
    message="Build a command-line tool that monitors a log file and alerts when error rate exceeds 5% in any 60-second window."
)

Practical Tips for Local Agent Workflows

Running agent frameworks locally introduces some practical considerations that don’t apply to cloud-based setups. Agents generate substantial context as conversations grow — each agent turn adds tokens to the accumulated history, and without careful context management, long workflows can exceed even generous context windows. Set reasonable max_consecutive_auto_reply limits in AutoGen and use explicit termination conditions (the TERMINATE keyword or a custom function) to prevent runaway conversations that consume memory and generate unhelpful outputs as context overflows. For CrewAI, limit task descriptions to the essential requirements rather than including exhaustive background — the agents will ask for clarification if they need it, and keeping task inputs concise reduces context growth.

Model selection has a bigger impact in agent frameworks than in simple chat. Agent tasks involve following complex instructions across multiple turns, planning and reasoning about tool use, and synthesising outputs from previous steps — all areas where larger models with strong instruction following significantly outperform smaller ones. For production agent workflows, use at least a 7–8B model (Llama 3.1 8B, Qwen 2.5 7B) and prefer 14B+ where your hardware allows. Very small models (3B and below) often lose track of the task across multiple agent turns, producing outputs that drift from the original requirement. Test your specific workflow with whatever model you’re considering rather than assuming benchmark scores predict agent performance — agent capability can vary significantly from single-turn chat performance.

CrewAI vs AutoGen: Choosing the Right Framework

Both frameworks are actively maintained and capable of sophisticated agent workflows with Ollama as the backend. The choice comes down to the nature of your task and your preferred mental model. CrewAI’s role-based approach makes it easier to design workflows where the structure is known upfront — you define the agents, their responsibilities, and the sequence of tasks before running. This works well for content pipelines, report generation, data processing workflows, and any task where the steps are predictable. The crew analogy helps with thinking through the design: who needs to be involved, what is each person responsible for, and in what order do things happen?

AutoGen’s conversational approach handles tasks where the exact steps aren’t fully known in advance — where agents need to make decisions about what to do next based on intermediate results. Code generation with automatic execution and error fixing is the canonical example: the assistant doesn’t know exactly what errors will appear or how many iterations will be needed, so a rigid sequential structure would fail. The conversation loop handles this naturally. AutoGen also has better support for human-in-the-loop workflows where a real person reviews or approves intermediate outputs before the workflow continues — useful for production workflows where agent errors have real consequences. For new users, start with CrewAI for its clearer conceptual model, then explore AutoGen once you’re comfortable with the basics of multi-agent orchestration.

Keeping Agent Workflows Reliable

Agent workflows with local models can be less reliable than with frontier cloud models — local 7–8B models occasionally misfollow instructions, produce malformed tool calls, or generate outputs that don’t match the expected format. A few practices make local agent workflows more robust. Write explicit, specific task descriptions — vague tasks produce variable outputs; specify format, length, and exactly what the output should contain. Add validation steps: a simple check agent or a Python function that validates outputs between tasks can catch failures before they propagate. Set conservative iteration limits and implement graceful failure handling so a workflow that hits an error doesn’t silently produce garbage output. Log intermediate outputs during development so you can see where failures occur and refine the task descriptions or model choice accordingly. These same practices apply to cloud-based agent frameworks but matter more locally because the models have less capacity to infer intent from ambiguous instructions. The good news is that once a local agent workflow is well-designed and tested, it runs reliably, for free, as many times as you need it.