How to Use Ollama with Microsoft Semantic Kernel

by

Microsoft’s Semantic Kernel is an open-source SDK for building AI-powered applications in .NET, Python, and Java. It provides abstractions for LLM services, memory, plugins, and agents that let you build production-grade AI applications without being locked to any specific model provider. Because it supports OpenAI-compatible endpoints, connecting it to Ollama requires minimal configuration — and the combination is compelling for .NET developers who want to run enterprise-grade AI workflows locally.

Why Semantic Kernel + Ollama

Semantic Kernel is designed for enterprise AI application development — structured prompt templates, plugin systems, agent planning, memory integration, and observability. These are the same patterns you’d use building a production AI application targeting the OpenAI API, but Semantic Kernel’s abstractions make it straightforward to swap the underlying LLM provider. Pointing it at Ollama means you can develop and test locally with zero API costs, then switch to Azure OpenAI or the OpenAI API for production deployment with a single configuration change. The code doesn’t change — just the endpoint and credentials.

Python Setup

pip install semantic-kernel
import asyncio
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.ollama import OllamaChatCompletion
from semantic_kernel.connectors.ai.ollama import OllamaTextEmbedding

async def main():
    kernel = Kernel()

    # Add Ollama as the chat service
    kernel.add_service(
        OllamaChatCompletion(
            service_id="ollama-chat",
            ai_model_id="llama3.2",
            host="http://localhost:11434"
        )
    )

    # Simple invocation
    result = await kernel.invoke_prompt(
        "What is the capital of Australia? Answer in one sentence."
    )
    print(result)

asyncio.run(main())

Semantic Kernel has native Ollama connectors (not just the OpenAI-compatible path), which means it integrates more cleanly than tools that only support the compatibility endpoint.

Prompt Templates and Functions

One of Semantic Kernel’s core features is structured prompt templates with variable substitution — reusable prompt functions that can be called with different inputs:

from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.ollama import OllamaChatCompletion
from semantic_kernel.prompt_template import PromptTemplateConfig
import asyncio

async def main():
    kernel = Kernel()
    kernel.add_service(
        OllamaChatCompletion(
            service_id="ollama",
            ai_model_id="llama3.2",
            host="http://localhost:11434"
        )
    )

    # Define a reusable prompt function
    summarise_fn = kernel.add_function(
        function_name="summarise",
        plugin_name="TextUtils",
        prompt="Summarise the following text in {{$max_sentences}} sentences:

{{$input}}",
        template_format="semantic-kernel"
    )

    # Invoke with different inputs
    texts = [
        "Semantic Kernel is an open-source SDK from Microsoft...",
        "Ollama is a tool for running large language models locally..."
    ]

    for text in texts:
        result = await kernel.invoke(
            summarise_fn,
            input=text,
            max_sentences="2"
        )
        print(result)

asyncio.run(main())

Prompt functions are the building blocks of Semantic Kernel applications — reusable, testable, and composable. Each function encapsulates a prompt template, its input variables, and configuration settings. Functions can be called individually or chained together in pipelines.

Plugins: Wrapping Functions for AI Use

Semantic Kernel’s plugin system lets the AI call Python functions as tools — similar to OpenAI function calling but with SK’s orchestration layer handling the plumbing:

from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.ollama import OllamaChatCompletion
from semantic_kernel.functions import kernel_function
from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
import asyncio
import datetime

class TimePlugin:
    @kernel_function(description="Get the current date and time")
    def get_current_time(self) -> str:
        return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

    @kernel_function(description="Calculate days between two dates")
    def days_between(self, date1: str, date2: str) -> str:
        d1 = datetime.datetime.strptime(date1, "%Y-%m-%d")
        d2 = datetime.datetime.strptime(date2, "%Y-%m-%d")
        return str(abs((d2 - d1).days)) + " days"

async def main():
    kernel = Kernel()
    kernel.add_service(
        OllamaChatCompletion(
            service_id="ollama",
            ai_model_id="llama3.1",  # needs tool-use capable model
            host="http://localhost:11434"
        )
    )
    kernel.add_plugin(TimePlugin(), plugin_name="Time")

    settings = kernel.get_prompt_execution_settings_from_service_id("ollama")
    settings.function_choice_behavior = FunctionChoiceBehavior.Auto()

    from semantic_kernel.contents import ChatHistory
    history = ChatHistory()
    history.add_user_message("What's today's date and how many days until January 1st 2027?")

    response = await kernel.get_service("ollama").get_chat_message_content(
        chat_history=history,
        settings=settings,
        kernel=kernel
    )
    print(response)

asyncio.run(main())

The model sees the plugin’s functions as available tools and decides when to call them. Use a tool-capable model (Llama 3.1, Qwen 2.5) for reliable plugin invocation — smaller models sometimes fail to format tool calls correctly.

Figure 1 — Semantic Kernel Architecture with Ollama

Your App SK Kernel Plugins, Memory Prompt templates SK AI Connector OllamaChatCompletion OllamaTextEmbedding Handles API calls Translates SK→Ollama Ollama localhost:11434 Local Models llama3.2 / qwen2.5 SK abstractions insulate your app from the underlying LLM provider

Memory and Embeddings

Semantic Kernel has a memory system for storing and retrieving semantic memories — useful for building RAG applications, personalised assistants, and any application that needs to recall information across sessions. With Ollama providing embeddings, the full memory pipeline runs locally:

from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.ollama import OllamaChatCompletion, OllamaTextEmbedding
from semantic_kernel.memory import SemanticTextMemory
from semantic_kernel.core_plugins.memory_plugin import MemoryPlugin
import asyncio

async def main():
    kernel = Kernel()

    # Chat LLM
    kernel.add_service(OllamaChatCompletion(
        service_id="chat",
        ai_model_id="llama3.2",
        host="http://localhost:11434"
    ))

    # Embedding model for memory
    embedding_service = OllamaTextEmbedding(
        service_id="embedding",
        ai_model_id="nomic-embed-text",
        host="http://localhost:11434"
    )
    kernel.add_service(embedding_service)

    # In-memory vector store
    from semantic_kernel.memory import VolatileMemoryStore
    memory = SemanticTextMemory(
        storage=VolatileMemoryStore(),
        embeddings_generator=embedding_service
    )

    # Add facts to memory
    await memory.save_information(
        collection="company_facts",
        id="founding",
        text="Our company was founded in 2018 in Sydney, Australia."
    )
    await memory.save_information(
        collection="company_facts",
        id="product",
        text="Our main product is a pet food subscription service for dogs and cats."
    )

    # Search memory
    results = await memory.search("company_facts", "When was the company founded?")
    for result in results:
        print(f"Memory: {result.text} (relevance: {result.relevance:.2f})")

asyncio.run(main())

SemanticTextMemory uses embeddings to store facts and retrieve relevant ones by semantic similarity. Combined with a chat LLM, this creates a RAG system entirely within the Semantic Kernel framework — no external vector database required for development, though SK also integrates with Chroma, Qdrant, and other stores for production use.

C# / .NET Setup

Semantic Kernel started as a .NET library and has its strongest features there. For .NET developers, the setup to use Ollama is similar to the Python path:

dotnet add package Microsoft.SemanticKernel
dotnet add package Microsoft.SemanticKernel.Connectors.Ollama --prerelease
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;

var builder = Kernel.CreateBuilder();

builder.AddOllamaChatCompletion(
    modelId: "llama3.2",
    endpoint: new Uri("http://localhost:11434")
);

var kernel = builder.Build();
var chatService = kernel.GetRequiredService<IChatCompletionService>();

var history = new ChatHistory();
history.AddUserMessage("What is Semantic Kernel?");

var response = await chatService.GetChatMessageContentAsync(history);
Console.WriteLine(response.Content);

The .NET connector is marked as preview but functional. For production .NET applications, the OpenAI-compatible path via AddOpenAIChatCompletion with the Ollama base URL is more stable:

builder.AddOpenAIChatCompletion(
    modelId: "llama3.2",
    apiKey: "ollama",
    httpClient: new HttpClient { BaseAddress = new Uri("http://localhost:11434/v1/") }
);

Planners and Agents

Semantic Kernel’s planner feature lets the model decide which sequence of plugin functions to call to accomplish a goal — the kernel provides the tools, and the LLM plans how to use them. This is SK’s agent capability and it works with Ollama, though it requires a capable model:

from semantic_kernel.planners.function_calling_stepwise_planner import (
    FunctionCallingStepwisePlanner,
    FunctionCallingStepwisePlannerOptions
)

planner = FunctionCallingStepwisePlanner(
    service_id="ollama",
    options=FunctionCallingStepwisePlannerOptions(max_iterations=5)
)

# The planner figures out which functions to call
result = await planner.invoke(
    kernel=kernel,
    question="What is today's date and how many days until New Year 2027?"
)
print(result.final_answer)

The planner works best with models specifically trained for tool use — Llama 3.1, Qwen 2.5, or Mistral with function calling training. With weaker models, the planner may fail to correctly select or invoke functions. For production agent applications, test thoroughly with your target model before deploying.

Figure 2 — SK vs LangChain vs LlamaIndex for Ollama Projects

Framework Language Best at Ollama support Choose when… Semantic Kernel.NET / Python / JavaEnterprise patterns, .NETNative + OpenAI compatC#/.NET apps, enterprise LangChainPython / JSBroad integrations, agentsNative + OpenAI compatPython, broad use cases LlamaIndexPythonRAG, document Q&ANative + OpenAI compatDocument RAG focus

When Semantic Kernel Makes Sense

Semantic Kernel is the right framework when you’re building .NET applications with AI features, when you’re in a Microsoft-ecosystem shop where Azure OpenAI is the eventual deployment target, or when you specifically need SK’s structured prompt management and plugin system for an enterprise application. It’s more opinionated and structured than LangChain, which makes it easier to maintain in large codebases but less flexible for quick experimentation. The local Ollama backend during development followed by Azure OpenAI or the OpenAI API in production is a natural fit for teams already using Azure infrastructure — the code is identical, just the service endpoint changes. For Python-first teams building data-heavy AI applications, LangChain or LlamaIndex is usually the more natural fit. For .NET developers, Semantic Kernel is the most mature and well-supported AI SDK available, and the Ollama integration makes local development practical before committing to cloud API costs for the full development cycle.

Observability and Prompt Tracing

One of Semantic Kernel’s production-readiness features is its observability hooks — you can log every prompt, response, token count, and function invocation to trace exactly what’s happening in your AI application. This is more structured than most frameworks and particularly valuable when debugging why an agent made a wrong decision or why a prompt produced unexpected output. With Ollama as the backend, these traces capture the full request-response cycle including the model name and endpoint — useful when you’re comparing outputs from different Ollama models to determine which performs best for your specific tasks. SK’s integration with OpenTelemetry means traces can flow into your existing observability infrastructure (Jaeger, Grafana Tempo, Azure Monitor) without additional tooling. For serious production AI applications, this level of traceability is often a requirement — and it’s one of the reasons to choose Semantic Kernel over lighter-weight frameworks when building something that needs to be maintainable and debuggable long-term.

Practical Getting-Started Recommendation

If you’re new to Semantic Kernel and Ollama together, the fastest path to something useful is the basic chat completion example above — five lines to call Ollama from SK, confirm it works, then add a prompt function for a task you do regularly. From there, add plugins for any external capabilities you need (web search, database queries, API calls), then explore memory if your application needs to recall information across sessions. The SK documentation is comprehensive and the Ollama connector is well-tested — most of what you find in SK’s docs applies directly to the Ollama backend without modification. The key conceptual shift from working directly with Ollama’s API is that SK introduces an abstraction layer — you work with SK’s Kernel, ChatHistory, and Function objects rather than raw HTTP requests to Ollama. This abstraction is what makes the local-to-production switch seamless, and it’s the primary reason to use SK rather than calling Ollama directly for enterprise applications.

Migrating from OpenAI to Ollama in a SK Application

If you have an existing Semantic Kernel application targeting the OpenAI API and want to switch to Ollama for local development, the migration is as simple as changing the service registration. In Python:

# Original OpenAI setup
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion
kernel.add_service(OpenAIChatCompletion(
    service_id="openai",
    ai_model_id="gpt-4o",
    api_key="sk-your-key"
))

# Swap to Ollama — rest of your code unchanged
from semantic_kernel.connectors.ai.ollama import OllamaChatCompletion
kernel.add_service(OllamaChatCompletion(
    service_id="openai",  # keep the same service_id
    ai_model_id="llama3.2",
    host="http://localhost:11434"
))

Because the rest of your SK code interacts with the kernel and the service abstraction rather than the concrete connector, keeping the same service_id means no other code changes are needed. Your prompt functions, plugins, memory operations, and agent code all continue working with the new backend. The output quality and behaviour will differ between GPT-4o and Llama 3.2 — that’s expected — but the code structure is identical. This is the primary architectural benefit of Semantic Kernel over calling LLM APIs directly: provider portability without code changes.

Model Recommendations for Semantic Kernel Tasks

SK’s different features have different model requirements. For basic chat completion and prompt functions, any capable 7B+ model works well — Llama 3.2 3B is sufficient for simple prompt functions, Llama 3.1 8B or Qwen 2.5 7B handles complex prompt chaining reliably. For plugin/function calling with the planner, you need a model specifically trained for tool use — Llama 3.1, Qwen 2.5, or Mistral with function calling. Models without tool-use training produce unreliable tool calls even with SK’s structured prompting. For embeddings and memory, use a dedicated embedding model rather than a general LLM — nomic-embed-text or mxbai-embed-large are both excellent and fast via the OllamaTextEmbedding connector. Having separate models for chat (general or coding) and embeddings (dedicated embedding model) is the right architecture for any SK application using memory, and Ollama’s ability to keep multiple models loaded simultaneously makes this practical without constant model switching overhead.

Leave a Comment