Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1. This is one of its most useful and underappreciated features — any application, library, or tool built for the OpenAI API can be pointed at your local Ollama instance with two changes: the base URL and the model name. No code changes to the actual logic, no switching libraries, no learning a new API. If you’ve built something with the OpenAI Python SDK and want to run it locally, this is how you do it in under a minute.
How the Compatibility Layer Works
Ollama’s OpenAI-compatible endpoint implements the subset of the OpenAI API that most applications actually use: /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models. The request and response schemas match OpenAI’s exactly, including streaming with server-sent events. The only differences: you use your Ollama model name (e.g. llama3.2) instead of an OpenAI model name (e.g. gpt-4o), and the API key can be any non-empty string since Ollama doesn’t authenticate.
This means the compatibility isn’t a translation layer that might break on edge cases — it’s a direct implementation of the same API schema. Code that correctly calls the OpenAI API will work with the Ollama endpoint without modification to the API call logic.
Switching the OpenAI Python SDK to Ollama
# Original OpenAI code
from openai import OpenAI
client = OpenAI(api_key="sk-...")
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}]
)
# Same code pointing at Ollama — two changes only
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1",
api_key="ollama" # any non-empty string
)
response = client.chat.completions.create(
model="llama3.2", # your local model name
messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)For applications with many OpenAI calls, make base_url and model configurable via environment variables so you can switch between local and cloud with a single env file change rather than editing code:
import os
from openai import OpenAI
client = OpenAI(
base_url=os.getenv("OPENAI_BASE_URL", "http://localhost:11434/v1"),
api_key=os.getenv("OPENAI_API_KEY", "ollama")
)
model = os.getenv("LLM_MODEL", "llama3.2")
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Hello"}]
)Now switching to the real OpenAI API is just setting OPENAI_BASE_URL=https://api.openai.com/v1, OPENAI_API_KEY=sk-yourkey, and LLM_MODEL=gpt-4o in your environment.
Streaming with the OpenAI SDK
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
stream = client.chat.completions.create(
model="llama3.2",
messages=[{"role": "user", "content": "Explain gradient descent step by step."}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()Streaming works identically to the OpenAI API — server-sent events, same chunk structure, same delta content field. Any code that handles OpenAI streaming works with Ollama streaming without changes.
Embeddings via the OpenAI-Compatible Endpoint
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.embeddings.create(
model="nomic-embed-text",
input="The transformer architecture uses self-attention mechanisms."
)
embedding = response.data[0].embedding
print(f"Embedding dimensions: {len(embedding)}")The embeddings endpoint is particularly useful for migrating applications that use OpenAI’s text-embedding-3-small or text-embedding-ada-002 to local embeddings. Replace the model name with nomic-embed-text or mxbai-embed-large and the output is a compatible embedding vector — same format, different model, zero API cost.
Figure 1 — OpenAI API → Ollama: What Changes, What Stays the Same
Frameworks and Tools That Work via OpenAI Compatibility
The list of tools that work with Ollama via the OpenAI-compatible endpoint is extensive. Here are the most commonly used ones:
LangChain: Use ChatOpenAI pointed at the Ollama base URL. Every LangChain component that takes an LLM — chains, agents, retrievers, memory — works with Ollama this way without needing the dedicated Ollama LangChain integration.
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
base_url="http://localhost:11434/v1",
api_key="ollama",
model="llama3.2"
)LlamaIndex: Same pattern — use the OpenAI LLM class with the Ollama endpoint. Though LlamaIndex also has a dedicated Ollama integration that’s generally cleaner.
from llama_index.llms.openai import OpenAI as LlamaOpenAI
llm = LlamaOpenAI(
api_base="http://localhost:11434/v1",
api_key="ollama",
model="llama3.2"
)DSPy: The Stanford prompt optimisation framework works with any OpenAI-compatible endpoint. Point it at Ollama to run DSPy programs with local models.
AutoGen: Microsoft’s agent framework supports OpenAI-compatible endpoints. Configure the config_list with the Ollama base URL to run multi-agent conversations locally.
CrewAI: The agent orchestration framework supports OpenAI-compatible LLMs. Use Ollama as the backend for private, cost-free agent workflows.
Any REST client or cURL: The endpoint is plain HTTP — useful for testing:
curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "llama3.2",
"messages": [{"role": "user", "content": "What is 2+2?"}]
}'What the Compatibility Layer Doesn’t Support
The OpenAI-compatible endpoint doesn’t implement the full OpenAI API surface — only the parts most applications use. Features not supported include: fine-tuning endpoints, moderation endpoint, assistants API (threads, runs, files), audio transcription and generation, image generation (DALL-E), function calling in the structured tool format (though Ollama’s native chat API supports tool calling), batch API, and usage-based billing fields in responses. For applications that use these features, the compatibility layer won’t help — you’ll need to use Ollama’s native API or the dedicated Python library for those use cases. The most commonly hit limitation is the assistants API (threads and runs), which some applications use for conversation management. If your application depends on assistants API features, you’ll need to refactor to use the chat completions endpoint with your own conversation history management instead.
Using Ollama as a Local Development Backend
One of the best use cases for the OpenAI compatibility layer is local development of applications intended to run against OpenAI in production. During development, point your app at Ollama — zero API costs, no rate limits, no latency from network calls, and your data stays local. Before deploying, switch to the real OpenAI API by changing two environment variables. This workflow makes development significantly cheaper and faster: you can iterate on prompts, test edge cases, and build the application without spending on API calls, then validate against the real API before shipping. The model behaviour between local Llama/Mistral and cloud GPT-4 will differ, so final testing should always happen against the production API — but the bulk of development can happen locally at zero cost. Teams building multiple AI applications benefit most from this pattern: a single capable Ollama instance on a shared development machine serves multiple developers simultaneously, eliminating the need for each developer to have their own API key or manage their own usage budget during development.
Figure 2 — OpenAI-Compatible Tools Working with Ollama
Practical Migration Guide: OpenAI → Ollama
If you have an existing application using the OpenAI API and want to migrate it to run locally with Ollama, here’s the practical sequence. First, audit which OpenAI API features you actually use — chat completions and embeddings migrate easily; assistants API and fine-tuning don’t. Second, identify the models you use and find appropriate Ollama equivalents: GPT-3.5-turbo maps well to Llama 3.2 8B or Mistral 7B; GPT-4-class tasks map to Qwen 2.5 72B or Llama 3.1 70B; embedding models map to nomic-embed-text or mxbai-embed-large. Third, make base URL and model name configurable via environment variables rather than hardcoded. Fourth, test with the Ollama endpoint in development and compare outputs — pay particular attention to cases where your application expects a specific output format or follows up on previous responses, since local models may behave differently from GPT-4 on edge cases. Fifth, adjust system prompts if needed — local models sometimes need more explicit instructions to match the behaviour you got from OpenAI models, particularly around output formatting and instruction following precision. The migration is usually smoother than expected for chat completion-based applications, and the zero-cost, zero-latency local inference more than justifies the time invested in getting it right.
JavaScript and TypeScript: Using the OpenAI SDK with Ollama
The OpenAI JavaScript/TypeScript SDK works with the Ollama endpoint exactly the same way as the Python SDK — change the base URL and API key, swap the model name:
import OpenAI from 'openai';
const client = new OpenAI({
baseURL: 'http://localhost:11434/v1',
apiKey: 'ollama',
});
const response = await client.chat.completions.create({
model: 'llama3.2',
messages: [{ role: 'user', content: 'Hello from TypeScript!' }],
});
console.log(response.choices[0].message.content);Streaming in JavaScript:
const stream = await client.chat.completions.create({
model: 'llama3.2',
messages: [{ role: 'user', content: 'Count to ten slowly.' }],
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || '');
}
console.log();This works in Node.js, Deno, and any JavaScript environment that can make HTTP requests. For frontend JavaScript (browser-based apps), calling localhost:11434 from a browser works fine for local development, but requires CORS headers if the browser and Ollama are on different origins — set OLLAMA_ORIGINS=* (or a specific origin) to enable this. Browser-based AI apps that call a local Ollama instance directly are an interesting category — private, zero-cost, entirely client-side once the page loads — and the OpenAI JS SDK makes building them as straightforward as any server-side integration.
Cost and Rate Limit Comparison
The economics of switching to Ollama via the OpenAI compatibility layer deserve explicit mention. The OpenAI API costs money per token — GPT-4o at $2.50 per million input tokens and $10 per million output tokens, GPT-3.5-turbo at $0.50 and $1.50 respectively. For development, testing, and batch processing workflows, these costs add up quickly: a batch job processing 10,000 documents averaging 500 tokens of output each is $50 with GPT-3.5-turbo or $100 with GPT-4o. The same job with Ollama on your own hardware costs nothing in API fees — just the electricity to run your GPU for the inference time. Rate limits are also eliminated: OpenAI’s free and paid tiers impose requests-per-minute and tokens-per-minute limits that slow batch processing and can interrupt production workflows. Ollama has no rate limits — it processes as fast as your hardware allows, for as many requests as you send. For individuals and teams doing significant AI workloads, the switch to local inference via the OpenAI compatibility layer pays for itself in API cost savings quickly, often within weeks for heavy users. The hardware investment — a machine capable of running useful models — is the main upfront cost, but for teams already running capable workstations or servers, the marginal cost is zero.