Ollama Concurrent Requests: How to Configure Parallel Inference

Ollama handles concurrent requests but its default configuration is deliberately conservative — optimised for a single interactive user rather than multi-user production throughput. Understanding how Ollama manages parallelism, and how to tune it with OLLAMA_NUM_PARALLEL and related settings, is essential for anyone building applications that receive more than one request at a time.

How Ollama Handles Parallel Requests by Default

When multiple requests arrive at Ollama simultaneously, the default behaviour depends on whether they target the same model or different models. For requests targeting the same model, Ollama queues them and processes them one at a time unless OLLAMA_NUM_PARALLEL is set above 1. For requests targeting different models that are both loaded in memory, Ollama processes them in parallel — each model handles its own request concurrently. The default OLLAMA_NUM_PARALLEL is 1 on most configurations, meaning a second request to the same model waits until the first completes before beginning. This is suitable for a personal assistant setup but becomes a bottleneck the moment two users or two application threads make simultaneous requests.

OLLAMA_NUM_PARALLEL

OLLAMA_NUM_PARALLEL controls how many requests a single loaded model processes simultaneously. Setting it to 2 allows two concurrent generations from the same model; setting it to 4 allows four. Each parallel request requires additional KV cache memory proportional to the context length, so raising OLLAMA_NUM_PARALLEL increases memory consumption. The practical limit depends on your hardware: on a machine with 24GB unified memory running a 14B Q4 model (~10GB), you might set OLLAMA_NUM_PARALLEL to 2 or 3 and still have comfortable headroom. On a machine running a 70B model that already occupies most of the available memory, OLLAMA_NUM_PARALLEL of 1 may be the only option. Set it as an environment variable before starting Ollama: OLLAMA_NUM_PARALLEL=4 ollama serve, or as a system variable on Windows, or in a systemd service override on Linux.

OLLAMA_MAX_LOADED_MODELS

OLLAMA_MAX_LOADED_MODELS controls how many distinct models Ollama keeps loaded in memory simultaneously. The default is 1 for GPU inference and 3 for CPU inference. With the default of 1, switching between models causes the current model to unload and the new one to load, adding cold start latency. Set it to 2 or 3 if you regularly switch between models or if your application routes requests to different models dynamically. Each loaded model occupies its full memory footprint, so this setting only makes sense if your hardware has enough RAM or VRAM to hold multiple models simultaneously. Set it alongside OLLAMA_NUM_PARALLEL for a complete concurrency configuration.

OLLAMA_MAX_QUEUE

OLLAMA_MAX_QUEUE controls how many requests Ollama will queue while waiting for a processing slot. The default is 512. When the queue fills (more than 512 requests are waiting), Ollama returns a 503 error to new incoming requests. For most applications this default is more than sufficient. The main reason to lower it is to fail fast rather than let requests pile up during traffic spikes — returning a 503 immediately is often better than making users wait several minutes in a queue. The main reason to raise it is for batch processing applications where a large number of tasks are submitted simultaneously and you want them all to eventually complete rather than be rejected.

Multi-Model Parallel Requests

If your application routes requests to two different models, setting OLLAMA_MAX_LOADED_MODELS to 2 (or more) allows both to be in memory simultaneously and served in parallel. This is particularly useful for applications that use one model for fast autocomplete (1.5B or 3B) and another for detailed chat responses (7B or 14B) — both stay loaded, requests to either model are handled without unload/reload delays, and OLLAMA_NUM_PARALLEL can be set independently for each request stream. The combined memory footprint of both models must fit within your available RAM or VRAM.

When to Switch from Ollama to vLLM

Ollama’s parallelism improvements are meaningful for small-scale multi-user deployments, but its architecture is not designed for high-throughput serving. At 10+ concurrent users, vLLM’s PagedAttention and continuous batching produce dramatically higher aggregate token throughput than Ollama at the same hardware. The practical threshold: for a personal or small-team deployment with 1–5 concurrent users, Ollama with tuned OLLAMA_NUM_PARALLEL is sufficient and much simpler to operate. For an API serving dozens of concurrent users, vLLM is the right choice despite its significantly higher operational complexity. The key signal is whether users are experiencing noticeable queuing delay — if they are, and OLLAMA_NUM_PARALLEL tuning is not resolving it, switching to vLLM is the appropriate next step.

Frequently Asked Questions

How do I check how many requests Ollama is currently processing?

ollama ps shows loaded models but not live request counts. For request monitoring, set up Prometheus scraping on Ollama’s metrics endpoint (enabled with OLLAMA_PROMETHEUS_PORT) and watch the request queue and duration metrics.

Does OLLAMA_NUM_PARALLEL affect output quality?

No. Each parallel request processes independently with its own context and KV cache. The outputs are identical to sequential processing — parallelism affects throughput, not generation quality.

What happens when the queue is full?

Ollama returns HTTP 503 Service Unavailable to new requests when OLLAMA_MAX_QUEUE is reached. Your application should implement retry logic with exponential backoff to handle these gracefully.

Last updated: October 2026

Benchmarking Your Ollama Concurrency Setup

Before tuning OLLAMA_NUM_PARALLEL, measure your baseline. A simple benchmark: send N concurrent requests using Python’s asyncio and aiohttp, measuring total time and per-request latency. Start with OLLAMA_NUM_PARALLEL=1 (default) and record baseline latency. Then set it to 2 and repeat. If total throughput (total tokens generated / total time) increases without per-request latency becoming unacceptably high, increase further. The key metric is not aggregate throughput but the latency experienced by the slowest request — your users care about when their request completes, not the average. At OLLAMA_NUM_PARALLEL=4 on a machine with enough memory, you will typically see aggregate throughput increase by 2–3x over the default while per-request latency increases only modestly, because the batched processing shares prefill computation. Benchmark on your target hardware with representative prompt lengths before committing to a production configuration.

Open WebUI and Multiple Users

If you are using Open WebUI as a frontend for Ollama and want to support multiple simultaneous users, OLLAMA_NUM_PARALLEL is the most important setting to configure. Each Open WebUI user session that generates a response sends an independent request to Ollama. With the default OLLAMA_NUM_PARALLEL=1, a second user’s message waits until the first user’s response completes. Set OLLAMA_NUM_PARALLEL to the number of simultaneous users you want to support comfortably — typically 2–4 for a home or small office deployment. Set OLLAMA_KEEP_ALIVE to -1 alongside this so the model stays loaded rather than unloading between sessions, eliminating cold start delays for users who request a response after a period of inactivity.

Concurrent request handling is one of the most practically impactful Ollama configurations for anyone building shared applications on top of local models. The default settings are conservative by design — appropriate for a personal interactive assistant. A few environment variables, set once and verified with a simple benchmark, transform Ollama into a capable backend for small-team multi-user deployments. For larger scale, that benchmark data also makes the case clearly when and why to move to vLLM.

Persistent Configuration Across Restarts

Environment variables set in the terminal are lost when the terminal closes. For persistent OLLAMA_NUM_PARALLEL configuration that survives reboots and Ollama restarts, set it at the system level. On Linux with systemd, use sudo systemctl edit ollama.service and add an Environment line with your OLLAMA_NUM_PARALLEL value, then reload and restart the service. On macOS, add an export line to your .zshrc or .bash_profile, or set it as a launchctl environment variable to apply globally before Ollama launches. On Windows, set it as a System environment variable in Advanced System Settings so it is available to the Ollama process regardless of which terminal or application starts it. Verify the setting took effect after restarting Ollama by checking that concurrent requests are processed in parallel rather than sequentially — send two requests simultaneously from separate terminals and confirm both begin generating immediately rather than one waiting for the other to complete.

Monitoring Queue Depth in Production

For production Ollama deployments serving real users, monitoring queue depth and request latency tells you when OLLAMA_NUM_PARALLEL needs adjusting. Enable Ollama’s Prometheus metrics endpoint by setting OLLAMA_PROMETHEUS_PORT to a port number (e.g. 9090) in your environment. This exposes metrics including ollama_request_duration_seconds (latency histogram), ollama_requests_total (request count by status), and queue depth. Scrape these with Prometheus and visualise in Grafana alongside GPU utilisation and memory usage for a complete picture of your inference server’s health. A queue depth that regularly climbs above 2–3 during peak periods is a clear signal to increase OLLAMA_NUM_PARALLEL (if memory allows) or to consider scaling to additional Ollama instances behind a load balancer. A sustained queue depth of 0 with low GPU utilisation means you have headroom and could reduce OLLAMA_NUM_PARALLEL to free memory for a larger model.