Ollama is designed around interactive single-user inference, but many real workflows need to process dozens, hundreds, or thousands of items through a model — classifying records, generating descriptions, summarising documents, extracting structured data. Getting this right requires understanding how Ollama queues requests, when to use parallel processing, how to structure batch jobs for throughput, and how to handle failures gracefully. This guide covers all of it with working code patterns.
How Ollama Handles Concurrent Requests
By default, Ollama processes one request at a time — a second request that arrives while the first is being processed waits in a queue. This is the OLLAMA_NUM_PARALLEL=1 default. The queue is unlimited in depth, so your requests won’t be rejected — they’ll just wait. For batch workloads running on a single machine with a single user, this serial behaviour is usually fine. For server deployments handling multiple concurrent users, or for exploiting multi-GPU setups, you need to increase parallelism.
export OLLAMA_NUM_PARALLEL=2
export OLLAMA_MAX_LOADED_MODELS=2Increasing OLLAMA_NUM_PARALLEL to 2 means two requests process simultaneously, each with its own KV cache. The memory cost: roughly double the context memory overhead, plus the base model weight stays shared. On a GPU with 24GB VRAM running a 7B model at 8K context, this might go from ~7GB used to ~10GB — still well within headroom. Check with ollama ps during concurrent use to see memory consumption. Going beyond NUM_PARALLEL=2 is rarely beneficial unless you have multiple GPUs, since a single GPU can only execute one kernel at a time.
Serial Batch Processing: The Simple Path
For most batch jobs — processing a list of items one at a time — a simple loop with good error handling is the right approach. Ollama’s serial processing means concurrency doesn’t help throughput anyway on a single GPU, so keep it simple:
import ollama
import json
import time
from pathlib import Path
def process_item(item: dict, model: str = "llama3.2") -> dict:
start = time.time()
try:
response = ollama.chat(
model=model,
messages=[
{
"role": "system",
"content": 'Extract sentiment and topic. Respond with JSON only: {"sentiment": "positive/negative/neutral", "topic": "..."}'
},
{"role": "user", "content": item["text"]}
],
options={"temperature": 0, "num_predict": 100}
)
result_text = response["message"]["content"].strip()
if result_text.startswith("```"):
result_text = result_text.split("\n", 1)[1].rsplit("```", 1)[0].strip()
result = json.loads(result_text)
return {**item, **result, "success": True, "latency_s": round(time.time() - start, 2)}
except Exception as e:
return {**item, "sentiment": None, "topic": None, "success": False, "error": str(e)}
def batch_process(items, model="llama3.2", checkpoint_file="progress.json"):
checkpoint = Path(checkpoint_file)
done = json.loads(checkpoint.read_text()) if checkpoint.exists() else []
done_ids = {r["id"] for r in done}
remaining = [i for i in items if i["id"] not in done_ids]
if done:
print(f"Resuming: {len(done)} done, {len(remaining)} remaining")
for i, item in enumerate(remaining, 1):
result = process_item(item, model)
done.append(result)
if i % 10 == 0:
checkpoint.write_text(json.dumps(done, indent=2))
success_rate = sum(1 for r in done if r["success"]) / len(done)
print(f"Progress: {len(done)}/{len(items)} | Success: {success_rate:.1%}")
checkpoint.write_text(json.dumps(done, indent=2))
return done
items = [
{"id": "1", "text": "The new product launch exceeded all expectations."},
{"id": "2", "text": "Delivery was three days late and the package was damaged."},
{"id": "3", "text": "Average experience, nothing special."},
]
results = batch_process(items)
print(json.dumps([r for r in results], indent=2))The checkpoint pattern is essential for large batches — if you’re processing 5,000 items and something fails at item 3,847, you want to resume rather than restart. Writing progress every 10 items balances disk I/O against data loss risk.
Async Concurrent Processing
When you’ve set OLLAMA_NUM_PARALLEL=2 or higher, async processing keeps the queue fed rather than waiting for each item sequentially:
import asyncio
import ollama
async def process_item_async(item: dict, semaphore: asyncio.Semaphore, model: str) -> dict:
async with semaphore:
client = ollama.AsyncClient()
try:
response = await client.chat(
model=model,
messages=[
{"role": "system", "content": "Summarise in one sentence."},
{"role": "user", "content": item["text"]}
],
options={"temperature": 0.3, "num_predict": 150}
)
return {**item, "summary": response["message"]["content"], "success": True}
except Exception as e:
return {**item, "summary": None, "success": False, "error": str(e)}
async def batch_async(items, max_concurrent=2, model="llama3.2"):
semaphore = asyncio.Semaphore(max_concurrent)
tasks = [process_item_async(item, semaphore, model) for item in items]
return await asyncio.gather(*tasks)
items = [{"id": str(i), "text": f"Document {i} text here."} for i in range(20)]
results = asyncio.run(batch_async(items, max_concurrent=2))
print(f"Done: {sum(1 for r in results if r['success'])}/{len(results)}")The semaphore limits concurrent requests to match your OLLAMA_NUM_PARALLEL setting — sending more requests than Ollama can handle in parallel just creates a queue in Ollama rather than improving throughput. Set max_concurrent to match your OLLAMA_NUM_PARALLEL value.
Figure 1 — Ollama Request Handling: Serial vs Parallel
Throughput vs Latency Trade-offs
For batch workloads, throughput (items per hour) matters more than per-item latency. A few settings specifically help throughput:
Reduce num_ctx: Smaller context means less KV cache memory per request, allowing more efficient processing. For batch items that don’t need long context (short documents, single sentences), set num_ctx to 2048 or even 1024. This also reduces the time Ollama spends on prefill, which is the bottleneck for short prompts.
Reduce num_predict: If your outputs are short (a classification label, a short summary, a JSON object), set num_predict to cap generation at the expected output length. A classification model that generates at most 20 tokens doesn’t need num_predict=512 — setting it to 50 ensures fast completion for short outputs and prevents the model from rambling if it doesn’t stop naturally.
Batch size (num_batch): Controls how many prompt tokens are processed simultaneously during prefill. Increasing num_batch from the default 512 to 1024 or 2048 speeds up the prefill phase for long prompts, at the cost of slightly more peak VRAM during that phase. For workloads with long system prompts or document context, this reduces time-to-first-token.
options={
"num_ctx": 2048, # minimal context for batch items
"num_predict": 50, # cap output for classification
"temperature": 0, # deterministic for consistent extraction
"num_batch": 1024, # faster prefill on capable GPU
}Structured Output for Reliable Parsing
Batch processing pipelines fail most often at the parsing step — the model doesn’t output valid JSON, adds explanatory text before the JSON, or wraps it in markdown code fences. Three strategies address this:
Strict system prompt: Be explicit: “Respond ONLY with valid JSON. No explanation, no markdown, no code fences. Example: {“result”: “value”}”. The word “ONLY” and a concrete example are both important — models that see an example of the exact expected format are much more consistent.
Robust parsing: Strip common noise before parsing:
import json, re
def parse_json_response(text: str) -> dict:
# Remove markdown fences
text = re.sub(r'^`+ *(?:json)?\n?', '', text.strip(), flags=re.MULTILINE)
text = re.sub(r'\n?`+$', '', text.strip(), flags=re.MULTILINE)
# Find first JSON object or array
match = re.search(r'[{\[].*[}\]]', text, re.DOTALL)
if match:
return json.loads(match.group())
raise ValueError(f"No JSON found in: {text[:100]}")Retry on parse failure: Attempt parsing; if it fails, retry with a more forceful prompt that shows the model what it output and asks it to fix it. Two retries cover almost all transient formatting failures.
Monitoring a Running Batch Job
For long batch jobs, real-time monitoring helps you know whether the job is on track. The simplest approach adds progress tracking to your loop:
from tqdm import tqdm # pip install tqdm
import statistics
latencies = []
results = []
for item in tqdm(items, desc="Processing", unit="item"):
result = process_item(item)
results.append(result)
latencies.append(result["latency_s"])
if len(latencies) >= 5:
avg = statistics.mean(latencies[-20:]) # rolling average
eta_s = avg * (len(items) - len(results))
tqdm.write(f"Avg: {avg:.1f}s/item | ETA: {eta_s/60:.0f}min")The rolling average over the last 20 items gives a more accurate ETA than a cumulative average, since model speed can vary (first items may be slower as the model warms up, or later items may be longer). For multi-day batch jobs, write ETA and progress to a log file so you can check status without leaving the terminal open.
Figure 2 — Batch Job Settings by Scale
Estimating Batch Job Duration
Before starting a large batch job, estimate the runtime so you can plan accordingly. The formula: total items × average seconds per item = total seconds. Average seconds per item depends on your model, hardware, and output length. A rough baseline for a 7B model on a mid-range GPU (e.g. RTX 4070): 3–8 seconds per short item (100-token output), 10–25 seconds per medium item (300-token output). Run a sample of 20 items with your actual prompt and model, measure the average latency, then multiply by your total item count. Add 20% for overhead (retries, checkpointing, script startup). For 10,000 items at 5 seconds each: 50,000 seconds = ~14 hours. That’s a reasonable overnight run on a dedicated machine. If that’s too long, switch to a smaller model (3B runs 2–3x faster with lower quality) or a smaller quantization. The quality-speed trade-off for batch classification tasks often favours smaller models more than for interactive use — the marginal quality difference between 3B and 8B for binary sentiment classification is much smaller than for open-ended generation.
When to Use a Smaller Model for Batch Work
Interactive use justifies larger models because each conversation involves a human waiting. Batch processing is different — you’re not watching each item generate, just waiting for the job to finish. This changes the quality-speed calculation. For tasks with narrow, well-defined outputs (sentiment classification: positive/negative/neutral; topic extraction: one of N predefined categories; short data extraction from structured text), a 3B model often achieves 90%+ of the quality of an 8B model at 2–3x the speed. Running 10,000 items at 3B instead of 8B could reduce an 8-hour job to 3 hours at minimal quality cost. Test this: run 100 items through both models, compare the outputs on a sample, and make a data-driven decision rather than defaulting to the largest model out of habit. The right model for batch work is the smallest model that meets your quality threshold — not the best model you have.
Multi-Model Pipelines
Some batch workflows benefit from using different models for different steps. A common pattern: a small fast model for initial filtering or classification, then a larger model for detailed processing of the items that pass the filter. This reduces the total work sent to the expensive model while keeping overall quality high. For example, processing customer feedback at scale: first pass all 10,000 items through a 3B model to classify as positive/negative/neutral and flag the negatives (quick, cheap); second pass only the 2,000 negative items through an 8B model for detailed issue categorisation and suggested response drafting (slower, higher quality where it matters). The 3B pass costs roughly 5,000 seconds; the 8B pass costs roughly 8,000 seconds — total ~13,000 seconds. Running all 10,000 through the 8B model would take ~25,000 seconds. The two-model pipeline is 2x faster and produces better results on the items that matter most. Implement this by storing intermediate results from the first model pass and loading them as the input set for the second pass — the checkpoint pattern above handles this naturally.
Scaling Beyond a Single Machine
For truly large batch jobs that exceed what a single machine can handle in reasonable time, distributing across multiple Ollama instances is straightforward — each machine runs Ollama, and your Python script distributes work across them using round-robin or a queue. The simplest implementation uses multiple Ollama clients pointing at different hosts and distributes items across them with asyncio. A job queue like Celery or RQ can also coordinate work distribution if you have an existing queue infrastructure. The architecture: one coordinator script that reads items from a database or file, distributes them to worker scripts running Ollama on different machines, and collects results back to a shared database. Each worker is independent — no inter-process communication needed — which makes this architecture easy to scale and debug. The primary constraint is data movement: if items are large documents, network transfer time between the coordinator and workers can become significant. For most text processing tasks, items are small enough that this isn’t a bottleneck.