TorchServe is PyTorch’s official model serving framework, designed to take a trained PyTorch model from a .pt or .pth file to a production HTTP endpoint with minimal configuration. It handles the serving infrastructure — request batching, multi-model management, logging, metrics, and health checks — so you can focus on your model and handler logic rather than building a serving layer from scratch. This guide covers how to package a model, configure TorchServe, and run inference at production quality.
How TorchServe Works
TorchServe organises model serving around three concepts. A model archive (.mar file) bundles everything needed to serve a model: the model weights, a handler script that defines how to preprocess inputs and postprocess outputs, and any additional files the model needs. The model store is a directory where .mar files live — TorchServe loads models from here on startup or on demand. The management API lets you register, unregister, and scale models at runtime without restarting the server.
Requests arrive at TorchServe’s inference endpoint, get routed to the appropriate model worker, pass through the handler’s preprocess function, run inference, pass through postprocess, and return a response. The handler is where your model-specific logic lives; everything else is handled by TorchServe’s infrastructure. This separation makes it straightforward to update model logic without touching serving configuration.
Creating a Model Archive
The torch-model-archiver tool packages a model into a .mar file. You provide the model name, version, the serialised model file (or TorchScript file), and the handler script. Optional additions include extra files like label mappings, configuration files, or vocabulary files the handler needs at inference time.
The handler script is the most important part of the archive. A handler class inherits from BaseHandler and implements three methods: preprocess (converts raw request data into model inputs), inference (runs the model), and postprocess (converts model outputs into the response format). TorchServe provides default handlers for common tasks — image classification, object detection, text classification — that you can use as-is or extend. For custom models, writing a handler from scratch takes 30–60 minutes and is well-documented.
Figure 1 — TorchServe: request flow from HTTP to model and back
Configuring TorchServe
TorchServe’s configuration lives in a config.properties file. The most important settings for production:
- inference_address and management_address — the host and port for each API. Keep management on a private interface not exposed to the public internet.
- number_of_netty_threads — threads for the HTTP server layer. Set to the number of CPU cores.
- default_workers_per_model — how many worker processes to spawn per model. Each worker loads the full model into memory; more workers increase throughput at the cost of memory.
- batch_size and max_batch_delay — batch size controls how many requests are grouped for a single inference call; max_batch_delay is how long TorchServe waits to fill a batch before processing it. Tuning these together is the primary lever for throughput optimisation.
- model_store — path to the directory containing .mar files.
Dynamic Batching
Dynamic batching is one of TorchServe’s most valuable features for GPU utilisation. Without batching, each incoming request triggers a separate forward pass — GPU utilisation is low because the GPU waits for the next request between passes. With batching, TorchServe accumulates requests for up to max_batch_delay milliseconds (typically 50–200ms), then runs a single forward pass with a batch of inputs. This can increase GPU utilisation from 10–20% to 70–90% on variable-rate workloads, dramatically improving throughput without increasing hardware costs.
The tradeoff is latency: a request that arrives just after a batch is dispatched must wait for the next batch window. For latency-sensitive applications where P99 latency matters more than throughput, a smaller batch size and shorter delay (or no batching) is better. For throughput-oriented applications — batch processing, async tasks — larger batches and longer delays produce better economics.
Multi-Model Serving
TorchServe supports serving multiple models simultaneously from a single instance, with each model getting its own worker pool and configuration. Register models at startup by listing them in config.properties, or register them dynamically through the management API while the server is running. This makes it practical to serve a family of related models — a base model and fine-tuned variants, or models for different tasks — from a single serving deployment, reducing infrastructure costs compared to separate deployments per model.
Model versioning is built in: you can register multiple versions of the same model under the same name and route traffic between them. This supports gradual rollouts — send 10% of traffic to the new version, monitor metrics, then increase the percentage — without client-side changes.
Metrics and Monitoring
TorchServe emits metrics in multiple formats: log files, Prometheus-compatible metrics endpoint, and CloudWatch (on AWS). The built-in metrics cover request count, latency percentiles (P50, P90, P99), queue size, worker utilisation, and model load time. These metrics are sufficient for basic production monitoring; add custom metrics from the handler using TorchServe’s metrics API when you need model-specific instrumentation like prediction confidence distributions or preprocessing latency.
Deploying on Kubernetes
TorchServe runs well in containers and is commonly deployed on Kubernetes for production scale. The official TorchServe Docker images (available on Docker Hub) include all dependencies; build from them with your .mar files copied into the model store directory. For Kubernetes deployments, expose the inference endpoint as a ClusterIP service and the management endpoint as an internal service not exposed outside the cluster. Use a Horizontal Pod Autoscaler keyed on GPU utilisation or request queue depth to scale worker pods in response to traffic. The model store can be a shared PersistentVolume or an object store (S3, GCS) — TorchServe supports loading .mar files from S3 URLs directly, which simplifies model updates without rebuilding container images.
TorchServe removes most of the infrastructure work from deploying PyTorch models to production. The handler abstraction keeps model-specific logic clean and testable; the management API makes runtime updates straightforward; dynamic batching provides GPU efficiency without custom implementation. For teams deploying PyTorch models at scale, it is the natural choice over hand-built FastAPI wrappers or generic serving frameworks that lack PyTorch-specific optimisations.
GPU Memory Management
TorchServe manages GPU memory allocation across multiple model workers automatically, but understanding the model is critical for sizing your deployment correctly. Each worker process loads a complete copy of the model into GPU memory. For a model that requires 4GB of GPU memory, two workers require 8GB, three require 12GB, and so on. Over-provisioning workers wastes GPU memory that could serve a larger model or more models; under-provisioning leaves GPU compute idle between requests. The right worker count for a GPU deployment is typically floor(available_GPU_memory / model_memory_per_worker), leaving a small buffer for activation memory during inference. For multi-GPU setups, TorchServe can be configured to shard workers across GPUs, but single-model GPU sharding requires setting up the model with torch.distributed before archiving rather than using TorchServe’s built-in worker management.
Updating Models Without Downtime
TorchServe’s management API supports zero-downtime model updates through versioning. Register the new model version alongside the existing one, gradually shift traffic to the new version using the default_version and version_status endpoints, and unregister the old version once traffic has fully migrated. The key operational detail: the new .mar file must be in the model store before registering the new version — TorchServe loads it at registration time, not at traffic shift time. For teams with frequent model updates (daily retraining pipelines, A/B testing new model variants), scripting this versioned update process into a deployment workflow ensures reliable updates without manual intervention or service restarts.
Alternatives to TorchServe
TorchServe is the natural choice for PyTorch model serving but not the only option. Triton Inference Server (NVIDIA) supports PyTorch, TensorFlow, ONNX, and TensorRT in a single serving layer — the better choice for deployments with mixed-framework model portfolios or when TensorRT optimisation is a priority. BentoML is a Python-native serving framework with simpler packaging and a cloud deployment platform, better suited for teams that want less configuration overhead at the cost of some performance headroom. FastAPI with manual inference endpoints remains common for simple deployments where a custom serving layer adds no value. TorchServe’s strengths — production-grade features, PyTorch-specific optimisations, and official support — make it the default for serious PyTorch production deployments, but the alternatives are worth knowing when the use case does not align with TorchServe’s design assumptions.
TorchServe continues to evolve alongside PyTorch itself — new releases add support for newer PyTorch features, improved quantisation integration, and better cloud-native tooling. Checking the TorchServe GitHub release notes when upgrading PyTorch versions is worthwhile, as handler APIs occasionally change between major versions and the release notes document any migration steps required. For teams with existing TorchServe deployments, the upgrade path is generally smooth when staying within a major version; cross-major-version upgrades benefit from testing in a staging environment before rolling to production, particularly for custom handlers that depend on internal TorchServe APIs.