ONNX Runtime is a high-performance inference engine for machine learning models in the ONNX (Open Neural Network Exchange) format. It accelerates model inference across CPUs, GPUs, and specialised hardware, with optimisations that often produce 2–10x speedups over running the same model in its original framework. For teams deploying ML models in production — whether on edge devices, cloud servers, or mobile applications — ONNX Runtime is one of the most practical tools for closing the gap between research-quality models and production-viable latency.
What ONNX Runtime Does
ONNX Runtime takes a model exported to ONNX format and runs inference on it efficiently. It applies a set of graph optimisations — operator fusion, constant folding, memory planning — that reduce the computational work required for each inference pass. It also provides execution providers that route computation to specialised hardware: CUDA for NVIDIA GPUs, TensorRT for maximum GPU throughput, CoreML for Apple Silicon, DirectML for Windows devices, and others. The same ONNX model runs on any supported hardware through the appropriate execution provider, without requiring framework-specific deployment code for each target.
This hardware abstraction is one of ONNX Runtime’s most practical benefits. A model trained in PyTorch can be exported to ONNX once and deployed on Intel CPUs in the cloud, Apple Silicon on MacBooks, and NVIDIA GPUs in data centres — all using the same ONNX Runtime API with different execution providers configured for each environment.
Exporting Models to ONNX
The first step in using ONNX Runtime is exporting your model from its training framework to ONNX format. PyTorch’s torch.onnx.export() handles this for most models. The export requires a sample input that defines the input shape the exported model expects. Key export decisions:
- Dynamic axes — specify which dimensions can vary between inference calls (typically batch size and sequence length for NLP models). Without dynamic axes, the model only accepts inputs with exactly the shape used during export.
- Opset version — the ONNX operator set version. Use opset 17 or 18 for modern models; older opsets may not support all operators used by recent architectures.
- Model verification — run
onnx.checker.check_model()on the exported model to verify it is valid before attempting inference.
After export, verify that ONNX Runtime produces the same outputs as the original PyTorch model on a test input. Numerical differences up to floating-point precision (1e-5) are normal; larger differences indicate an export problem worth investigating before deployment.
Figure 1 — ONNX Runtime: model path from training to deployment
Running Inference with ONNX Runtime
Inference with ONNX Runtime follows a straightforward pattern. Create an InferenceSession with the model path and optional session options. Inspect the session’s input and output names. Prepare input data as NumPy arrays matching the model’s expected input shape and dtype. Call session.run() with the output names and input dictionary, and receive the results as NumPy arrays.
Session creation is the expensive step — it loads the model, applies graph optimisations, and compiles any device-specific kernels. Create the session once at application startup and reuse it for all inference requests. Session creation taking 2–10 seconds is normal for large models; individual inference calls after session creation are typically milliseconds to low seconds depending on model size and hardware.
Execution Providers and Hardware Acceleration
Execution providers are the mechanism by which ONNX Runtime routes computation to specific hardware. Specify them as an ordered list when creating the session — ONNX Runtime uses the first provider that is available and supports the model’s operators, falling back to subsequent providers if the first is unavailable. The standard fallback chain for most deployments is CUDAExecutionProvider (NVIDIA GPU) falling back to CPUExecutionProvider, which handles any hardware configuration from GPU-equipped servers to CPU-only instances without code changes.
For maximum performance on NVIDIA hardware, TensorRTExecutionProvider uses NVIDIA’s TensorRT engine, which applies further optimisations beyond ONNX Runtime’s standard CUDA provider. TensorRT requires an additional compilation step the first time a model runs on a given GPU, which can take minutes but only happens once — subsequent runs use the compiled engine. The performance improvement over the standard CUDA provider is typically 20–50% for CNN-based models and varies for transformer architectures.
Quantisation for Faster CPU Inference
Quantisation converts model weights from 32-bit floats to 8-bit integers (INT8), reducing model size by 4x and increasing CPU inference speed by 2–4x at a small cost to accuracy. ONNX Runtime’s quantisation tools support both static quantisation (which requires a calibration dataset to determine optimal quantisation parameters) and dynamic quantisation (which quantises at runtime without calibration, with slightly lower performance gains but no calibration overhead).
Dynamic quantisation is the practical first step — apply it to a model and measure the latency improvement and accuracy impact on your evaluation set before investing in static quantisation’s calibration process. For transformer-based NLP models, dynamic quantisation of the linear layers typically yields 2–3x CPU speedup with less than 1% accuracy degradation on most benchmarks. For production deployments that need to run on CPU without GPU hardware, quantisation is often the difference between viable and unviable latency.
Optimising Session Options
Several session options significantly affect inference performance and are worth configuring explicitly rather than relying on defaults. The graph optimisation level controls how aggressively ONNX Runtime optimises the model graph — ORT_ENABLE_ALL applies all optimisations and is the right setting for production. The intra-op thread count controls how many threads ONNX Runtime uses for parallelisable operations within a single inference call — setting this to the number of physical CPU cores (not logical cores with hyperthreading) typically produces the best throughput for batch workloads. For latency-sensitive single-request serving, setting intra-op threads to 1 and scaling horizontally across multiple processes often outperforms a single multi-threaded process.
Benchmarking and Profiling
ONNX Runtime includes a built-in profiling tool that records the execution time of each operator in the model graph, helping identify bottlenecks. Enable profiling by setting a profile file prefix in session options — ONNX Runtime writes a JSON profile file after the session completes that you can analyse to find which operators consume the most time. This is the right first step when ONNX Runtime inference is slower than expected — the profile often reveals that a single operator is responsible for most of the latency and either a different execution provider or a model architecture change would address it.
ONNX Runtime is the practical standard for ML model deployment across heterogeneous hardware. The combination of framework-agnostic model format, hardware-specific execution providers, and graph optimisations makes it easier to go from a trained PyTorch model to a production inference service than any alternative approach. The path — export to ONNX, verify outputs, configure execution providers, tune session options — takes an afternoon for a new model and produces deployment-ready inference code that runs efficiently across every target hardware configuration you need to support.
Serving ONNX Models in Production
For production deployments, ONNX Runtime inference is typically wrapped in a serving layer that handles HTTP requests, batching, and scaling. ONNX Runtime Server (a standalone binary) provides a gRPC and REST serving interface for ONNX models without requiring custom serving code. For Python-based services, wrapping an InferenceSession in a FastAPI endpoint is a practical approach that gives full control over request handling, preprocessing, and postprocessing. The serving architecture should create the InferenceSession at application startup (not per request), use a thread pool or async executor to handle concurrent requests without blocking, and implement health checks that verify the session is loaded and responding before routing production traffic to the instance. For batch serving where latency is less critical than throughput, dynamic batching — accumulating requests for a short time window before running a single batched inference — significantly improves GPU utilisation at the cost of slightly higher individual request latency.
Common Export Problems and Fixes
Several model architectures cause predictable export problems worth knowing about before attempting export. Models with dynamic control flow — if/else branches that depend on tensor values — may not export cleanly with the standard tracing-based exporter; use torch.onnx.export() with dynamo=True (available in PyTorch 2.0+) for these cases. Models using custom PyTorch operators have no ONNX equivalent and need either a custom ONNX operator registration or a refactoring of the custom op into standard operations. Models with very deep graphs (thousands of nodes) sometimes hit memory limits during session creation — increase the session’s graph optimisation level incrementally if creation fails. For transformer models, exporting with fixed sequence lengths (no dynamic axes) significantly speeds up session creation and inference at the cost of flexibility — a worthwhile tradeoff for production services with known input constraints.