Why the H100 vs A100 Decision Matters
The NVIDIA A100 has been the workhorse GPU for LLM training and inference since 2020. The H100, released in 2022 and widely available by 2023, represents the next generation — faster, more memory-efficient, and with new hardware features specifically designed for transformer workloads. For teams provisioning GPU infrastructure today, choosing between A100 and H100 instances involves weighing a 2–3x performance difference against a 2–4x price premium, across a range of training and inference workloads where the performance gap varies considerably.
This guide breaks down the technical differences, the practical performance gap on real LLM workloads, the cost comparison across major cloud providers, and a decision framework for choosing the right GPU for your specific use case.
Technical Specifications: A100 vs H100
Specification | A100 SXM 40GB | A100 SXM 80GB | H100 SXM 80GB | H100 NVL 188GB
-----------------------|---------------|---------------|---------------|----------------
GPU Memory | 40 GB HBM2 | 80 GB HBM2e| 80 GB HBM3 | 188 GB HBM3e
Memory Bandwidth | 1,555 GB/s | 2,039 GB/s | 3,350 GB/s | 7,680 GB/s
FP16 Tensor Core TFLOPS| 312 | 312 | 989 | 1,979
BF16 Tensor Core TFLOPS| 312 | 312 | 989 | 1,979
FP8 Tensor Core TFLOPS | N/A | N/A | 1,979 | 3,958
NVLink Bandwidth | 600 GB/s | 600 GB/s | 900 GB/s | 900 GB/s
TDP (power) | 400W | 400W | 700W | 700W
PCIe Generation | 4 | 4 | 5 | 5The three headline differences are memory bandwidth (H100 delivers 64% more than A100 80GB), compute throughput (roughly 3x higher in BF16), and the new FP8 precision mode on H100 that has no equivalent on A100. For LLM inference, which is bandwidth-bound, the bandwidth improvement matters most. For training, the compute improvement matters more.
The Transformer Engine: H100’s Key Advantage for LLMs
H100 introduces the Transformer Engine, a hardware and software co-design feature that automatically manages mixed-precision computation within transformer layers. The key capability is FP8 computation — 8-bit floating point precision for matrix multiplications within the forward and backward passes, with careful scaling to maintain training stability. FP8 roughly doubles throughput over BF16 on the same hardware because more operations fit in the same cycle.
For inference, FP8 means an H100 can process roughly twice as many tokens per second as a BF16 A100 for transformer models, on top of the already higher raw bandwidth. For training, the Transformer Engine’s automatic mixed precision handling reduces the engineering overhead of manually managing precision for stability, while achieving throughput that A100 cannot match even with carefully tuned BF16 training.
The Transformer Engine requires framework support — PyTorch, JAX, and TensorFlow all support it via the transformer_engine library. Enabling it is typically a one or two line change in existing training code:
import transformer_engine.pytorch as te
# Replace standard PyTorch layers with TE equivalents
linear = te.Linear(in_features, out_features, bias=True)
# Or use the context manager for automatic FP8 scaling
with te.fp8_autocast(enabled=True):
output = model(input)Performance Comparison on LLM Workloads
Real-world performance comparisons from published benchmarks and cloud provider data:
Workload | A100 80GB | H100 80GB | H100 speedup
----------------------------------|-----------|-----------|-------------
Llama 3.1 70B inference (tok/s) | 25 | 65 | 2.6x
Llama 3.1 70B training (tok/s) | 850 | 2,100 | 2.5x
Llama 3.1 405B inference (8 GPU) | 8 | 22 | 2.8x
Fine-tuning 7B (samples/sec) | 85 | 220 | 2.6x
Embedding generation (docs/sec) | 1,200 | 2,900 | 2.4x
RAG batch processing | 950 | 2,400 | 2.5xThe H100 delivers roughly 2.4–2.8x better throughput than A100 on LLM workloads. The speedup is relatively consistent across inference and training, which reflects that both are primarily bandwidth and compute-bound in similar proportions for transformer architectures. The FP8 path delivers the highest gains; BF16-only workloads still see 1.5–2x improvement from the higher raw bandwidth alone.
Cloud Pricing: A100 vs H100
On-demand GPU instance pricing as of mid-2026 (prices vary; verify before purchasing):
Cloud Provider | Instance Type | GPU Config | On-Demand $/hr | Spot $/hr
--------------------|------------------------|----------------|----------------|----------
AWS | p4d.24xlarge | 8x A100 40GB | $32.77 | ~$10
AWS | p4de.24xlarge | 8x A100 80GB | $40.96 | ~$13
AWS | p5.48xlarge | 8x H100 80GB | $98.32 | ~$35
GCP | a2-highgpu-8g | 8x A100 40GB | $29.39 | ~$9
GCP | a3-highgpu-8g | 8x H100 80GB | $98.68 | ~$32
Azure | Standard_ND96asr_v4 | 8x A100 80GB | $32.77 | ~$11
Azure | Standard_ND96isr_H100 | 8x H100 80GB | $98.32 | ~$28H100 instances cost approximately 2.5–3x more than equivalent A100 instances on-demand. On spot/preemptible instances, the ratio is similar. The key calculation: if H100 delivers 2.5x the throughput at 2.5–3x the price, the cost per token is roughly equal or slightly higher on H100. H100 wins on wall-clock time — the same job finishes 2.5x faster — but does not necessarily win on cost efficiency for throughput-normalised comparisons.
When to Choose H100
Time-sensitive training runs. If you are iterating rapidly on model training and each run matters to your development cycle, H100’s 2.5x speedup is directly valuable — you can run 2.5 experiments in the time an A100 completes one. For research teams and rapid prototyping, this time compression often justifies the cost premium even if total compute cost is similar.
Large model inference under latency SLAs. For production inference where time-to-first-token and generation latency are customer-facing metrics, H100 delivers substantially lower p99 latencies at high concurrency. If your application has strict latency requirements, the throughput advantage of H100 may be necessary to meet SLAs at your traffic volume.
Very large models (70B+). The performance gap between H100 and A100 is most pronounced for large models where bandwidth is the dominant constraint. For 405B models requiring 8-GPU configurations, H100’s higher per-GPU bandwidth and faster NVLink interconnects deliver the full 2.5x benefit. For 7B models that fit easily on a single GPU, the gap narrows.
Future-proofing for FP8 workloads. As FP8 training and inference mature and gain broader framework support, the H100’s advantage will increase. Teams investing in infrastructure for a 2–3 year horizon should weight the H100’s FP8 capability heavily.
When to Choose A100
Cost-constrained inference. For steady-state inference where throughput-per-dollar matters more than throughput-per-GPU, A100 is often the more economical choice. Spot A100 instances are significantly cheaper and widely available. If your workload fits on A100 within your latency budget, the cost savings are real.
Existing infrastructure. Teams with existing A100 clusters and established MLOps pipelines have no compelling reason to migrate mid-project. A100 remains an excellent GPU that will be in active production use for years. Upgrade to H100 at the next infrastructure refresh cycle rather than mid-deployment.
Smaller models (7B–13B). For inference on smaller models that comfortably fit on a single GPU, the raw bandwidth advantage of H100 over A100 is proportionally smaller relative to total inference cost. A100 handles these workloads efficiently at lower cost.
Availability and spot pricing. A100 instances are more available on spot markets than H100, with more predictable pricing. For workloads that can tolerate preemption (training checkpointing is standard), spot A100 is often the most cost-effective path.
The H100 NVL: A Different Category
Beyond the standard H100 SXM, NVIDIA offers the H100 NVL — a dual-H100 configuration on a single card with 188 GB of HBM3e memory and 7,680 GB/s of memory bandwidth. This is not two separate GPUs; it is a single logical GPU with combined memory. The NVL is specifically designed for large language model inference where a single very large memory pool is more valuable than two separate 80 GB pools.
For serving 405B models on a smaller cluster footprint, or for running 70B models at FP16 quality on a single device, the NVL is the only off-the-shelf option. It commands a significant price premium and is primarily available on specialised cloud providers and for direct purchase. Most teams do not need it — but for specific use cases requiring very large single-device memory, it is worth knowing it exists.
Practical Recommendation
For new deployments in 2026: if your budget allows and your workloads are LLM-heavy, default to H100. The 2.5x throughput improvement is substantial, the FP8 training path will become increasingly valuable, and the per-token cost is competitive with A100 at similar utilisation. For cost-sensitive deployments or teams already on A100 infrastructure, A100 remains an excellent choice that handles all current LLM workloads effectively. Do not let perfect be the enemy of good — an A100 cluster running vLLM efficiently will serve your users well while H100 capacity is reserved for training runs where the time-to-completion directly impacts your development velocity.
When to Choose A100
Cost-constrained inference. For steady-state inference where throughput-per-dollar matters more than throughput-per-GPU, A100 is often the more economical choice. Spot A100 instances are significantly cheaper and widely available. If your workload fits on A100 within your latency budget, the cost savings are real and meaningful at scale.
Existing infrastructure. Teams with established A100 clusters and mature MLOps pipelines have no compelling reason to migrate mid-project. A100 remains an excellent GPU that will be in active production use for years. Upgrade at the next infrastructure refresh rather than mid-deployment.
Smaller models at low concurrency. For inference on 7B–13B models that fit comfortably on a single GPU with low concurrent request volume, the throughput advantage of H100 over A100 is proportionally smaller relative to total cost. A100 handles these workloads efficiently.
Spot market availability. A100 spot instances are more available and more predictably priced than H100 on most cloud providers. For training jobs with checkpointing, spot A100 is often the most cost-effective path for teams that can tolerate occasional preemption.
The H100 NVL: 188 GB of Unified Memory
Beyond the standard H100 SXM, NVIDIA offers the H100 NVL — a dual-H100 configuration on a single card with 188 GB of HBM3e and 7,680 GB/s of combined bandwidth. This is a single logical GPU with combined memory, not two separate GPUs. The NVL is designed specifically for LLM inference where a large single memory pool is more valuable than two separate 80 GB pools. For serving 405B models on a smaller cluster footprint, or running 70B models at FP16 quality on a single device, the NVL is the only off-the-shelf option. It commands a significant price premium and is primarily available on specialised cloud providers. Most teams do not need it, but for specific use cases requiring very large single-device memory, it is worth evaluating.
Multi-GPU Scaling: NVLink Matters
For models requiring multiple GPUs — 70B at FP16, 405B at any precision — the interconnect between GPUs becomes a bottleneck. Both A100 and H100 use NVLink, but H100’s NVLink 4.0 delivers 900 GB/s bidirectional bandwidth versus A100’s 600 GB/s. For tensor-parallel inference across 4–8 GPUs, this 50% higher interconnect bandwidth reduces the all-reduce communication overhead that limits scaling efficiency. In practice, H100 clusters achieve better multi-GPU scaling efficiency than A100 clusters for large model inference, amplifying the per-GPU performance advantage further at scale.
Availability and Lead Times
H100 availability on cloud providers has improved significantly through 2025 and 2026 as supply constraints eased. On-demand H100 instances are now reliably available on AWS, GCP, and Azure in major regions without multi-week waitlists. Spot H100 availability is more variable but improving. A100 instances remain more immediately available with better spot market depth. For teams that need guaranteed capacity on short notice, A100 is still the more reliable choice; for planned deployments with lead time to reserve, H100 is fully accessible.
Practical Recommendation
For new LLM infrastructure deployments in 2026: if your budget allows and workloads are LLM-heavy, default to H100. The 2.5x throughput improvement is substantial, the FP8 training path will become increasingly valuable as frameworks mature, and the per-token inference cost is competitive with A100 at similar utilisation levels. For cost-sensitive deployments or teams already running A100 infrastructure, A100 remains excellent — it handles all current LLM workloads effectively and will continue to do so for years. The right answer is workload-specific: measure your actual throughput requirements, run the cost calculation for your specific traffic patterns, and choose the GPU that delivers your needed performance within your budget rather than defaulting to the newest hardware without evaluating whether the premium is justified for your use case.