The learning rate stands as perhaps the most critical hyperparameter in training neural networks, yet it remains one of the most poorly understood by practitioners. Set it too high, and your model diverges into numerical chaos. Set it too low, and training crawls along at a glacial pace, potentially getting stuck in poor local minima. The question “which learning rate works best?” doesn’t have a single answer—but understanding the principles, patterns, and practical techniques for finding optimal learning rates can transform your model training from frustrating guesswork into systematic optimization.
Understanding What the Learning Rate Actually Does
Before diving into which learning rates work best, we need to understand precisely what this hyperparameter controls. During training, neural networks adjust their weights using gradient descent: they compute how much each weight contributes to the loss (the gradient) and update weights in the opposite direction to reduce loss. The learning rate determines the size of these update steps.
Mathematically, the weight update follows a simple rule: new_weight = old_weight – learning_rate × gradient. This deceptively simple equation governs the entire training dynamics. The gradient tells us which direction to move in weight space, while the learning rate determines how far we step in that direction.
When the learning rate is too large, weight updates overshoot the optimal values. Imagine trying to descend a hill by taking giant leaps—you might jump right over the valley at the bottom and end up climbing the opposite slope. In neural network terms, this causes the loss to oscillate wildly or even increase over time. In extreme cases, weights can explode to infinity, producing NaN (Not a Number) values that crash training entirely.
Conversely, when the learning rate is too small, training becomes painfully slow. Each update nudges weights only slightly toward better values, requiring exponentially more iterations to reach good solutions. Small learning rates also increase the risk of getting trapped in shallow local minima or flat regions of the loss landscape. The optimization process lacks the momentum to escape these suboptimal regions.
The ideal learning rate balances these competing concerns: large enough to make steady progress and escape local minima, but small enough to converge reliably toward good solutions. However, this ideal point isn’t static—it changes throughout training and varies dramatically across different models, datasets, and optimization algorithms.
The Empirical Starting Points: What Actually Works
Despite the theoretical complexity, decades of deep learning research have revealed practical patterns for effective learning rates. These empirical guidelines provide excellent starting points, though they should always be validated for your specific use case.
For standard SGD (Stochastic Gradient Descent) without momentum, learning rates typically fall between 0.01 and 0.1. A learning rate of 0.01 serves as a safe default for many problems—conservative enough to avoid divergence while still making reasonable progress. More aggressive practitioners often start with 0.1, particularly for well-behaved problems with clear gradients.
Adam and other adaptive optimizers require very different learning rate scales. The most common default for Adam is 0.001 (1e-3), which has become something of an industry standard. This value appears in countless research papers and production systems. Some practitioners prefer 0.0001 (1e-4) for more stable training, particularly with large models or noisy data.
RMSprop, another adaptive optimizer, typically works well with learning rates between 0.001 and 0.01. The learning rate of 0.001 again serves as a reasonable default, though the optimal value depends heavily on the problem structure.
These defaults aren’t magic numbers—they emerge from the mathematical properties of different optimizers. Adaptive methods like Adam automatically scale gradients for each parameter based on historical gradient magnitudes, effectively implementing per-parameter learning rates. This scaling means Adam can work with smaller base learning rates than SGD while achieving similar or faster convergence.
Common Learning Rate Ranges by Optimizer
SGD (no momentum)0.01 – 0.1
0.010.1
SGD with momentum0.001 – 0.1
0.0010.1
Adam / AdamW0.0001 – 0.001
0.00010.001
RMSprop0.001 – 0.01
0.0010.01
These ranges represent typical starting points. Optimal values depend on model architecture, batch size, and dataset characteristics.
The Learning Rate Range Test: Finding Your Sweet Spot
Rather than relying solely on default values, sophisticated practitioners use the learning rate range test to systematically identify optimal learning rates for their specific problem. This technique, popularized by Leslie Smith, provides an empirical method for finding the learning rate range where training is most effective.
The process is straightforward but powerful. Start training with an extremely small learning rate—something like 1e-8 that will definitely not cause divergence. After each mini-batch, increase the learning rate exponentially. Track the loss as the learning rate increases. Plot loss versus learning rate on a log scale. The resulting curve reveals critical information about your model’s training dynamics.
Initially, as the learning rate increases from tiny values, the loss decreases steadily. This region indicates learning rates that are too conservative—training works but proceeds slowly. As the learning rate continues increasing, you’ll observe a region where loss decreases most rapidly. This is your sweet spot. The learning rate where loss decreases fastest represents the optimal value for fast, stable training.
Keep increasing the learning rate beyond this point, and eventually the loss starts increasing or oscillating wildly. This indicates learning rates that are too large—the optimization is overshooting and failing to converge properly. The point where loss starts increasing marks the upper bound of usable learning rates.
Smith’s key insight was to use the learning rate roughly one order of magnitude below where the loss starts increasing. If loss explodes at 0.1, try 0.01. If the steepest descent occurs at 0.003, that’s your target learning rate. This conservative margin ensures stable training while maintaining fast convergence.
Here’s how to implement a learning rate range test in practice:
import numpy as npimport matplotlib.pyplot as pltdeflearning_rate_range_test(model,train_loader,start_lr=1e-8,end_lr=10,num_iter=100):""" Performs learning rate range test to find optimal learning rate. Args: model: Neural network model train_loader: Training data loader start_lr: Initial learning rate (very small) end_lr: Final learning rate (potentially too large) num_iter: Number of iterations to test Returns: learning_rates: List of tested learning rates losses: Corresponding loss values""" optimizer = torch.optim.SGD(model.parameters(),lr=start_lr) lr_mult =(end_lr / start_lr)**(1/ num_iter) learning_rates =[] losses =[] best_loss =float('inf') model.train() iterator =iter(train_loader)for iteration inrange(num_iter):# Get batchtry: inputs, targets =next(iterator)exceptStopIteration: iterator =iter(train_loader) inputs, targets =next(iterator)# Forward pass optimizer.zero_grad() outputs =model(inputs) loss =criterion(outputs, targets)# Track loss learning_rates.append(optimizer.param_groups[0]['lr']) losses.append(loss.item())# Stop if loss explodesif loss.item()>4* best_loss:breakif loss.item()< best_loss: best_loss = loss.item()# Backward pass loss.backward() optimizer.step()# Increase learning rate optimizer.param_groups[0]['lr']*= lr_multreturn learning_rates, losses# Run test and plot resultslrs, losses =learning_rate_range_test(model, train_loader)plt.plot(lrs, losses)plt.xscale('log')plt.xlabel('Learning Rate')plt.ylabel('Loss')plt.title('Learning Rate Range Test')plt.show()
The learning rate range test should be performed at the start of every significant training run. Different model architectures, dataset characteristics, and batch sizes can dramatically shift the optimal learning rate. Five minutes spent running this test can save hours or days of frustrating training runs with poorly chosen learning rates.
Batch Size and Learning Rate: The Critical Relationship
One of the most important yet frequently overlooked aspects of learning rate selection is its relationship with batch size. These two hyperparameters are intrinsically linked through the mathematics of gradient estimation, and changing one often requires adjusting the other.
When you increase batch size, you compute gradients over more examples before updating weights. This larger sample provides a more accurate estimate of the true gradient direction. With more accurate gradients, you can safely take larger steps without overshooting—meaning you can increase the learning rate. Conversely, smaller batches produce noisier gradient estimates, requiring smaller learning rates to avoid unstable training.
The empirical relationship discovered by researchers is approximately linear: if you double the batch size, you can often double the learning rate while maintaining similar training dynamics. This linear scaling rule holds surprisingly well across many problems, though it breaks down at very large batch sizes where other effects dominate.
For example, if you’re training with a batch size of 32 and learning rate of 0.01, increasing to batch size 64 suggests trying a learning rate of 0.02. Moving to batch size 256 would support a learning rate around 0.08. This scaling allows you to leverage larger batch sizes for faster training on parallel hardware without destabilizing optimization.
However, the linear scaling rule has limits. At very large batch sizes (thousands of examples), the relationship breaks down. Extremely large batches can actually hurt generalization—a phenomenon known as the “generalization gap.” Models trained with very large batches often achieve lower training loss but perform worse on test data. In these regimes, you may need to scale the learning rate sublinearly or employ specialized techniques like learning rate warmup.
Learning Rate Schedules: Dynamic Adjustment Over Time
While finding a good initial learning rate is crucial, the learning rate that works best at the start of training may not remain optimal throughout. Learning rate schedules—systematic changes to the learning rate during training—can dramatically improve final model performance.
The intuition behind learning rate schedules is straightforward. Early in training, when weights are far from optimal values, larger learning rates enable rapid progress toward better solutions. As training progresses and weights approach good values, smaller learning rates allow fine-tuning and prevent overshooting. This mirrors how you might navigate: take large steps when you’re far from your destination, then smaller steps as you approach to avoid walking past it.
Step Decay represents the simplest schedule: reduce the learning rate by a fixed factor at predetermined epochs. A common pattern is to multiply the learning rate by 0.1 every 30 epochs. For instance, start at 0.1, drop to 0.01 after 30 epochs, then to 0.001 after 60 epochs. This simple schedule works surprisingly well for many problems and requires minimal tuning.
Cosine Annealing provides a smoother decay, following a cosine curve from the initial learning rate down to a minimum value (often near zero). The learning rate decreases slowly at first, then more rapidly in the middle of training, then slowly again near the end. This schedule has become popular because it avoids the sudden drops of step decay while still providing aggressive reduction when needed.
Exponential Decay continuously decreases the learning rate by multiplying it by a constant factor each epoch or iteration. This creates a smooth, gradually decreasing schedule. Choose a decay rate close to 1 (like 0.95 or 0.99) for gradual decay over many epochs.
Warmup addresses a different problem: training instability in the first few iterations. When starting with randomly initialized weights, large learning rates can cause immediate divergence. Warmup starts with a very small learning rate and gradually increases it to the target value over the first few epochs. This gentle start gives the model time to find reasonable weight values before full-speed optimization begins. Warmup has become standard practice when training large models, particularly transformers.
Here’s a practical implementation of a cosine annealing schedule with warmup:
import mathclassCosineAnnealingWithWarmup:""" Learning rate scheduler with warmup and cosine annealing."""def__init__(self,optimizer,warmup_epochs,total_epochs,max_lr,min_lr=0,warmup_start_lr=1e-6):self.optimizer = optimizerself.warmup_epochs = warmup_epochsself.total_epochs = total_epochsself.max_lr = max_lrself.min_lr = min_lrself.warmup_start_lr = warmup_start_lrself.current_epoch =0defstep(self):"""Update learning rate for the next epoch."""ifself.current_epoch <self.warmup_epochs:# Linear warmup lr =(self.max_lr -self.warmup_start_lr)*\(self.current_epoch /self.warmup_epochs)+\self.warmup_start_lrelse:# Cosine annealing progress =(self.current_epoch -self.warmup_epochs)/\(self.total_epochs -self.warmup_epochs) lr =self.min_lr +(self.max_lr -self.min_lr)*\0.5*(1+ math.cos(math.pi * progress))for param_group inself.optimizer.param_groups: param_group['lr']= lrself.current_epoch +=1return lr# Example usageoptimizer = torch.optim.Adam(model.parameters(),lr=0.001)scheduler =CosineAnnealingWithWarmup( optimizer,warmup_epochs=5,total_epochs=100,max_lr=0.001,min_lr=1e-6)for epoch inrange(100):train_one_epoch(model, optimizer) scheduler.step()
The choice of schedule depends on your problem characteristics and computational budget. If you have limited time for hyperparameter tuning, step decay with a drop every 30-50 epochs provides a robust default. For maximum performance and if you know your total training budget, cosine annealing with warmup often delivers the best results.
Different neural network architectures exhibit dramatically different sensitivities to learning rates, requiring architecture-specific considerations when choosing this crucial hyperparameter.
Convolutional Neural Networks (CNNs) for computer vision typically train well with moderate learning rates. For training from scratch with SGD, learning rates between 0.01 and 0.1 work well for standard architectures like ResNet or EfficientNet. When using Adam, reduce this to 0.0001 to 0.001. However, when fine-tuning pre-trained vision models, much smaller learning rates—often 0.0001 or even 0.00001—prevent catastrophic forgetting of learned features.
Transformer models for natural language processing require careful learning rate selection. Large transformers like BERT or GPT variants typically use Adam with learning rates around 0.0001 to 0.0005. These models are particularly sensitive to learning rate during the initial training phase, making warmup essentially mandatory. A common pattern is 10,000 warmup steps with linear increase, followed by linear or cosine decay. The large number of parameters in transformers means small learning rates help prevent instability.
Recurrent Neural Networks (RNNs and LSTMs) present unique challenges due to vanishing and exploding gradient problems. These architectures often require smaller learning rates than comparable feedforward networks—typically 0.001 to 0.01 with Adam. Gradient clipping becomes essential to prevent exploding gradients, which allows slightly more aggressive learning rates than would otherwise be stable.
Very Deep Networks (100+ layers) need special attention. As networks get deeper, gradients must flow through more layers during backpropagation, making training increasingly sensitive to learning rate. Techniques like batch normalization and residual connections help, but ultra-deep networks still typically require smaller learning rates and longer warmup periods than shallow networks.
Learning Rate Impact on Training Dynamics
The optimal learning rate balances convergence speed with training stability. Too high causes oscillation or divergence; too low results in unnecessarily slow training.
Common Mistakes and How to Avoid Them
Even experienced practitioners make learning rate mistakes that torpedo otherwise well-designed training runs. Understanding these common pitfalls helps you avoid hours of wasted computation.
Mistake 1: Using the same learning rate across different optimizers. A learning rate of 0.1 might work perfectly with SGD but will cause immediate divergence with Adam. Always adjust learning rates when switching optimizers. As a rule of thumb, divide your SGD learning rate by 10 when moving to Adam.
Mistake 2: Ignoring the batch size connection. Practitioners often change batch size to fit different hardware without adjusting the learning rate. If you halve your batch size to fit on a smaller GPU, you should typically halve your learning rate too. The linear scaling rule provides good guidance here.
Mistake 3: Not using warmup for large models. Transformer models and other large architectures almost always benefit from learning rate warmup. Starting with the full learning rate can cause immediate instability or NaN losses. Include at least a few hundred warmup steps when training large models.
Mistake 4: Applying learning rate decay too early. Some practitioners start decaying the learning rate before the model has made significant progress. Let the model train at the higher learning rate until training loss plateaus, then apply decay. Premature decay wastes the fast early learning period.
Mistake 5: Using the same learning rate for fine-tuning as for training from scratch. When fine-tuning pre-trained models, you need much smaller learning rates—typically 10x to 100x smaller. Pre-trained weights already encode useful features; large learning rates destroy this knowledge before the model can adapt it to your task.
Mistake 6: Not monitoring learning rate during training. Many practitioners set a learning rate and never check what value the optimizer actually uses. Learning rate schedules, adaptive optimizers, and other mechanisms can change the effective learning rate. Always log the current learning rate alongside your loss curves.
Practical Decision Framework
Given all these considerations, how should you actually choose a learning rate for a new project? Here’s a systematic approach that works across different scenarios:
For standard supervised learning with established architectures: Start with optimizer defaults (0.001 for Adam, 0.01 for SGD with momentum). Train for a few epochs. If training is stable and loss decreases steadily, you’re done. If training is unstable or loss oscillates, reduce learning rate by 10x. If training is very slow, try increasing by 2-3x.
For new architectures or datasets: Run a learning rate range test. This takes 10-15 minutes but provides empirical evidence about what works for your specific setup. Use the learning rate one order of magnitude below where loss starts increasing.
For large-scale training: Use the linear scaling rule to adjust learning rate based on batch size. If previous work used batch size 32 with learning rate 0.01, and you’re using batch size 256, try learning rate 0.08. Include warmup—at least 5-10% of total training as warmup steps.
For fine-tuning: Start with 10x smaller learning rate than training from scratch. For vision models, this means 0.0001 or smaller with Adam. For language models, often 0.00001 to 0.00005. Use cosine decay or step decay to further reduce learning rate during fine-tuning.
For limited computational budget: Use step decay—simple to implement and robust across problems. Drop learning rate by 10x every 30-50 epochs, or when validation loss plateaus for 5+ epochs.
Conclusion
The question “which learning rate works best?” has no single answer, but the principles and techniques outlined here provide a systematic approach to finding optimal values. Start with empirically validated defaults for your optimizer and architecture, adjust based on batch size using the linear scaling rule, and validate your choice through learning rate range tests. Learning rate schedules, particularly cosine annealing with warmup, consistently improve final performance across diverse problems.
The most important lesson is that learning rate selection deserves serious attention—it’s not a detail to gloss over with default values. A well-chosen learning rate can be the difference between a model that trains in hours versus days, or between one that achieves state-of-the-art performance versus one that underperforms. Invest time in systematic learning rate selection at the start of your project, and you’ll reap rewards throughout training and in your final results.
