momentum

Momentum and adaptive learning rate methods like Adam share a fundamental mechanism—exponential moving averages that smooth gradient information across optimization steps—yet their parameters (momentum coefficient for SGD with momentum, beta1 and beta2 for Adam) require fundamentally different tuning strategies due to how they interact with learning rates and loss landscapes. SGD with momentum uses a … Read more