Bao Wang: “Momentum in Stochastic Gradient Descent and Deep Neural Nets“

Deep Learning and Medical Applications 2020 “Momentum in Stochastic Gradient Descent and Deep Neural Nets“ Bao Wang - University of California, Los Angeles (UCLA), Mathematics Abstract: Stochastic gradient-based optimization algorithms play perhaps the most important role in modern machine learning, in particular, deep learning. Nesterov accelerated gradient (NAG) is a celebrated technique to accelerate gradient descent, however, the NAG technique will fail in stochastic gradient descent (SGD). In this ta