Note: a better formatted version (due to lack of LaTeX support on medium) is here Why do we get diminishing returns with larger batch sizes? As you increase the mini-batch size, the estimate of the gradient gets more accurate. …

A better formatted version of this article is on Ghost, which has proper Latex support: https://machine-learning-etc.ghost.io/optimal-learning-rate-for-high-dimensional-quadratic/ Suppose our problem is to minimize a quadratic y=cx2 by using gradient descent. Gradient descent has a learning rate parameter α, also known as step size, what value should we use? For the problem…

Suppose you have a simple composition of d dense functions. Computing Jacobian needs d matrix multiplications. What about computing Hessian vector product? You can calculate it manually by differentiating function composition twice, grouping shared work together in temporary messages, and then counting the number of matrix multiplications. One trick is…

Suppose you have the following scalar function of matrix variable W. What’s the derivative with respect to matrix W? Define“matrix derivative” Df as “the thing that you subtract from your variable to go in the steepest descent direction”. IE, your gradient descent update would use Df as follows:

I was recently reminded of a lesson learned at Stephen Boyd’s Convex Optimization class at Stanford a few years ago, back when Google was paying for random classes taken by employees. The lesson is that you should rely on context to drop redundant information from mathematical notation. It may feel…

Background In “AI and Compute”, OpenAI reported that the cost of training AI models has been growing exponentially, observing a doubling period of 3.5 months. At the current rate, in 4 years, training the largest model will cost more than launching a rocket into orbit. If the trend continues, it would…

Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions (part I, part II) Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, Roger Grosse https://arxiv.org/abs/1903.03088 One approach to hyper-parameter choice is to apply gradient descent in the hyper-parameter space. For each setting of hyper-parameters, you run your optimization to convergence, get resulting loss, and then backprop through these steps to…

part I, part III Critical Learning Periods in Deep Neural Networks https://arxiv.org/abs/1711.08856 They observe a “critical period” when interfering with learning can have the large effect. Blur the image for a couple of epochs and see what happens. The largest effect happened if blur was introduced several epochs after the training started. This suggests a “critical period”…

(part II, part III) In this series of posts I will talk about optimization papers that caught my eye at ICLR 2019. The first post in the series is an overview of “Fluctuation-dissipation relations for stochastic gradient descent” by Sho Yaida. This paper uses beautifully simple math in order to…