Amari's natural gradient descent is a well-known optimisation algorithm for functionals defined on statistical manifolds. It consists of preconditioning the vanilla gradient descent update rule with the inverse of the Fisher information metric tensor.

My intuition for the natural gradient is that it follows the steepest descent of the objective function while minimising the information distance of each update step (as a result of preconditioning).

(Q1) This leads me to ask whether, at the limit where the step-size tends to zero, natural gradient descent follows geodesics locally on the statistical manifold, according to the Fisher information metric?

(Q2) If this is not true in general, does this hold true in the context of variational inference (i.e. variational Bayes), where the objective function is the variational free energy, or evidence lower bound?

(Q3) If not, does this hold true with the same objective function, but when the approximate posterior distribution that one is trying to optimise is a categorical distribution? In this case, the statistical manifold is the standard simplex with Fisher-Rao distance becoming proportional to the Euclidean distance if one applies the canonical diffeomorphism onto the upper quadrant of the sphere.

If this is the case, could you explain how? If not, could you explain why not to help me improve my understanding? I would be grateful for answers to any of these questions.

• Can you provide reference for Amari's work ? In particular, gradient descent learning rates going to zero is not at all typical, so I am curious about that. – meh Jul 30 '19 at 15:02
• I have added a link to the main paper on natural gradient (to my knowledge). In practice, of course, one would not apply such small learning rates -- this question is more from a theoretical perspective, to know whether this scheme, if you see it as a flow on the manifold, follows geodesics locally. – Lance Jul 30 '19 at 15:07