Amari's natural gradient descent is a well-known optimisation algorithm for functionals defined on statistical manifolds. It consists of preconditioning the vanilla gradient descent update rule with the inverse of the Fisher information metric tensor.
My intuition for the natural gradient is that it follows the steepest descent of the objective function while minimising the information distance of each update step (as a result of preconditioning).
(Q1) This leads me to ask whether, at the limit where the step-size tends to zero, natural gradient descent follows geodesics locally on the statistical manifold, according to the Fisher information metric?
(Q2) If this is not true in general, does this hold true in the context of variational inference (i.e. variational Bayes), where the objective function is the variational free energy, or evidence lower bound?
(Q3) If not, does this hold true with the same objective function, but when the approximate posterior distribution that one is trying to optimise is a categorical distribution? In this case, the statistical manifold is the standard simplex with Fisher-Rao distance becoming proportional to the Euclidean distance if one applies the canonical diffeomorphism onto the upper quadrant of the sphere.
If this is the case, could you explain how? If not, could you explain why not to help me improve my understanding? I would be grateful for answers to any of these questions.