Relationship between the Fisher distance and Kulback Leibler divergence

Question

I am reading the 2017 book "Information geometry" by Ay, Jost, Lê, Schwachhöfer. The Fisher distance is given by $$ d^F(\mu, \nu) := \inf_{\gamma} L(\gamma) $$ for curves $\gamma:[0,1]\to P$ with arclength

$$ L(\gamma) = \int_0^1 \|\dot\gamma\|_{\gamma(t)} dt $$

where $\|a\|_\mu = \langle a, a\rangle_\mu$ is induced by the fisher metric.

Now Ay motivates the Kullback Leibler divergence as a generalization of the squared norm, as its derivative are the geodesics with regard to the (e) and (m) connections. So in this sense the connections induce distance functions. But why are these distances reasonable and compatible with the fisher metric, when the fisher metric induces a very unique distance above?

Are these distances related? If I understand correctly, then the fisher distance $d^F$ is induced by the Levi-Cevita connection? Is this connection a member of the alpha connections family?

My understanding of differential geometry is somewhat limited so these questions might be very basic.

Answer 1

There seems to be a bit of a confusion here, since the geodesics of the (e) and (m) connections are not the third derivatives of the KL divergence. Instead, the third derivatives of the KL divergence give the Christoffel symbols for the (e) and (m) connections (depending on which argument is differentiated twice). For these connections, one can then consider geodesics with respect to these connections, which will be curves that satisfy $\nabla^{(e)}_{\dot \gamma} \dot \gamma =0$ (and similarly for m). One can compute the length of such curves with respect to the Fisher metric. Such curves will always be longer then their Levi-Civita counterparts.

As for a relationship between the KL divergence and distance induced by the KL divergence, the Fisher metric is the second-order Taylor polynomial of the KL divergence. In other words, given a fixed distribution $P(\theta_0)$ $$D_{\mathrm{KL}}[P(\theta_0) \| P(\theta)] = \frac{1}{2} \sum_{jk}\Delta\theta^j\Delta\theta^k g_{jk}(\theta_0) + \mathrm{O}(\Delta\theta^3).$$ As a result, we should expect the KL divergence to have comparisons between the squared distance with respect to the Fisher metric.

In many cases, it is possible to bound the distance with respect to the Fisher metric from above in terms of the KL divergence, but generally it's not possible to get bounds going the other way. For an example of the former, see this answer about the symmetrized KL divergence of multivariate normal distributions. A similar bound holds for multinomial distributions (i.e., $d_F(P,Q)^2 \leq C D_{KL}(P||Q))$ where $P$ and $Q$ are two multinomial distributions and $C$ is a positive constant (which I haven't computed carefully). To see why we cannot get a bound going in the opposite direction in this case, note that with respect to the Fisher metric the space of multinomial distributions is the positive orthant of a unit sphere, so the total diameter of the space is bounded. However, if there is an event which has non-zero probability with respect to $P$ but zero probability with respect to $Q$, the KL divergence $D_{KL}(P||Q)=\infty$, but the distance with respect to the Fisher metric is bounded, so we cannot hope to prove a bound going in the other direction.

And finally, if one considers the entire $(\alpha)$-family of connections where the (e) connection is $\alpha=1$ and the (m) connection is $\alpha=-1$, the Levi-Civita connection will correspond to the $\alpha=0$ connection.