Relationship between $\alpha$-divergences?

Question

I am working with $\alpha$-divergences and was wondering how understand the relationship between the definitions of Renyi and Amari?

Renyi:

$D_{\alpha}[p||q] = \frac{1}{\alpha - 1} \log \int p^{\alpha} \, q^{1-\alpha} d \mu$

Amari (for probability distributions):

$D_{\alpha}[p||q] = \frac{4}{1-\alpha^2}( 1 - \int p^{\frac{1+\alpha}{2}} q^{\frac{1-\alpha}{2}} d \mu$)

I have framed my problem in terms of Renyi's definition, but was hoping to utilize information geometric results from Amari.

Thanks in advance!

Answer 1

First things first, let's change notation so that "$D_\alpha$" isn't being used for two different things. Let's write $R_\alpha$ for the Rényi divergence and $A_\alpha$ for the Amari divergence.

The simplest thing to say is that $R_\alpha$ and $A_{(1 + \alpha)/2}$ are related to one another by an invertible transformation that you can easily derive. So if you know the Rényi divergence of order $\alpha$, then you know the Amari divergence of order $(1 + \alpha)/2$, and vice versa. Moreover, that transformation is increasing (as long as $\alpha > 0$), so an increase in one corresponds to an increase in the other.

Here's a more lofty viewpoint which I find helpful. For $\alpha \in \mathbb{R}$, let $\ln_\alpha: (0, \infty) \to \mathbb{R}$ be the function defined by $$ \ln_\alpha(x) = \int_1^x t^{-\alpha} dt. $$ This is called the $\alpha$-logarithm. (Actually, it's more often called the $q$-logarithm, because the people who use this terminology tend to call the parameter $q$ rather than $\alpha$.) Explicitly, if $\alpha = 1$ then $\ln_\alpha = \log$, and otherwise $$ \ln_\alpha(x) = \frac{x^{1 - \alpha} - 1}{\alpha - 1}. $$ So you can view the functions $(\ln_\alpha)$ as a one-parameter family of deformations of the natural logarithm.

Now define $$ N_\alpha[p||q] = \biggl( \int p^\alpha q^{1 - \alpha} \biggr)^{\alpha - 1}. $$ This is a kind of exponential divergence. And the point is this:

• if you take the ordinary logarithm of $N_\alpha$, you get the Rényi divergence $R_\alpha$;

• if you take the $\alpha$-logarithm of $N_\alpha$, you get more or less the Amari divergence $A_{(1 + \alpha)/2}$.

That "more or less" is because it's actually a factor of $(1 + \alpha)/2$ off. I confess, I don't know why that Amari divergence is normalized as it is. Note that if we write $\beta = (1 + \alpha)/2$ then $$ A_\alpha(p||q) = \frac{1}{\beta(1 - \beta)}\Bigl( 1 - \int p^\beta q^{1 - \beta} \Bigr), $$ and I'm surprised at the extra factor of $\beta$ in the prefactor. Anyway, it's only a constant factor.

Answer 2

This is an old question but touches on some of the most important topics in information geometry. The good news is that in the information geometry sense, they are equivalent.

Recall how the usual Fisher information metric and the exponential/mixture connections on a statistical manifold can be derived from KL divergence- evaluating the Hessian of $D_{KL}(P||Q)$ with respect to the manifold parameters at P = Q gives the metric tensor as the infinitesimal curvature of the KL divergence. Evaluating the 3rd order partials with respect to two parameters of P and one of Q gives the Christoffel symbols of the exponential connection, and for two parameters of Q and one of P the mixture connection symbols result.

The reason that the alpha divergence is scaled and parameterized the way it is has a few purposes. Note that the normalization factor that the alpha divergence is divided by can be written as $\frac{1+\alpha}{2} \frac{1-\alpha}{2}$ making it more obvious that $D_\alpha(P||Q) = D_{-\alpha}(Q||P)$, where Renyi divergence needs to be scaled to get a relationship: $R_{1-\alpha}(P||Q) \frac{\alpha}{1-\alpha} = R_\alpha (Q||P)$.

While the symmetry is nice, normalizing differently has a more important effect- as you know, in the limit as $\alpha$ goes to 1 in Renyi divergence, the KL divergence (from Q to P in this notation) is obtained. This is a useful derivation to do yourself, and it might give you some more intuition. Apply L'Hopital's rule and differentiate with respect to $\alpha$. You can find the same result for the alpha divergence and taking the limit to -1 or 1, resulting in KL from P to Q or Q to P depending on the limit. Renyi divergence in the limit as alpha goes to 0 is interesting in its own right- the negative log probability under Q of values in the support of P- but for it to limit to the "reverse KL" it requires dividing by alpha.

So the alpha divergence smoothly parameterizes a family of divergences that, between -1 and 1, ranges from the primal KL divergence and the dual KL divergence, and it is "symmetric" and "closed" in the sense that the primal connection for alpha is the dual connection for negative alpha. But, for an exercise, take Renyi divergence and divide by alpha to rescale it to be symmetric in the same way as the alpha divergence. Take the 3rd order partials of each and evaluate them at P = Q. Spoiler alert: they are equal. That is, the alpha divergence and revealed Renyi divergence induce the same infinitesimal structure (after reparameterizing of course) and this is the ultimate takeaway. The metric tensor induced by KL divergence, alpha divergence, and Renyi divergence is all the same, and after rescaling, the induced connections of Renyi and alpha divergence are equivalent.

As a final note- denoting the connections induced by alpha divergence by $\nabla^{\alpha}$, the dual connection is $\nabla^{-\alpha}$ and perhaps startlingly we have $$\nabla ^\alpha = \frac{1+\alpha}{2}\nabla + \frac{1-\alpha}{2}\nabla^{*}$$

Which shows that the alpha connection is a linear combination of the primal and dual KL connections.

Interestingly, for $\alpha = 0$ the alpha connection is the Levi-Civita connection under the Fisher-Rao metric (and the same is true for Renyi divergence with $\alpha = \frac{1}{2}$).