About optimization with Renyi divergence

Question

• Can someone link me to some pedagogic example of computing the Renyi divergence between two discrete/continuous distributions? Like examples where someone has been able to obtain a neat closed form or such answer?

• Is there any example of doing an optimization on Renyi divergence? Like given a distribution and some constraints on a second one, being able to write down the second distribution such that their mutual $\alpha$-Renyi divergence is minimized. Is there such an example?

Answer 1

For the requested examples see Rényi Divergence and Kullback-Leibler Divergence (2012).

• Two continuous distributions: Equation 10 gives the Rényi divergence between two Gaussian distributions (mean $\mu_i$, variance $\sigma_i$):

$$ D_\alpha\Big({\cal N}(\mu_0,\sigma_0^2)\|{\cal N}(\mu_1,\sigma_1^2)\Big) = \frac{\alpha(\mu_1 - \mu_0)^2}{2\sigma_\alpha^2} + \frac{1}{1-\alpha} \log\left( \frac{\sigma_\alpha}{\sigma_0^{1-\alpha}\sigma_1^\alpha}\right), $$ for $\sigma_\alpha^2 = (1-\alpha)\sigma_0^2 + \alpha \sigma_1^2 > 0$

• Two discrete distributions: Figures 2 and 3 show the Rényi divergence $D_\alpha(P||Q)$ for fixed $Q$ as $P$ varies over a sample space containing two or three elements.

Answer 2

For your first question, the MSc thesis by Manuel Gil provides/derives closed form expressions for Renyi divergences associated with several of the continuous distributions.