Can someone link me to some pedagogic example of computing the Renyi divergence between two discrete/continuous distributions? Like examples where someone has been able to obtain a neat closed form or such answer?
Is there any example of doing an optimization on Renyi divergence? Like given a distribution and some constraints on a second one, being able to write down the second distribution such that their mutual $\alpha$Renyi divergence is minimized. Is there such an example?
2 Answers
For the requested examples see Rényi Divergence and KullbackLeibler Divergence (2012).
• Two continuous distributions: Equation 10 gives the Rényi divergence between two Gaussian distributions (mean $\mu_i$, variance $\sigma_i$):
$$ D_\alpha\Big({\cal N}(\mu_0,\sigma_0^2)\{\cal N}(\mu_1,\sigma_1^2)\Big) = \frac{\alpha(\mu_1  \mu_0)^2}{2\sigma_\alpha^2} + \frac{1}{1\alpha} \log\left( \frac{\sigma_\alpha}{\sigma_0^{1\alpha}\sigma_1^\alpha}\right), $$ for $\sigma_\alpha^2 = (1\alpha)\sigma_0^2 + \alpha \sigma_1^2 > 0$
• Two discrete distributions: Figures 2 and 3 show the Rényi divergence $D_\alpha(PQ)$ for fixed $Q$ as $P$ varies over a sample space containing two or three elements.

$\begingroup$ Thanks! And any example of doing the kind of optimization that I was looking for? $\endgroup$ Commented Apr 4, 2016 at 2:35

$\begingroup$ for $\alpha=2$ there exist closedform expressions for mixtures of Gaussians, which have been used in an optimization problem in ncbi.nlm.nih.gov/pmc/articles/PMC2921653 $\endgroup$ Commented Apr 4, 2016 at 9:49

$\begingroup$ Thanks! I am looking for examples where one is given a distribution and one is trying to find another distribution so that some $\alpha$Renyi divergence between them is minimized. Know of any? $\endgroup$ Commented Apr 4, 2016 at 13:58
For your first question, the MSc thesis by Manuel Gil provides/derives closed form expressions for Renyi divergences associated with several of the continuous distributions.

1$\begingroup$ The link is dead but I think this is what you are referring to: qspace.library.queensu.ca/handle/1974/6680 $\endgroup$– muammarCommented Oct 9, 2021 at 12:45