# About optimization with Renyi divergence

• Can someone link me to some pedagogic example of computing the Renyi divergence between two discrete/continuous distributions? Like examples where someone has been able to obtain a neat closed form or such answer?

• Is there any example of doing an optimization on Renyi divergence? Like given a distribution and some constraints on a second one, being able to write down the second distribution such that their mutual $\alpha$-Renyi divergence is minimized. Is there such an example?

For the requested examples see Rényi Divergence and Kullback-Leibler Divergence (2012).

Two continuous distributions: Equation 10 gives the Rényi divergence between two Gaussian distributions (mean $$\mu_i$$, variance $$\sigma_i$$):

$$D_\alpha\Big({\cal N}(\mu_0,\sigma_0^2)\|{\cal N}(\mu_1,\sigma_1^2)\Big) = \frac{\alpha(\mu_1 - \mu_0)^2}{2\sigma_\alpha^2} + \frac{1}{1-\alpha} \log\left( \frac{\sigma_\alpha}{\sigma_0^{1-\alpha}\sigma_1^\alpha}\right),$$ for $$\sigma_\alpha^2 = (1-\alpha)\sigma_0^2 + \alpha \sigma_1^2 > 0$$

Two discrete distributions: Figures 2 and 3 show the Rényi divergence $$D_\alpha(P||Q)$$ for fixed $$Q$$ as $$P$$ varies over a sample space containing two or three elements.

• Thanks! And any example of doing the kind of optimization that I was looking for? Commented Apr 4, 2016 at 2:35
• for $\alpha=2$ there exist closed-form expressions for mixtures of Gaussians, which have been used in an optimization problem in ncbi.nlm.nih.gov/pmc/articles/PMC2921653 Commented Apr 4, 2016 at 9:49
• Thanks! I am looking for examples where one is given a distribution and one is trying to find another distribution so that some $\alpha-$Renyi divergence between them is minimized. Know of any? Commented Apr 4, 2016 at 13:58

For your first question, the MSc thesis by Manuel Gil provides/derives closed form expressions for Renyi divergences associated with several of the continuous distributions.