5
$\begingroup$

The Rényi entropy for a probability density function $f$ with dominating measure $\mu$ of order $\alpha>0$ is defined as $$H_\alpha(f)={1 \over {\alpha-1}}\log\int f^\alpha d\mu.$$ If $f$ is smooth enough, as $\alpha \to 1$, the limit is the classical Shannon entropy $$H_1(f)=\int f\log f d\mu.$$ The Rényi entropy naturally defines a class of divergence $$M_\alpha(f,g)={1 \over {\alpha-1}}\log\int f^\alpha g^{1-\alpha} d\mu.$$ For fixed $f,g$ smooth enough, the limit divergence as we push $\alpha \to 1$ is the classical Kullback-Leibler divergence $$M_1(f,g)=\int f\log {f \over g} d\mu.$$ Now let's consider a slightly more complicated scenario: If $||g_\alpha/g||_\infty=1+o(1)$, then \begin{equation*} \begin{split} M_\alpha(f,g_\alpha)&={1 \over {\alpha-1}}\log\int f^\alpha g^{1-\alpha}\bigg(\frac{g_\alpha}{g}\bigg)^{1-\alpha} d\mu\\ &={1 \over {\alpha-1}}\bigg(\log \int f^\alpha g^{1-\alpha} d\mu+(1-\alpha)\log(1+o(1)\bigg)\\ &\to M_1(f,g). \end{split} \end{equation*} However, at the entropy level, the simple argument wouldn't apply to show $$H_\alpha(f_\alpha) \to H_1(f)$$ even if $||f_\alpha/f||_\infty=1+o(1)$. This is also tricky when think about divergence in its first entry. Is this something special about the asymmetry of the divergence $M_\alpha(\cdot,\cdot)$? Is there counterexample for which the convergence fails if $||f_\alpha/f||_\infty=1+o(1)$? Or any possible convergence conditions on $f_\alpha \to f$, without controlling the rate of convergence in terms of $\alpha$ that guarantees the convergence?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.