4
$\begingroup$

$\DeclareMathOperator\Ent{End}$The $\phi$ entropy is defined as $\Ent_{\phi}[X]= \mathbb{E}[\phi (X)]-\phi(\mathbb{E}[ X])$ where $X$ is a random variable and $\phi$ is a convex function ($\Ent_{\phi}[X] \geq 0$). By choosing $\phi(x)=x^2$ we get $\Ent_{\phi}[X]=\operatorname{Var}(X)$. If $\phi(x)=x\log x$ and $X=\frac{dv}{d\mu} $ (Radon–Nikodym derivative) we get $\Ent_{\phi}[X]= D_{\text{KL}}(v\Vert\mu)$. Can we recover the classical entropy $H(X)= -\mathbb{E}[\log p(X)]$ in a similar way?

[Edit: added a minus sign in the definition of $H(X)$ -AK]

Improve this question
$\endgroup$
1
  • $\begingroup$ You were missing a minus sign in the definition of the classical entropy. $\endgroup$
    – Aryeh Kontorovich
    Commented Nov 21, 2017 at 20:15

2 Answers 2

Reset to default
4
$\begingroup$

The choice $\phi(x)=x\log(x)$ yields an entropy distinct from Shannon entropy. See p. 94 of Concentration Inequalities: A Nonasymptotic Theory of Independence by Boucheron, Lugosi, Massart.

The two entropies are related but distinct.

Improve this answer
$\endgroup$
0
$\begingroup$

I think you just want $\phi(x) = \log1/x$. Then $$Ent_\phi(P(X)) = E[\log P(X)] - \log[EP(x)] = H(X) - \log(1) = H(X).$$

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .