2
$\begingroup$

Background

For a real-valued random variable $X$, define its entropy by $H(X) = E[\phi(X)] - \phi(E[X])$, where $\phi(u) = u \log u$. It can be shown that, if the entropy satisfies the bound $$ H(e^{\lambda X}) \leq \frac{\sigma^2 \lambda^2}{2} E[e^{\lambda X}] $$ for all real $\lambda$, then $X$ must be a sub-Gaussian variable with parameter $\sigma$, by which I mean the following holds for all real $\lambda$: $$ E[e^{\lambda (X - E[X])}] \leq \exp \left( \frac{\sigma^2 \lambda^2}{2} \right) \text{.} $$ This is known as the Herbst argument.

Question

Is the converse also true? i.e. do any sub-Gaussian variables satisfy the following entropy bound? $$ H(e^{\lambda X}) \leq \frac{\sigma^2 \lambda^2}{2} E[e^{\lambda X}] \quad \forall \lambda \in \mathbb{R} $$

$\endgroup$
2
  • 1
    $\begingroup$ It’s true up to a factor of 4, see e.g. Problem 3.12 of van Handel’s lecture notes on high-dimensional probability. $\endgroup$ Apr 1, 2023 at 19:11
  • $\begingroup$ @JasonGaitonde Thank you a lot for a nice reference! I will check it. $\endgroup$
    – aest
    Apr 3, 2023 at 7:32

1 Answer 1

2
$\begingroup$

As in the comment, the converse is true up to a factor of 4. The proof is given below.

Suppose $X$ is $\sigma^2/4$-subgaussian, i.e. the following holds for any $\lambda \in \mathbb{R}$: $$ \mathbb{E}[e^{\lambda X}] \leq \exp \left( \lambda \mathbb{E}[X] + \frac{\lambda^2 \sigma^2}{8} \right) \text{.} $$ Let $Z := e^{\lambda X} / \mathbb{E}[e^{\lambda X}]$. Then, one has \begin{align*} \mathbb{E}[Z \log Z] =& \mathbb{E} \left[ \frac{e^{\lambda X}}{\mathbb{E}[e^{\lambda X}]} \log \frac{e^{\lambda X}}{\mathbb{E}[e^{\lambda X}]} \right] \\=& \frac{1}{\mathbb{E}[e^{\lambda X}]} \mathbb{E} \left[ e^{\lambda X} \left( \log e^{\lambda X} - \log \mathbb{E}[e^{\lambda X}] \right) \right] \\=& \frac{1}{\mathbb{E}[e^{\lambda X}]} \left( \mathbb{E}[e^{\lambda X} \log e^{\lambda X}] - \mathbb{E}[e^{\lambda X}] \log \mathbb{E}[e^{\lambda X}] \right) \\=& \frac{H[e^{\lambda X}]}{\mathbb{E}[e^{\lambda X}]} \text{.} \end{align*} Moreover, note that $$ \mathbb{E}[Z \log Z] = \frac{\mathbb{E}[e^{\lambda X} \log Z]}{\mathbb{E}[e^{\lambda X}]} = \mathbb{E}_\lambda[\log Z] \text{,} $$ where $\mathbb{E}_\lambda[f(X)] := \mathbb{E}[e^{\lambda X} f(X)] / \mathbb{E}[e^{\lambda X}]$ denotes the Gibbs expectation. Now it suffices to show $\mathbb{E}_\lambda[\log Z] \leq \frac{\lambda^2 \sigma^2}{2}$.

The concavity of $\log$ and the Jensen inequality gives $$ \mathbb{E}_\lambda[\log Z] \leq \log \mathbb{E}_\lambda[Z] \text{.} $$ By the definition of the Gibbs expectation, we have \begin{align*} \mathbb{E}_\lambda[Z] =& \frac{\mathbb{E}_\lambda[e^{\lambda X}]}{\mathbb{E}[e^{\lambda X}]} \\=& \frac{\mathbb{E}[e^{2 \lambda X}] / \mathbb{E}[e^{\lambda X}]}{\mathbb{E}[e^{\lambda X}]} \\=& \frac{\mathbb{E}[e^{2 \lambda X}]}{\mathbb{E}[e^{\lambda X}]^2} \end{align*} Now the assumption gives $$ \mathbb{E}[e^{2 \lambda X}] \leq \exp \left(2 \lambda \mathbb{E}[X] + \frac{\lambda^2 \sigma^2}{2} \right) \text{,} $$ and the convexity of $\exp$ and the Jensen inequality yields $\mathbb{E}[e^{\lambda X}]^2 \geq e^{2 \lambda \mathbb{E}[X]}$. Hence, we have $$ \mathbb{E}_\lambda[\log Z] \leq \log \frac{\exp \left(2 \lambda \mathbb{E}[X] + \frac{\lambda^2 \sigma^2}{2}\right)}{\exp(2 \lambda \mathbb{E}[X])} = \frac{\lambda^2 \sigma^2}{2} $$ as desired.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.