# Converse of the Herbst argument?

## Background

For a real-valued random variable $$X$$, define its entropy by $$H(X) = E[\phi(X)] - \phi(E[X])$$, where $$\phi(u) = u \log u$$. It can be shown that, if the entropy satisfies the bound $$H(e^{\lambda X}) \leq \frac{\sigma^2 \lambda^2}{2} E[e^{\lambda X}]$$ for all real $$\lambda$$, then $$X$$ must be a sub-Gaussian variable with parameter $$\sigma$$, by which I mean the following holds for all real $$\lambda$$: $$E[e^{\lambda (X - E[X])}] \leq \exp \left( \frac{\sigma^2 \lambda^2}{2} \right) \text{.}$$ This is known as the Herbst argument.

## Question

Is the converse also true? i.e. do any sub-Gaussian variables satisfy the following entropy bound? $$H(e^{\lambda X}) \leq \frac{\sigma^2 \lambda^2}{2} E[e^{\lambda X}] \quad \forall \lambda \in \mathbb{R}$$

• It’s true up to a factor of 4, see e.g. Problem 3.12 of van Handel’s lecture notes on high-dimensional probability. Apr 1, 2023 at 19:11
• @JasonGaitonde Thank you a lot for a nice reference! I will check it.
– aest
Apr 3, 2023 at 7:32

Suppose $$X$$ is $$\sigma^2/4$$-subgaussian, i.e. the following holds for any $$\lambda \in \mathbb{R}$$: $$\mathbb{E}[e^{\lambda X}] \leq \exp \left( \lambda \mathbb{E}[X] + \frac{\lambda^2 \sigma^2}{8} \right) \text{.}$$ Let $$Z := e^{\lambda X} / \mathbb{E}[e^{\lambda X}]$$. Then, one has \begin{align*} \mathbb{E}[Z \log Z] =& \mathbb{E} \left[ \frac{e^{\lambda X}}{\mathbb{E}[e^{\lambda X}]} \log \frac{e^{\lambda X}}{\mathbb{E}[e^{\lambda X}]} \right] \\=& \frac{1}{\mathbb{E}[e^{\lambda X}]} \mathbb{E} \left[ e^{\lambda X} \left( \log e^{\lambda X} - \log \mathbb{E}[e^{\lambda X}] \right) \right] \\=& \frac{1}{\mathbb{E}[e^{\lambda X}]} \left( \mathbb{E}[e^{\lambda X} \log e^{\lambda X}] - \mathbb{E}[e^{\lambda X}] \log \mathbb{E}[e^{\lambda X}] \right) \\=& \frac{H[e^{\lambda X}]}{\mathbb{E}[e^{\lambda X}]} \text{.} \end{align*} Moreover, note that $$\mathbb{E}[Z \log Z] = \frac{\mathbb{E}[e^{\lambda X} \log Z]}{\mathbb{E}[e^{\lambda X}]} = \mathbb{E}_\lambda[\log Z] \text{,}$$ where $$\mathbb{E}_\lambda[f(X)] := \mathbb{E}[e^{\lambda X} f(X)] / \mathbb{E}[e^{\lambda X}]$$ denotes the Gibbs expectation. Now it suffices to show $$\mathbb{E}_\lambda[\log Z] \leq \frac{\lambda^2 \sigma^2}{2}$$.
The concavity of $$\log$$ and the Jensen inequality gives $$\mathbb{E}_\lambda[\log Z] \leq \log \mathbb{E}_\lambda[Z] \text{.}$$ By the definition of the Gibbs expectation, we have \begin{align*} \mathbb{E}_\lambda[Z] =& \frac{\mathbb{E}_\lambda[e^{\lambda X}]}{\mathbb{E}[e^{\lambda X}]} \\=& \frac{\mathbb{E}[e^{2 \lambda X}] / \mathbb{E}[e^{\lambda X}]}{\mathbb{E}[e^{\lambda X}]} \\=& \frac{\mathbb{E}[e^{2 \lambda X}]}{\mathbb{E}[e^{\lambda X}]^2} \end{align*} Now the assumption gives $$\mathbb{E}[e^{2 \lambda X}] \leq \exp \left(2 \lambda \mathbb{E}[X] + \frac{\lambda^2 \sigma^2}{2} \right) \text{,}$$ and the convexity of $$\exp$$ and the Jensen inequality yields $$\mathbb{E}[e^{\lambda X}]^2 \geq e^{2 \lambda \mathbb{E}[X]}$$. Hence, we have $$\mathbb{E}_\lambda[\log Z] \leq \log \frac{\exp \left(2 \lambda \mathbb{E}[X] + \frac{\lambda^2 \sigma^2}{2}\right)}{\exp(2 \lambda \mathbb{E}[X])} = \frac{\lambda^2 \sigma^2}{2}$$ as desired.