Let the random variable $Y = X_1+X_2$, where $X_1$ follows an unknown distribution and $Y$ has finite variance.

Assuming as measurement of normality the entropy, is it correct to support that the entropy of $Y$ is maximized if and only if $X_2$ follows the normal distribution?

  • $\begingroup$ For a very short counterexample, suppose $Y$ has finite variance=1 and $X_1\sim |\mathcal{N}(0,1)|$, i.e., the absolute value of a normally distributed r.v. Then the entropy would be maximized if $X_2\sim -|\mathcal{N}(0,1)|$, i.e., the negative of the absolute value of a normally distributed r.v. $\endgroup$ – Bill Bradley Nov 18 at 20:18
  • $\begingroup$ @BillBradley : How do you prove this? Also, if I understood your example correctly, then the variance of $Y$ will be $2-4/\pi\ne1$ (if that matters). $\endgroup$ – Iosif Pinelis Nov 18 at 21:16
  • $\begingroup$ @IosifPinelis Ah, you're right! I had my head screwed on sideways; I was thinking of adding the distributions, not the r.v.s themselves. $\endgroup$ – Bill Bradley Nov 19 at 2:11

I interpret the problem as follows:

Let $Y=X_1+X_2$, where $X_1$ and $X_2$ are independent random variables (r.v.'s). We want to maximize the entropy of $Y$ with respect $X_2$ given a finite variance of $Y$, with the r.v. $X_1$ fixed. Is it true that the entropy of $Y$ is maximized if and only if $X_2$ has a normal distribution?

The answer to this question is no. Indeed, suppose that $P(X_1=1)=P(X_1=-1)=1/2$ and $Var\,Y=1+s^2$ for some real $s>0$, so that $Var X_2=s^2$. Suppose that $X_2$ is normally distributed, with a pdf $g$. Since the entropy of a pdf is invariant with respect to shifts, without loss of generality $X_2\sim N(0,s^2)$.

Let now $h$ be another pdf with mean $0$ and variance $s^2$. For $t\in[0,1]$, let $$f_t:=(1-t)g+th,$$ which is the pdf of a r.v. $X_{2,t}$ with variance $s^2$ such that $X_{2,t}$ is independent of $X_1$. So, the pdf $p_t$ of $Y_t:=X_1+X_{2,t}$ (with the same variance, $1+s^2$, as $Y$) is given by the formula $$p_t(y)=\tfrac12\,f_t(y+1)+\tfrac12\,f_t(y-1) $$ for real $y$. The entropy of $Y_t$ is $$H(t):=-\int_{\mathbb R}p_t(y)\ln p_t(y)\, dy, $$ and $H(0)$ is the entropy of $Y$. Next, $$H'(0)=-\int_{\mathbb R}[1+\ln p_0(y)]\,\frac\partial{\partial t}\,p_t(y)\Big|_{t=0}\, dy \\ =\int_{\mathbb R}\left[\frac{g(y+1)+g(y-1)}2-\frac{h(y+1)+h(y-1)}2\right]\, \ln\frac{g(y+1)+g(y-1)}2\, dy. $$ To disprove the conjecture, it is enough to find a suitable pdf $h$ such that $H'(0)>0$. In view of Chebyshev's integral inequality, it is then natural to choose $h$ which is almost mutually singular with $g$, so that $h$ is small where $g$ is large, and vice versa. With this in mind, let $h$ be the half-and-half mixture of the pdf's of $N(s\sqrt{1-u^2},u^2s^2)$ and $N(-s\sqrt{1-u^2},u^2s^2)$ with $u\in(0,1)$, so that the mean and variance of $h$ are indeed $0$ and $s^2$, respectively. Then $H'(0)=0.05121\ldots>0$ if $s=1/2$ and $u=1/20$, and we are done.

Here are graphs of the almost mutually singular $g$ (blue) and $h$ (yellow):

enter image description here

  • $\begingroup$ Thank you for your rigorus proof. I evaluated numerically your example and indeed for $t=0.0028$ the entropy is maximized. However, the half-and-half mixture has variance $0.2381 \neq s^2$. Maybe a correction is needed. I am wondering if there is a explicit method to approximate the optimal $X_2$ given $X_1$. Also if there is a bound for the optimality of the gaussian rv since the maximum is extremely close to $H(0)$ in this example. $\endgroup$ – Ioannis Papoutsidakis Nov 19 at 14:02
  • $\begingroup$ @IoannisPapoutsidakis : The second moment of each of the two distributions $N(\pm s\sqrt{1-u},u^2s^2)$ is $(\pm s\sqrt{1-u})^2+(us)^2=s^2$. So, the second moment of any mixture of these two distributions is also $s^2$. Since the mean of the half-and-half mixture is $0$, its variance is indeed $s^2$. As for the explicit optimal solution or the bound for the optimality, I don't know; you may want to ask those questions in (a) separate post(s). $\endgroup$ – Iosif Pinelis Nov 19 at 15:07
  • $\begingroup$ I think there is a typo. Because $(\pm s \sqrt{1-u})^2+(us)^2 = s^2-us^2+u^2s^2$. Maybe the mean should be defined as $ \pm s \sqrt{1-u^2}$. $\endgroup$ – Ioannis Papoutsidakis Nov 19 at 16:16
  • $\begingroup$ @IoannisPapoutsidakis : Thank you for spotting the mistake. It is now corrected. The main conclusion is not affected. $\endgroup$ – Iosif Pinelis Nov 19 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.