# Maximizing entropy of summation of unknown distributions

Let the random variable $$Y = X_1+X_2$$, where $$X_1$$ follows an unknown distribution and $$Y$$ has finite variance.

Assuming as measurement of normality the entropy, is it correct to support that the entropy of $$Y$$ is maximized if and only if $$X_2$$ follows the normal distribution?

• For a very short counterexample, suppose $Y$ has finite variance=1 and $X_1\sim |\mathcal{N}(0,1)|$, i.e., the absolute value of a normally distributed r.v. Then the entropy would be maximized if $X_2\sim -|\mathcal{N}(0,1)|$, i.e., the negative of the absolute value of a normally distributed r.v. – Bill Bradley Nov 18 at 20:18
• @BillBradley : How do you prove this? Also, if I understood your example correctly, then the variance of $Y$ will be $2-4/\pi\ne1$ (if that matters). – Iosif Pinelis Nov 18 at 21:16
• @IosifPinelis Ah, you're right! I had my head screwed on sideways; I was thinking of adding the distributions, not the r.v.s themselves. – Bill Bradley Nov 19 at 2:11

I interpret the problem as follows:

Let $$Y=X_1+X_2$$, where $$X_1$$ and $$X_2$$ are independent random variables (r.v.'s). We want to maximize the entropy of $$Y$$ with respect $$X_2$$ given a finite variance of $$Y$$, with the r.v. $$X_1$$ fixed. Is it true that the entropy of $$Y$$ is maximized if and only if $$X_2$$ has a normal distribution?

The answer to this question is no. Indeed, suppose that $$P(X_1=1)=P(X_1=-1)=1/2$$ and $$Var\,Y=1+s^2$$ for some real $$s>0$$, so that $$Var X_2=s^2$$. Suppose that $$X_2$$ is normally distributed, with a pdf $$g$$. Since the entropy of a pdf is invariant with respect to shifts, without loss of generality $$X_2\sim N(0,s^2)$$.

Let now $$h$$ be another pdf with mean $$0$$ and variance $$s^2$$. For $$t\in[0,1]$$, let $$f_t:=(1-t)g+th,$$ which is the pdf of a r.v. $$X_{2,t}$$ with variance $$s^2$$ such that $$X_{2,t}$$ is independent of $$X_1$$. So, the pdf $$p_t$$ of $$Y_t:=X_1+X_{2,t}$$ (with the same variance, $$1+s^2$$, as $$Y$$) is given by the formula $$p_t(y)=\tfrac12\,f_t(y+1)+\tfrac12\,f_t(y-1)$$ for real $$y$$. The entropy of $$Y_t$$ is $$H(t):=-\int_{\mathbb R}p_t(y)\ln p_t(y)\, dy,$$ and $$H(0)$$ is the entropy of $$Y$$. Next, $$H'(0)=-\int_{\mathbb R}[1+\ln p_0(y)]\,\frac\partial{\partial t}\,p_t(y)\Big|_{t=0}\, dy \\ =\int_{\mathbb R}\left[\frac{g(y+1)+g(y-1)}2-\frac{h(y+1)+h(y-1)}2\right]\, \ln\frac{g(y+1)+g(y-1)}2\, dy.$$ To disprove the conjecture, it is enough to find a suitable pdf $$h$$ such that $$H'(0)>0$$. In view of Chebyshev's integral inequality, it is then natural to choose $$h$$ which is almost mutually singular with $$g$$, so that $$h$$ is small where $$g$$ is large, and vice versa. With this in mind, let $$h$$ be the half-and-half mixture of the pdf's of $$N(s\sqrt{1-u^2},u^2s^2)$$ and $$N(-s\sqrt{1-u^2},u^2s^2)$$ with $$u\in(0,1)$$, so that the mean and variance of $$h$$ are indeed $$0$$ and $$s^2$$, respectively. Then $$H'(0)=0.05121\ldots>0$$ if $$s=1/2$$ and $$u=1/20$$, and we are done.

Here are graphs of the almost mutually singular $$g$$ (blue) and $$h$$ (yellow): • Thank you for your rigorus proof. I evaluated numerically your example and indeed for $t=0.0028$ the entropy is maximized. However, the half-and-half mixture has variance $0.2381 \neq s^2$. Maybe a correction is needed. I am wondering if there is a explicit method to approximate the optimal $X_2$ given $X_1$. Also if there is a bound for the optimality of the gaussian rv since the maximum is extremely close to $H(0)$ in this example. – Ioannis Papoutsidakis Nov 19 at 14:02
• @IoannisPapoutsidakis : The second moment of each of the two distributions $N(\pm s\sqrt{1-u},u^2s^2)$ is $(\pm s\sqrt{1-u})^2+(us)^2=s^2$. So, the second moment of any mixture of these two distributions is also $s^2$. Since the mean of the half-and-half mixture is $0$, its variance is indeed $s^2$. As for the explicit optimal solution or the bound for the optimality, I don't know; you may want to ask those questions in (a) separate post(s). – Iosif Pinelis Nov 19 at 15:07
• I think there is a typo. Because $(\pm s \sqrt{1-u})^2+(us)^2 = s^2-us^2+u^2s^2$. Maybe the mean should be defined as $\pm s \sqrt{1-u^2}$. – Ioannis Papoutsidakis Nov 19 at 16:16
• @IoannisPapoutsidakis : Thank you for spotting the mistake. It is now corrected. The main conclusion is not affected. – Iosif Pinelis Nov 19 at 18:14