Maximizing mutual information between two linearly projected random variables

Question

Consider continuous random variable $X$ with bounded support $\mathcal{X} \subset \mathbb{R}$, discrete random variable $Y$ with finite support $\mathcal{Y}$ , and constants $c_1 ,c_2 \in \mathbb{R}$. Let $p_{XY}(x,y)$ be the probability measure on $\mathcal{X} \times \mathcal{Y}$. Define $H(X,Y)$ and $H(Y\mid X)$ as the joint and conditional entropies. Define $Z = c_1 X_1 + c_2 X_2$ where $X_1$ and $X_2$ are i.i.d. realizations from the probability measure on $X$. What are $c_1,c_2$ that maximize $I(Y;Z)$ where $I$ is the mutual information? what about general case with $n$ constants $c_1,...,c_n$?

Answer 1

One has to be careful with the definitions as (unlike in the discrete case) there is no such thing as the "entropy" of continuous random variables, and one can only talk about the differential entropy of their densities with respect to certain reference measures (usually the Lebesgue ones). Anyway, with any reasonable definition the difference $H(Y,Z)-H(Y|Z)$ should coincide with the entropy of the condition $H(Z)$. By the basic properties of the entropy $$ H(Z) \le H(X_1) + H(X_2) \;, $$ and the equality holds if and only if the map $$ (x_1,x_2)\mapsto c_1 x_1 + c_2 x_2 $$ is one-to-one on a set of full measure. Since the distribution of $X_i$ has a finite support, one can always choose the constants $c_1,c_2$ in such a way that the latter condition is satisfied. The same argument works for an arbitrary finite number of summands as well.