# Mutual information between continuous and discrete variables from numerical data

I am looking for references/measures to estimate the mutual information between a continuous (C) and discrete (D) variable, given a real-world (i.e. finite sample) data set. C is uniformly distributed between Cmin and Cmax, and D is a positive integer whose distribution depends on the value of C (e.g., a poisson distribution with mean f(C)). Any suggestions (methods based on binning over the data set, or independent of binning) are welcome.

• Since you already know the prior distribution of C, it would be probably easier to estimate the conditional entropy of C given D to estimate mutual information. (Curiously, this is something you may encounter in neuroscience applications for spike counts.) As for the binning question, MO is probably not the best place to ask since it is not really a math question. Try stats.SE? – Memming Jan 21 at 13:24
• Thanks for the clue. Will try implementing it. – Gaurang Jan 23 at 7:45

Then the mutual information between $$D$$ and $$X$$ may/should be defined as follows: $$\begin{equation*} I(D,X):=\sum_{i\in I}\int_S d\mu\;\rho_i\ln\frac{\rho_i}{p_i}. \tag{!} \end{equation*}$$
Justification: Take any natural $$n$$. Let $$J:=J_n:=\{0,\dots,n\}^I$$, the set of families $$j=(j_i)_{i\in I}$$ with $$j_i\in\{0,\dots,n\}$$. Let us assume that $$I$$ is finite, so that $$J$$ is countable. For any $$j\in J$$, let $$\begin{equation*} \de_j:=\de_j(n):=\{x\in S\colon j_i/n\le\rho_i(x)<(j_i+1)/n\ \forall i\in I\}. \tag{2} \end{equation*}$$ Let $$J_{n;0}:=\{j\in J\colon\de_j\ne\emptyset\}$$. Note that the sets $$\de_j$$ with $$j\in J_{n;0}$$ form a partition of $$S$$. For each $$j\in J_{n;0}$$, pick then an arbitrary $$x_j\in\de_j$$, and let $$X_n$$ be the r.v. taking value the constant value $$x_j$$ on $$\de_j$$ for each $$j\in J_{n;0}$$, so that $$X_n$$ is a discrete approximation to the r.v. $$X$$. Then, according to the standard definition, the mutual information between $$D$$ and $$X_n$$ is \begin{align} I(D,X_n)&:=\sum_i\sum_j P(D=d_i,X_n=x_j)\ln\frac{P(D=d_i,X_n=x_j)}{P(D=d_i)P(X_n=x_j)} \\ &=\sum_i\sum_j P(D=d_i,X\in\de_j)\ln\frac{P(D=d_i,X\in\de_j)}{P(D=d_i)P(X\in\de_j)} \\ &=\sum_i\sum_j \mu_i(\de_j)\ln\frac{\mu_i(\de_j)}{p_i\mu(\de_j)}; \end{align} here and in what follows, $$\sum_i:=\sum_{i\in I}$$ and $$\sum_j:=\sum_{j\in J_{n;0}}$$; as usual, the $$(i,j)$$-summand here is understood as $$0$$ whenever $$\mu_i(\de_j)=0$$. Next, by (1) and (2), $$\begin{equation*} \frac{\mu_i(\de_j)}{\mu(\de_j)}=\frac1{\mu(\de_j)}\,\int_{\de_j}d\mu\;\rho_i =\rho_i(x_j)+\thh_{ij}/n \end{equation*}$$ for some $$\thh_{ij}\in[0,1]$$. So, \begin{align} I(D,X_n)&=\sum_i\sum_j [\rho_i(x_j)+\thh_{ij}/n]\mu(\de_j)\ln\frac{\rho_i(x_j)+\thh_{ij}/n}{p_i} \\ &=\sum_i\sum_j \rho_i(x_j)\mu(\de_j)\ln\frac{\rho_i(x_j)}{p_i}+O(1/n) \\ &\to\sum_i\int_S d\mu\;\rho_i\ln\frac{\rho_i}{p_i} \end{align} as $$n\to\infty$$, which completes the justification of the definition (!).