Mutual information between continuous and discrete variables from numerical data I am looking for references/measures to estimate the mutual information between a continuous (C) and discrete (D) variable, given a real-world (i.e. finite sample) data set. C is uniformly distributed between Cmin and Cmax, and D is a positive integer whose distribution depends on the value of C (e.g., a poisson distribution with mean f(C)). Any suggestions (methods based on binning over the data set, or independent of binning) are welcome. 
 A: $\newcommand{\de}{\delta}
\newcommand{\ep}{\varepsilon}
\newcommand{\thh}{\theta}$
Let $D$ be any discrete random variable (r.v.) with distinct values $d_i$ taken with probabilities $p_i=P(D=d_i)>0$ for $i\in I$, where $I$ is a denumerable (that is, at most countable) set. Let $X$ be any r.v. (defined on the same probability space as $D$), with values in any nonempty set $S$ (given also some sigma-algebra $\Sigma$ over $S$, to make $S$ a measurable space). Let $\mu$ be the probability distribution of $X$, so that $\mu(B)=P(X\in B)$ for all $B\in\Sigma$. For each $i\in I$ and each $B\in\Sigma$, let 
\begin{equation*}
 \mu_i(B):=P(D=d_i,X\in B). 
\end{equation*}
Then $\mu_i$ is a (sub-probability) measure absolutely continuous with respect to $\mu$, so that we can consider a Radon--Nikodym density 
\begin{equation*}
 \rho_i:=\frac{d\mu_i}{d\mu} \tag{1}
\end{equation*}
of the measure $\mu_i$ with respect to $\mu$, so that the values of $\rho_i$ are in $[0,1]$. 
Then the mutual information between $D$ and $X$ may/should be defined as follows:
\begin{equation*}
 I(D,X):=\sum_{i\in I}\int_S d\mu\;\rho_i\ln\frac{\rho_i}{p_i}. \tag{!}
\end{equation*}
Justification: Take any natural $n$. Let $J:=J_n:=\{0,\dots,n\}^I$, the set of families $j=(j_i)_{i\in I}$ with $j_i\in\{0,\dots,n\}$. Let us assume that $I$ is finite, so that $J$ is countable. 
For any $j\in J$, let 
\begin{equation*}
 \de_j:=\de_j(n):=\{x\in S\colon j_i/n\le\rho_i(x)<(j_i+1)/n\ \forall i\in I\}. \tag{2}
\end{equation*}
Let $J_{n;0}:=\{j\in J\colon\de_j\ne\emptyset\}$. 
Note that the sets $\de_j$ with $j\in J_{n;0}$ form a partition of $S$. 
For each $j\in J_{n;0}$, pick then an arbitrary $x_j\in\de_j$, and let $X_n$ be the r.v. taking value the constant value $x_j$ on $\de_j$ for each $j\in J_{n;0}$, so that $X_n$ is a discrete approximation to the r.v. $X$. Then, according to the standard definition, the mutual information between $D$ and $X_n$ is
\begin{align}
 I(D,X_n)&:=\sum_i\sum_j P(D=d_i,X_n=x_j)\ln\frac{P(D=d_i,X_n=x_j)}{P(D=d_i)P(X_n=x_j)} \\ 
 &=\sum_i\sum_j P(D=d_i,X\in\de_j)\ln\frac{P(D=d_i,X\in\de_j)}{P(D=d_i)P(X\in\de_j)} \\   
 &=\sum_i\sum_j \mu_i(\de_j)\ln\frac{\mu_i(\de_j)}{p_i\mu(\de_j)};   
\end{align} 
here and in what follows, $\sum_i:=\sum_{i\in I}$ and $\sum_j:=\sum_{j\in J_{n;0}}$; as usual, the $(i,j)$-summand here is understood as $0$ whenever $\mu_i(\de_j)=0$. Next, by (1) and (2), 
\begin{equation*}
 \frac{\mu_i(\de_j)}{\mu(\de_j)}=\frac1{\mu(\de_j)}\,\int_{\de_j}d\mu\;\rho_i
 =\rho_i(x_j)+\thh_{ij}/n
\end{equation*}
for some $\thh_{ij}\in[0,1]$. So, 
\begin{align}
 I(D,X_n)&=\sum_i\sum_j [\rho_i(x_j)+\thh_{ij}/n]\mu(\de_j)\ln\frac{\rho_i(x_j)+\thh_{ij}/n}{p_i} \\  
 &=\sum_i\sum_j \rho_i(x_j)\mu(\de_j)\ln\frac{\rho_i(x_j)}{p_i}+O(1/n) \\    
 &\to\sum_i\int_S d\mu\;\rho_i\ln\frac{\rho_i}{p_i}    
\end{align} 
as $n\to\infty$, which completes the justification of the definition (!). 
