How is entropy of a general probability measure defined?
closed as off topic by Gjergji Zaimi, Steve Huntsman, Andrey Rekalo, Qiaochu Yuan, Yemon Choi Jul 23 '10 at 20:31Questions on MathOverflow are expected to relate to research level mathematics within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here. If this question can be reworded to fit the rules in the help center, please edit the question. 


The Entropy of a function $f$ with respect to a measure $\mu$ is $$ Ent_{\mu}(f)=\int f \log f d\mu  \int f d\mu \log(\int f d\mu ) $$ The entropy of a probability distribution $P$ with respect to $\mu$ is given by $ Ent(\frac{dP }{d\mu })$. I a not aware of a general definition that would not implie a reference (here it is $\mu$) measure ... 


Entropy is not defined for a single probability measure! Entropy is a relative thing, you define between two measure $\mu$ and $\nu$. Then the entropy is defined by $$ \mathcal{E}(\mu,\nu) = \begin{cases}  \int w(x) \log(w(x)) d\nu(x),& d \mu = w d\nu; \\\ \infty, & otherwise.\end{cases}, $$ I might have messed up the signs. $w$ is the RadonNikodym derivate of $\mu$ with respect to $\nu$. 


It is not. If a probability measure on $\mathbb{R}$ is absolutely continuous and has density $f$, then "entropy" usually refers to the differential entropy, defined in the Wikipedia page falagar linked to. If the probability measure has discrete support, entropy is defined by an analogous formula, given in this Wikipedia page. In the most classical treatments, these are the only situations covered at all. However, both of these are special cases of the more general notion of relative entropy that Helge and robin girard pointed out: in the continuous case the reference measure ($\nu$ in Helge's notation, $\mu$ in robin's notation) is Lebesgue measure, and in the discrete case the reference measure is counting measure on the support. 

