Ok first, the entropy you're talking about is the differential entropy $-\int f(x) \ln f(x) d\mu(x)$. The problem is that $\mu$ is Lebesgue measure. The set of continuous probability distributions is the set of distributions that have a density (i.e. radon-nikodym derivative) wrt Lebesgue measure. As such, if your distribution does NOT have such a density, then there isn't really a meaningful interpretation of the above quantity. Not only that, if your distribution had a density $f$ wrt to some other measure $\nu$, and you plugged $f$ into the above, you'd accidentally be computing the differential entropy of some completely different distribution $f d\mu$, which is a continuous distribution.

So to answer the question: if you want to deal with non-continuous distributions, you have to tweak your definition of differential entropy. if the only difference you make is to substitute in some well-behaved measure $\nu$, even if the derivation goes through the same, the distribution you get out will be wrt $\nu$, ie not the same as $\phi d\mu$ where $\phi$ is the density of the gaussian. (how much of that derivation you can re-use depends on the measure you choose.)

PS a good reference on this stuff is cover&thomas's information theory book, which has a derivation of gaussian being the max (differential) entropy (continuous) distribution with constant variance.

Ok first, the entropy you're talking about is the differential entropy $-\int f(x) \ln f(x) d\mu(x)$. The problem is that $\mu$ is Lebesgue measure. The set of continuous probability distributions is the set of distributions that have a density (i.e. radon-nikodym derivative) wrt Lebesgue measure. As such, if your distribution does NOT have such a density, then there isn't really a meaningful interpretation of the above quantity. Not only that, if your distribution had a density $f$ wrt to some other measure $\nu$, and you plugged $f$ into the above, you'd accidentally be computing the differential entropy of some completely different distribution $f d\mu$, which is a continuous distribution.

So to answer the question: if you want to deal with non-continuous distributions, you have to tweak your definition of differential entropy. if the only difference you make is to substitute in some well-behaved measure $\nu$, even if the derivation goes through the same, the distribution you get out will be wrt $\nu$, ie not the same as $\phi d\mu$ where $\phi$ is the density of the gaussian. (how much of that derivation you can re-use depends on the measure you choose.)

PS a good reference on this stuff is cover&thomas's information theory book, which has a derivation of gaussian being the max (differential) entropy (continuous) distribution with constant variance.

EDIT I misunderstood the question; I thought it was asking about entropy for distributions without a density wrt Lebesgue; all it is asking for is a proof without any conditions on the density. Deane Yang provides such a proof in his answer to the question.

Ok first, the entropy you're talking about is the differential entropy $-\int f(x) \ln f(x) d\mu(x)$. The problem is that $\mu$ is Lebesgue measure. The set of continuous probability distributions is the set of distributions that have a density (i.e. radon-nikodym derivative) wrt Lebesgue measure. As such, if your distribution does NOT have such a density, then there isn't really a meaningful interpretation of the above quantity. Not only that, if your distribution had a density $f$ wrt to some other measure $\nu$, and you plugged $f$ into the above, you'd accidentally be computing the differential entropy of some completely different distribution $f d\mu$, which is a continuous distribution.

So to answer the question: if you want to deal with non-continuous distributions, you have to tweak your definition of differential entropy. if the only difference you make is to substitute in some well-behaved measure $\nu$, then the derivation will go through the same, but the distribution you get out will be wrt $\nu$, ie not the same as $\phi d\mu$ where $\phi$ is the density of the gaussian.

PS a good reference on this stuff is cover&thomas's information theory book, which has a derivation of gaussian being the max (differential) entropy (continuous) distribution with constant variance.

Ok first, the entropy you're talking about is the differential entropy $-\int f(x) \ln f(x) d\mu(x)$. The problem is that $\mu$ is Lebesgue measure. The set of continuous probability distributions is the set of distributions that have a density (i.e. radon-nikodym derivative) wrt Lebesgue measure. As such, if your distribution does NOT have such a density, then there isn't really a meaningful interpretation of the above quantity. Not only that, if your distribution had a density $f$ wrt to some other measure $\nu$, and you plugged $f$ into the above, you'd accidentally be computing the differential entropy of some completely different distribution $f d\mu$, which is a continuous distribution.

So to answer the question: if you want to deal with non-continuous distributions, you have to tweak your definition of differential entropy. if the only difference you make is to substitute in some well-behaved measure $\nu$, even if the derivation goes through the same, the distribution you get out will be wrt $\nu$, ie not the same as $\phi d\mu$ where $\phi$ is the density of the gaussian. (how much of that derivation you can re-use depends on the measure you choose.)

PS a good reference on this stuff is cover&thomas's information theory book, which has a derivation of gaussian being the max (differential) entropy (continuous) distribution with constant variance.