Question about information measurement for continuous random variable

Question

I have a question about the information measurement for continuous random variables. Maybe it is some basic question, but it really drives me crazy.

Suppose we have a random variable $x \sim N(0, 1)$ and another random variable $Y = 2X$, so that $Y \sim N(0, 4)$.

Then, let's compute some information measurements of $X$ and $Y$:

First, we can compute the entropy. Since both $X$ and $Y$ are normal distribution, we can compute the entropy easily: $H(X) = \ln(\sqrt{2 \pi e})$ and $H(Y) = \ln(2 \sqrt{2 \pi e})$. It is already strange, since there is only a linear mapping between $X$ and $Y$. We create some more information by just multiplying a constant.

Then, let's try to measure the mutual information between $X$ and $Y$. We know that $I(X, Y) = H(X) - H(X | Y) = H(Y) - H(Y | X)$. Since $Y = 2X$, when we know $X$, $Y$ is also known, vice versa. So we should have $H(X | Y) = H(Y | X) = 0$. Plug it into the above equation, we get $I(X, Y) = H(X) = H(Y)$. But it is in contrast with the entropy we compute in the first step.

What I am missing here?

Answer 1

The core issue is that Shannon's definition of differential entropy was a mistake - it doesn't have any of the nice properties that you would expect from the discrete case. Here are a couple other serious problems:

Take the random variable $X$ on $[0,1]$ with density function $f(x) = 2x$. Then:

$$H(X) = -\int_0^1 2x \log(2x)\, dx = \frac{1}{2} - \log 2 < 0$$

So the differential entropy can be negative. What is that supposed to mean?

It gets worse. Consider the same random variable $X$, but rewrite its density function in terms of the coordinate $u = x^2$. The CDF of $X$ in the $x$ coordinate system was $F(x) = x^2$, so the CDF in the $u$ coordinate system is $G(u) = F(\sqrt{u}) = u$, and hence the density function is $g(u) = 1$. So the differential entropy is now:

$$H(X) = -\int_0^1 1 \log 1\, dx = 0$$

So differential entropy depends on the coordinate system used to describe $X$.

Instead, the correct object of study for information theory in the continuous setting is the KL-divergence:

$$KL(X \Vert Y) = \int f(x) \log \frac{f(x)}{g(x)}\, dx$$

where $f$ and $g$ are the density functions of $X$ and $Y$, respectively. This quantity is nonnegative, with equality iff $X = Y$ almost everywhere, and it is invariant under coordinate change. Also, it generalizes to an arbitrary measure space $\Omega$: in the general case you pull back $X$ and $Y$ to probability measures on $\Omega$ and take the expectation of the Radon-Nikodym derivative.

Of course you'll ask: what is $Y$ supposed to be? How are we supposed to define $H(X)$?

If you have a translation invariant probability measure lying around, e.g. if you're working with random variables on a compact interval, then use that! If not, Edwin Jaynes argued that the correct definition of entropy for a continuous random variable should involve a limiting process in order to arrive at an object which behaves like discrete entropy.

It is actually quite clarifying to view entropy as an inherently relative object - in the discrete case we tend not to notice because we can implicitly work relative to the uniform distribution.

Answer 2

There is actually nothing weird about this at all. What you have discovered is that for continuous quantities, information is scale-relative. Or, to put it another way, the "true" entropy of any continuous quantity not known exactly (or, known down to a finite number of exact candidates) is $\infty$. Which, of course, should make intuitive sense to you - it takes an infinite number of digits to name an arbitrary real number, and there are an infinite number of potential candidates (uncountably infinitely many actually, $\beth_1$), whenever you have a whole continuous smear of a probability distribution that is a nontrivial continuous function.

But, of course, just assigning "$\infty$" all the time would not be very useful. Hence, we have to introduce some form of relativity, by which we measure the information versus a moveable reference level. It's kind of like decibel measurements - $0\ \mathrm{dB}$ as an absolute measure must always be set arbitrarily for similar reasons.

The simple definition of entropy you give, i.e.

$$H[X] = -\int_S f_X(x) \lg f_X(x)\ dx$$

does this in a very simple manner, which can be "reverse engineered" from it by feeding it a suitably-wide uniform distribution: namely if you feed it

$$f_X(x) = \begin{cases}\frac{1}{b - a},\ \text{if $x \in [a, b]$}\\0,\ \text{otherwise}\end{cases}$$

you will find that it gives you $\lg(b - a)$ and this is 1 bit exactly when $b - a$ is 2, in other words, when you have a situation in which the position is known to within two units. When it is two bits, it is unknown to four units and, in general, $n$ bits, unknown to $2^n$ units.

That is, we have a uniform grid, formed by whatever measuring unit we are measuring $x$ in (e.g. meters, centimeters, millimeters, whatever), and the entropy is how much worse you are informed compared to knowing "to the nearest unit", as in a simple "digital" or "quantized" sense. Hence, it is no surprise you can have a negative entropy if you know the position better than the nearest unit! E.g. if your base unit is meters and you know it to $\frac{1}{128}$ meter, then you should have an entropy of -7 bits and, indeed, you do! But the entropy can keep getting more negative, which is what I was saying before, because there's still infinitely more information (at least in theory) you could still have yet to know about where exactly that thing is (or isn't) located!

And if you want to make this more explicit, if you have a set reference scale you want that is different from your measuring unit, i.e. you want "entropy relative to 0.001 m" when your positions are measured in m, you just add that to $H[X]$:

$$H_\mathrm{adjust}[X] = -\int_S f_X(x) \lg f_X(x)\ dx - H_0$$

where $H_0 = \lg(l_\mathrm{ref})$, and $l_\mathrm{ref}$ is your reference length measured in your unit of choice. So if your measured in meters that you know to 2 m, i.e. $H[X]$ is 1 bit, then relative to 0.001 m, where that $H_0 \approx -9.96$ then $H_\mathrm{adjust}[X] \approx 10.96\ \mathrm{bit}$.