2
$\begingroup$

On page 255 of the book "Elements of information theory" by Thomas M. Cover and Joy A. Thomas, there is a theorem: For any random variable $X$ and estimator $\hat{X}$, $$E(X-\hat{X})^2 \geq \min_{\hat{X}} E(X-\hat{X})^2 = E(X-E(X))^2 = \mathrm{var}(X) \geq \frac{1}{2\pi e}e^{2h(X)}.$$ Right after theorem there is a corollary relating estimation error and differential entropy: Given random variable $X$, side information $Y$ and estimator $\hat{X}(Y)$, $E(X-\hat{X}(Y))^2 \geq \frac{1}{2\pi e}e^{2h(X|Y)}$.

However, I'm not sure whether the corollary is true for multi-variables: Given multi random variable $X \in \mathbb{R}^m, m > 1 $, side information $Y \in \mathbb{R}^n, n \geq 1 $ and estimator $\hat{X}(Y)$, $E(|X-\hat{X}(Y)|^2) \geq \frac{1}{2\pi e}e^{2h(X|Y)}$, where $|\cdot|$ denotes the 2-norm.

$\endgroup$

1 Answer 1

3
$\begingroup$

You are missing dimensional constants, but you have the right idea. Namely, it is true that $$ E|X-\hat{X}(Y)|^2 \geq \frac{m}{2\pi e} e^{2 h(X|Y)/m }. $$ Like in the 1-dimensional case, this is a consequence of Gaussians maximizing entropy subject to a second moment constraint. In the multidimensional case you will also invoke the AM-GM inequality to bound the determinant of the covariance by the trace (to arrive at the $L^2$ error on the LHS).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.