2
$\begingroup$

On page 255 of the book "Elements of information theory" by Thomas M. Cover and Joy A. Thomas, there is a theorem: For any random variable $X$ and estimator $\hat{X}$, $$E(X-\hat{X})^2 \geq \min_{\hat{X}} E(X-\hat{X})^2 = E(X-E(X))^2 = \mathrm{var}(X) \geq \frac{1}{2\pi e}e^{2h(X)}.$$ Right after theorem there is a corollary relating estimation error and differential entropy: Given random variable $X$, side information $Y$ and estimator $\hat{X}(Y)$, $E(X-\hat{X}(Y))^2 \geq \frac{1}{2\pi e}e^{2h(X|Y)}$.

However, I'm not sure whether the corollary is true for multi-variables: Given multi random variable $X \in \mathbb{R}^m, m > 1 $, side information $Y \in \mathbb{R}^n, n \geq 1 $ and estimator $\hat{X}(Y)$, $E(|X-\hat{X}(Y)|^2) \geq \frac{1}{2\pi e}e^{2h(X|Y)}$, where $|\cdot|$ denotes the 2-norm.

$\endgroup$

1 Answer 1

3
$\begingroup$

You are missing dimensional constants, but you have the right idea. Namely, it is true that $$ E|X-\hat{X}(Y)|^2 \geq \frac{m}{2\pi e} e^{2 h(X|Y)/m }. $$ Like in the 1-dimensional case, this is a consequence of Gaussians maximizing entropy subject to a second moment constraint. In the multidimensional case you will also invoke the AM-GM inequality to bound the determinant of the covariance by the trace (to arrive at the $L^2$ error on the LHS).

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .