4
$\begingroup$

Let $\{X_1, X_2, \ldots, X_n\}$ be independent and identically distributed (i.i.d.) random variables sampled from a common distribution with density $f_{\theta}(x)$, where $\theta$ is an unknown parameter. We want to estimate $\theta$ given these $n$ samples. Suppose $\hat{\theta}$ is an estimator based on these samples. For simplicity, suppose this is unbiased, so that $E[\hat{\theta}] = \theta$.

Cramer-Rao bound theory implies that for any unbiased estimator: $$ E[(\hat{\theta} - \theta)^2] \geq \frac{1}{I(\theta)} = \Theta(1/n) $$ where $I(\theta)$ is the Fisher information. However, I am interested not in the mean-square error, but the mean absolute error: $$ E[|\hat{\theta} - \theta|] \geq ??? $$

This must be a well-studied problem. Any references or insights on this would be helpful.


Intuitively one expects $E[|\hat{\theta}-\theta|]\geq \Theta(1/\sqrt{n})$, and this is what I eventually want to show for my particular context (actually, eventually I am interested in possibly biased estimators). If one assumes the absolute error is at most $M$ then: $$ \Theta(1/n) \leq E[(\hat{\theta}-\theta)^2] \leq ME[|\hat{\theta}-\theta|] $$ but this inequality is weaker than I want since it means the absolute error also decays by at most $\Theta(1/n)$, whereas I want to increase the bound to $\Theta(1/\sqrt{n})$.


Actually, I can prove something of this form in a special case when $\theta$ represents the mean $E[X_1]$. I'm wondering if such a thing is known? Estimating the mean leads to the "obvious" estimator $\hat{\theta}=\frac{1}{n}\sum_{i=1}^nX_i$, but it is not obvious how to show this is "best" in some sense, particularly for the mean-absolute-error.

$\endgroup$
1
$\begingroup$

The derivation of the Cramer-Rao lower bound in Kay uses the weighted Cauchy-Schwarz inequality: $$ \left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x} $$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$.

Instead, we can use Holder's more general inequality: $$ \left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \left( \int w(\mathbf{x})\left|g(\mathbf{x})\right|^pd\mathbf{x}\right)^{\frac{1}{p}} \left(\int w(\mathbf{x}) \left|h(\mathbf{x}) \right|^q d\mathbf{x} \right)^\frac{1}{q} $$ where $\frac{1}{p}+\frac{1}{q} = 1$. Cauchy's inequality is the special case $p = q = 2$

If the estimator is unbiased: $$ E[\hat{\theta}] = \theta $$ or $$ \int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta $$

Differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta) $ yields:

$$ \int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $ \displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0 $ and then we have:

$$ \int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

Using Holder's inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$ 1 \leq \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} $$

In the limit $p \rightarrow 1$, $q \rightarrow \infty$ $$ \lim_{p \to 1} \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} = E\left[ \left|(\hat{\theta} -\theta)\right| \right] $$

$$ \lim_{q \to \infty} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} = \sup \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right| $$

Rearranging: $$ E\left[ \left|(\hat{\theta} -\theta)\right| \right] \geq \frac{1}{\sup\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|} $$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$, but the supremum of $\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|$

$\endgroup$
  • $\begingroup$ Thanks for your interest! I asked this 6 months ago, so I need to recall my line of thought on this problem. I will read over your answer this week when I get a chance. $\endgroup$ – Michael Nov 24 '15 at 4:12
  • $\begingroup$ Well, your absolute value inequality does not hold in general. For random variables $G,H$, it reduces to the claim $|E[GH]| \leq E[|G|]E[|H|]$. But let $G=H$ and let $G$ be a nonnegative random variable. This reduces to $E[G^2] \leq E[G]^2$, but this is violated whenever $G$ has nonzero variance. $\endgroup$ – Michael Nov 24 '15 at 4:24
  • $\begingroup$ I am definitely on weakest ground regarding the inequality, but taking your special case of $G = H$ reduces to $E[G^2] \leq E[|G|]^2$ (keeping the absolute value) which seems like it might hold conditionally on $E[G] = 0$. $\endgroup$ – Sealander Nov 24 '15 at 10:09
  • $\begingroup$ The inequality was kind of a shot in the dark and the more I think about it, the more I don't think it holds even if you restrict it to zero-mean random variables. $\endgroup$ – Sealander Nov 24 '15 at 11:16
  • $\begingroup$ Note that: $$E[|G|^2]=E[|G|]^2 \iff Var(|G|)=0 \iff \mbox{$|G|$ is constant with prob 1}$$ $\endgroup$ – Michael Nov 24 '15 at 19:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.