# Cramer-Rao type bound for absolute estimation error

Let $\{X_1, X_2, \ldots, X_n\}$ be independent and identically distributed (i.i.d.) random variables sampled from a common distribution with density $f_{\theta}(x)$, where $\theta$ is an unknown parameter. We want to estimate $\theta$ given these $n$ samples. Suppose $\hat{\theta}$ is an estimator based on these samples. For simplicity, suppose this is unbiased, so that $E[\hat{\theta}] = \theta$.

Cramer-Rao bound theory implies that for any unbiased estimator: $$E[(\hat{\theta} - \theta)^2] \geq \frac{1}{I(\theta)} = \Theta(1/n)$$ where $I(\theta)$ is the Fisher information. However, I am interested not in the mean-square error, but the mean absolute error: $$E[|\hat{\theta} - \theta|] \geq ???$$

This must be a well-studied problem. Any references or insights on this would be helpful.

Intuitively one expects $E[|\hat{\theta}-\theta|]\geq \Theta(1/\sqrt{n})$, and this is what I eventually want to show for my particular context (actually, eventually I am interested in possibly biased estimators). If one assumes the absolute error is at most $M$ then: $$\Theta(1/n) \leq E[(\hat{\theta}-\theta)^2] \leq ME[|\hat{\theta}-\theta|]$$ but this inequality is weaker than I want since it means the absolute error also decays by at most $\Theta(1/n)$, whereas I want to increase the bound to $\Theta(1/\sqrt{n})$.

Actually, I can prove something of this form in a special case when $\theta$ represents the mean $E[X_1]$. I'm wondering if such a thing is known? Estimating the mean leads to the "obvious" estimator $\hat{\theta}=\frac{1}{n}\sum_{i=1}^nX_i$, but it is not obvious how to show this is "best" in some sense, particularly for the mean-absolute-error.

## 1 Answer

The derivation of the Cramer-Rao lower bound in Kay uses the weighted Cauchy-Schwarz inequality: $$\left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x}$$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$.

Instead, we can use Holder's more general inequality: $$\left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \left( \int w(\mathbf{x})\left|g(\mathbf{x})\right|^pd\mathbf{x}\right)^{\frac{1}{p}} \left(\int w(\mathbf{x}) \left|h(\mathbf{x}) \right|^q d\mathbf{x} \right)^\frac{1}{q}$$ where $\frac{1}{p}+\frac{1}{q} = 1$. Cauchy's inequality is the special case $p = q = 2$

If the estimator is unbiased: $$E[\hat{\theta}] = \theta$$ or $$\int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta$$

Differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta)$ yields:

$$\int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1$$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $\displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$ and then we have:

$$\int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1$$

Using Holder's inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$1 \leq \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q}$$

In the limit $p \rightarrow 1$, $q \rightarrow \infty$ $$\lim_{p \to 1} \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} = E\left[ \left|(\hat{\theta} -\theta)\right| \right]$$

$$\lim_{q \to \infty} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} = \sup \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|$$

Rearranging: $$E\left[ \left|(\hat{\theta} -\theta)\right| \right] \geq \frac{1}{\sup\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|}$$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$, but the supremum of $\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|$

• Thanks for your interest! I asked this 6 months ago, so I need to recall my line of thought on this problem. I will read over your answer this week when I get a chance. – Michael Nov 24 '15 at 4:12
• Well, your absolute value inequality does not hold in general. For random variables $G,H$, it reduces to the claim $|E[GH]| \leq E[|G|]E[|H|]$. But let $G=H$ and let $G$ be a nonnegative random variable. This reduces to $E[G^2] \leq E[G]^2$, but this is violated whenever $G$ has nonzero variance. – Michael Nov 24 '15 at 4:24
• I am definitely on weakest ground regarding the inequality, but taking your special case of $G = H$ reduces to $E[G^2] \leq E[|G|]^2$ (keeping the absolute value) which seems like it might hold conditionally on $E[G] = 0$. – Sealander Nov 24 '15 at 10:09
• The inequality was kind of a shot in the dark and the more I think about it, the more I don't think it holds even if you restrict it to zero-mean random variables. – Sealander Nov 24 '15 at 11:16
• Note that: $$E[|G|^2]=E[|G|]^2 \iff Var(|G|)=0 \iff \mbox{|G| is constant with prob 1}$$ – Michael Nov 24 '15 at 19:26