Cramer-Rao type bound for absolute estimation error

Question

Let $\{X_1, X_2, \ldots, X_n\}$ be independent and identically distributed (i.i.d.) random variables sampled from a common distribution with density $f_{\theta}(x)$, where $\theta$ is an unknown parameter. We want to estimate $\theta$ given these $n$ samples. Suppose $\hat{\theta}$ is an estimator based on these samples. For simplicity, suppose this is unbiased, so that $E[\hat{\theta}] = \theta$.

Cramer-Rao bound theory implies that for any unbiased estimator: $$ E[(\hat{\theta} - \theta)^2] \geq \frac{1}{I(\theta)} = \Theta(1/n) $$ where $I(\theta)$ is the Fisher information. However, I am interested not in the mean-square error, but the mean absolute error: $$ E[|\hat{\theta} - \theta|] \geq ??? $$

This must be a well-studied problem. Any references or insights on this would be helpful.

---

Intuitively one expects $E[|\hat{\theta}-\theta|]\geq \Theta(1/\sqrt{n})$, and this is what I eventually want to show for my particular context (actually, eventually I am interested in possibly biased estimators). If one assumes the absolute error is at most $M$ then: $$ \Theta(1/n) \leq E[(\hat{\theta}-\theta)^2] \leq ME[|\hat{\theta}-\theta|] $$ but this inequality is weaker than I want since it means the absolute error also decays by at most $\Theta(1/n)$, whereas I want to increase the bound to $\Theta(1/\sqrt{n})$.

---

Actually, I can prove something of this form in a special case when $\theta$ represents the mean $E[X_1]$. I'm wondering if such a thing is known? Estimating the mean leads to the "obvious" estimator $\hat{\theta}=\frac{1}{n}\sum_{i=1}^nX_i$, but it is not obvious how to show this is "best" in some sense, particularly for the mean-absolute-error.

Answer 1

The derivation of the Cramer-Rao lower bound in Kay uses the weighted Cauchy-Schwarz inequality: $$ \left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x} $$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$.

Instead, we can use Holder's more general inequality: $$ \left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \left( \int w(\mathbf{x})\left|g(\mathbf{x})\right|^pd\mathbf{x}\right)^{\frac{1}{p}} \left(\int w(\mathbf{x}) \left|h(\mathbf{x}) \right|^q d\mathbf{x} \right)^\frac{1}{q} $$ where $\frac{1}{p}+\frac{1}{q} = 1$. Cauchy's inequality is the special case $p = q = 2$

If the estimator is unbiased: $$ E[\hat{\theta}] = \theta $$ or $$ \int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta $$

Differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta) $ yields:

$$ \int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $ \displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0 $ and then we have:

$$ \int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

Using Holder's inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$ 1 \leq \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} $$

In the limit $p \rightarrow 1$, $q \rightarrow \infty$ $$ \lim_{p \to 1} \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} = E\left[ \left|(\hat{\theta} -\theta)\right| \right] $$

$$ \lim_{q \to \infty} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} = \sup \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right| $$

Rearranging: $$ E\left[ \left|(\hat{\theta} -\theta)\right| \right] \geq \frac{1}{\sup\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|} $$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$, but the supremum of $\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|$