Cramér–Rao type bound for absolute estimation error

Question

Let $\{X_1, X_2, \dotsc, X_n\}$ be independent and identically distributed (i.i.d.) random variables sampled from a common distribution with density $f_{\theta}(x)$, where $\theta$ is an unknown parameter. We want to estimate $\theta$ given these $n$ samples. Suppose $\hat{\theta}$ is an estimator based on these samples. For simplicity, suppose this is unbiased, so that $E[\hat{\theta}] = \theta$.

Cramér–Rao bound theory implies that for any unbiased estimator: $$ E[(\hat{\theta} - \theta)^2] \geq \frac{1}{I(\theta)} = \Theta(1/n) $$ where $I(\theta)$ is the Fisher information. However, I am interested not in the mean-square error, but the mean absolute error: $$ E[|\hat{\theta} - \theta|] \geq {???} $$

This must be a well-studied problem. Any references or insights on this would be helpful.

---

Intuitively one expects $E[|\hat{\theta}-\theta|]\geq \Theta(1/\sqrt{n})$, and this is what I eventually want to show for my particular context (actually, eventually I am interested in possibly biased estimators). If one assumes the absolute error is at most $M$ then: $$ \Theta(1/n) \leq E[(\hat{\theta}-\theta)^2] \leq ME[|\hat{\theta}-\theta|] $$ but this inequality is weaker than I want since it means the absolute error also decays by at most $\Theta(1/n)$, whereas I want to increase the bound to $\Theta(1/\sqrt{n})$.

---

Actually, I can prove something of this form in a special case when $\theta$ represents the mean $E[X_1]$. I'm wondering if such a thing is known? Estimating the mean leads to the "obvious" estimator $\hat{\theta}=\frac{1}{n}\sum_{i=1}^nX_i$, but it is not obvious how to show this is "best" in some sense, particularly for the mean-absolute-error.

Answer 1

The derivation of the Cramér–Rao lower bound in Kay uses the weighted Cauchy–Schwarz inequality: $$ \left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x} $$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$.

Instead, we can use Hölder's more general inequality: $$ \left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \left( \int w(\mathbf{x})\left|g(\mathbf{x})\right|^pd\mathbf{x}\right)^{\frac{1}{p}} \left(\int w(\mathbf{x}) \left|h(\mathbf{x}) \right|^q d\mathbf{x} \right)^\frac{1}{q} $$ where $\frac{1}{p}+\frac{1}{q} = 1$. Cauchy's inequality is the special case $p = q = 2$.

If the estimator is unbiased: $$ E[\hat{\theta}] = \theta $$ or $$ \int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta $$

differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta) $ yields:

$$ \int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1. $$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $ \displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0 $ and then we have:

$$ \int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1. $$

Using Hölder's inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$ 1 \leq \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q}. $$

In the limit $p \rightarrow 1$, $q \rightarrow \infty$ $$ \lim_{p \to 1} \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} = E\left[ \left|(\hat{\theta} -\theta)\right| \right] $$

$$ \lim_{q \to \infty} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} = \sup \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|. $$

Rearranging: $$ E\left[ \left|(\hat{\theta} -\theta)\right| \right] \geq \frac{1}{\sup\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|}. $$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$, but the supremum of $\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|$.

Answer 2

Let's define the nonnegative random variable $X$ as $$ X := |\hat \theta - \theta| \geq 0 .$$

Markov's inequality states that: if $X$ is a nonnegative random variable and $a > 0$, then $$ \operatorname {Pr} (X\geq a)\leq {\frac {\operatorname {E} (X)}{a}}. $$

Therefore:

Result 1 $$ {\operatorname {E}[ |\hat \theta - \theta| ]} \geq a \cdot \operatorname {Pr} (|\hat \theta - \theta| \geq a). $$

A second result uses the corollary to Markov on the quantile function which states that: for a nonnegative random variable $X$, the quantile function of $X$ satisfies $$ Q_{X}(1-p) \leq {\frac {\operatorname {E} (X)}{p}}, \quad 0 < p < 1. $$ Therefore:

Result 2 $$ {\operatorname {E} [|\hat \theta - \theta|]} \geq p \cdot Q_{|\hat \theta - \theta|}(1-p), \quad 0 < p < 1. $$

Note: Regretfully, I do not currently have an idea how to derive any asymptotic formula, neither $\Theta(1/n)$ nor $\Theta(1/ \sqrt n)$, for any of my 2 results. All suggestions are welcome.