Return to Answer

The derivation of the Cramer-Rao lower bound in Kay uses the weighted Cauchy-Schwarz inequality: $$ \left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x} $$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$.

Instead, we can use Holder's more general inequality: $$ \left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \left( \int w(\mathbf{x})\left|g(\mathbf{x})\right|^pd\mathbf{x}\right)^{\frac{1}{p}} \left(\int w(\mathbf{x}) \left|h(\mathbf{x}) \right|^q d\mathbf{x} \right)^\frac{1}{q} $$ where $\frac{1}{p}+\frac{1}{q} = 1$. Cauchy's inequality is the special case $p = q = 2$

If the estimator is unbiased: $$ E[\hat{\theta}] = \theta $$ or $$ \int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta $$

Differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta) $ yields:

$$ \int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $ \displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0 $ and then we have:

$$ \int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

Using Holder's inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$ 1 \leq \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} $$

In the limit $p \rightarrow 1$, $q \rightarrow \infty$ $$ \lim_{p \to 1} \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} = E\left[ \left|(\hat{\theta} -\theta)\right| \right] $$

$$ \lim_{q \to \infty} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} = \sup \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right| $$

Rearranging: $$ E\left[ \left|(\hat{\theta} -\theta)\right| \right] \geq \frac{1}{\sup\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|} $$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$, but the supremum of $\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|$

The derivation of the Cramér–Rao lower bound in Kay uses the weighted Cauchy–Schwarz inequality: $$ \left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x} $$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$.

Instead, we can use Hölder's more general inequality: $$ \left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \left( \int w(\mathbf{x})\left|g(\mathbf{x})\right|^pd\mathbf{x}\right)^{\frac{1}{p}} \left(\int w(\mathbf{x}) \left|h(\mathbf{x}) \right|^q d\mathbf{x} \right)^\frac{1}{q} $$ where $\frac{1}{p}+\frac{1}{q} = 1$. Cauchy's inequality is the special case $p = q = 2$.

If the estimator is unbiased: $$ E[\hat{\theta}] = \theta $$ or $$ \int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta $$

differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta) $ yields:

$$ \int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1. $$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $ \displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0 $ and then we have:

$$ \int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1. $$

Using Hölder's inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$ 1 \leq \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q}. $$

In the limit $p \rightarrow 1$, $q \rightarrow \infty$ $$ \lim_{p \to 1} \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} = E\left[ \left|(\hat{\theta} -\theta)\right| \right] $$

$$ \lim_{q \to \infty} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} = \sup \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|. $$

Rearranging: $$ E\left[ \left|(\hat{\theta} -\theta)\right| \right] \geq \frac{1}{\sup\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|}. $$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$, but the supremum of $\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|$.

The derivation of the Cramer-Rao lower bound in Kay uses the weighted Cauchy-Schwarz inequality: $$ \left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x} $$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$. I don't have a source, but it seems reasonable that the same inequality would hold for absolute values (may be a special case of the more general Holder inequality?):

$$ \left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \int w(\mathbf{x})\left|g(\mathbf{x})\right|d\mathbf{x} \int w(\mathbf{x}) \left|h(\mathbf{x}) \right| d\mathbf{x} $$

If the estimator is unbiased: $$ E[\hat{\theta}] = \theta $$ or $$ \int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta $$

Differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta) $ yields:

$$ \int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $ \displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0 $ and then we have:

$$ \int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

Here our derivation differs from Kay by applying the modified Cauchy-Schwartz inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$ 1 \leq \int \left|(\hat{\theta} - \theta)\right| p(\mathbf{x}; \theta) \: d\mathbf{x} \int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right| p(\mathbf{x}; \theta) d\mathbf{x} $$

Rearranging: $$ E\left[ \left|(\hat{\theta} -\theta)\right| \right] \geq \frac{1}{E\left[\left|\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right| \right]} $$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$

The derivation of the Cramer-Rao lower bound in Kay uses the weighted Cauchy-Schwarz inequality: $$ \left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x} $$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$.

Instead, we can use Holder's more general inequality: $$ \left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \left( \int w(\mathbf{x})\left|g(\mathbf{x})\right|^pd\mathbf{x}\right)^{\frac{1}{p}} \left(\int w(\mathbf{x}) \left|h(\mathbf{x}) \right|^q d\mathbf{x} \right)^\frac{1}{q} $$ where $\frac{1}{p}+\frac{1}{q} = 1$. Cauchy's inequality is the special case $p = q = 2$

If the estimator is unbiased: $$ E[\hat{\theta}] = \theta $$ or $$ \int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta $$

Differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta) $ yields:

$$ \int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $ \displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0 $ and then we have:

$$ \int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

Using Holder's inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$ 1 \leq \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} $$

In the limit $p \rightarrow 1$, $q \rightarrow \infty$ $$ \lim_{p \to 1} \left(\int \left|(\hat{\theta} - \theta)\right|^p p(\mathbf{x}; \theta) \: d\mathbf{x} \right)^\frac{1}{p} = E\left[ \left|(\hat{\theta} -\theta)\right| \right] $$

$$ \lim_{q \to \infty} \left(\int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|^q p(\mathbf{x}; \theta) d\mathbf{x} \right)^\frac{1}{q} = \sup \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right| $$

Rearranging: $$ E\left[ \left|(\hat{\theta} -\theta)\right| \right] \geq \frac{1}{\sup\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|} $$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$, but the supremum of $\left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right|$

The derivation of the Cramer-Rao lower bound in Kay uses the weighted Cauchy-Schwarz inequality: $$ \left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x} $$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$. I don't have a source, but it seems reasonable that the same inequality would hold for absolute values (may be a special case of the more general Holder inequality?):

$$ \left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \int w(\mathbf{x})\left|g(\mathbf{x})\right|d\mathbf{x} \int w(\mathbf{x}) \left|h(\mathbf{x}) \right| d\mathbf{x} $$

If the estimator is unbiased: $$ E[\hat{\theta}] = \theta $$ or $$ \int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta $$

Differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta) $ yields:

$$ \int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $ \displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0 $ and then we have:

$$ \int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

Here our derivation differs from Kay by applying the modified Cauchy-Schwartz inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$ 1 \leq \int \left|(\hat{\theta} - \theta)\right| p(\mathbf{x}; \theta) \: d\mathbf{x} \int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right| p(\mathbf{x}; \theta) d\mathbf{x} $$

Rearranging: $$ E\left[ (\hat{\theta} -\theta) \right] \geq \frac{1}{E\left[\left|\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right| \right]} $$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$

The derivation of the Cramer-Rao lower bound in Kay uses the weighted Cauchy-Schwarz inequality: $$ \left[ \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right]^2 \leq \int w(\mathbf{x})g^2(\mathbf{x})d\mathbf{x} \int w(\mathbf{x}) h^2(\mathbf{x}) d\mathbf{x} $$

where $g$ and $h$ are arbitrary scalar functions, and $w(\mathbf{x}) \geq 0$ for all $\mathbf{x}$. I don't have a source, but it seems reasonable that the same inequality would hold for absolute values (may be a special case of the more general Holder inequality?):

$$ \left| \int w(\mathbf{x})g(\mathbf{x})h(\mathbf{x})d\mathbf{x} \right| \leq \int w(\mathbf{x})\left|g(\mathbf{x})\right|d\mathbf{x} \int w(\mathbf{x}) \left|h(\mathbf{x}) \right| d\mathbf{x} $$

If the estimator is unbiased: $$ E[\hat{\theta}] = \theta $$ or $$ \int \hat{\theta} \: p(\mathbf{x};\theta) \: d\mathbf{x} = \theta $$

Differentiating with respect to $\theta$ and using $\dfrac{\partial p(\mathbf{x}; \theta)}{\partial \theta} = \dfrac{\partial \ln p(\mathbf{x};\theta)}{\partial \theta} p(\mathbf{x};\theta) $ yields:

$$ \int \hat{\theta} \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

$\hat{\theta}$ in this expression can be replaced with $(\hat{\theta} - \theta)$ because the CRLB assumes the regularity condition $\displaystyle E\left[\frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0$, so $ \displaystyle \int \theta \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = \theta \: E\left[\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right] = 0 $ and then we have:

$$ \int (\hat{\theta} - \theta) \: \dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} p(\mathbf{x}; \theta) \: d\mathbf{x} = 1 $$

Here our derivation differs from Kay by applying the modified Cauchy-Schwartz inequality with $w(\mathbf{x}) = p(\mathbf{x};\theta)$, $g(\mathbf{x}) = \hat{\theta} - \theta$, and $h(\mathbf{x}) = \dfrac{\partial \ln p(\mathbf{x};\theta}{\partial \theta}$

$$ 1 \leq \int \left|(\hat{\theta} - \theta)\right| p(\mathbf{x}; \theta) \: d\mathbf{x} \int \left| \frac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta} \right| p(\mathbf{x}; \theta) d\mathbf{x} $$

Rearranging: $$ E\left[ \left|(\hat{\theta} -\theta)\right| \right] \geq \frac{1}{E\left[\left|\dfrac{\partial \ln p(\mathbf{x}; \theta)}{\partial \theta}\right| \right]} $$

The expectation in the denominator is no longer the Fisher Information $I(\theta)$