2
$\begingroup$

My question is:

Are there an alternative proof of Cramer-Rao lower bound that does not use Cauchy-Swartz inequality?

Let me outline the classical proof and explain why I am interested in this question.

Choose some function $g(X,Y)$. Then, \begin{align} E[ (X-E[X\mid Y]) g(X,Y)] \le \left| E[ (X-E[X\mid Y]) g(X,Y)] \right| \le \sqrt{ E \left[ (X-E[X\mid Y])^2 \right] E[g(X,Y)^2] }. \end{align}

Therefore, \begin{align} E \left[ (X-E[X\mid Y])^2 \right] \ge \frac{\left| E[ (X-E[X\mid Y]) g(X,Y)] \right|}{E[g(X,Y)^2]}. \end{align}

The proof is completed by choosing $g(x,y)=\frac{d}{dx} \log (f_{XY}(x,y) ) $ and noting that then $ E[ (X-E[X\mid Y]) g(X,Y)]=-1$. This gives us the Cramer-Rao lower bound \begin{align} E \left[ (X-E[X\mid Y])^2 \right] \ge \frac{1}{E \left[ \left(\frac{d}{dx} \log (f_{XY}(X,Y) ) \right)^2 \right]}. \end{align}

The choice of $g(X,Y)=\frac{d}{dx} \log (f_{XY}(x,y) ) $ always seemed mysterious to me (but this is not the main reason for ask this question). That is why I am wondering whether there is a more "natural" proof where the quantity $\frac{d}{dx} \log (f_{XY}(x,y) )$ appearance is more obvious.

For example, it would be nice if we can derive an inequality by showing that

\begin{align} E \left[ (X-E[X\mid Y])^2 \right] = \frac{1}{E \left[ \left(\frac{d}{dx} \log (f_{XY}(X,Y) ) \right)^2 \right]}+c, \end{align} where $c$ is non-negative.

$\endgroup$
12
  • $\begingroup$ It is possible to write this proof a bit differently in a way you find more natural. But the key step uses convexity, so there is no way to avoid Cauchy-Schwarz or an equivalent inequality. $\endgroup$
    – Deane Yang
    Oct 23, 2017 at 14:27
  • $\begingroup$ @DeaneYang I don't see how convexity comes in. Could you show this proof? Another question. Why do you think there is no proof that avoids Cauchy-Schwartz but uses some other inequality? $\endgroup$
    – Boby
    Oct 23, 2017 at 14:36
  • $\begingroup$ The Cauchy-Schwarz inequality is equivalent to saying that the function $x \mapsto x^2$ is convex. Any proof of the Cramer-Rao inequality has to use a convexity inequality like that, and the Cauchy-Schwarz inequality is the simplest possible one, since $x\mapsto x^2$ is the simplest convex function. $\endgroup$
    – Deane Yang
    Oct 23, 2017 at 22:15
  • $\begingroup$ @DeaneYang So, how about Jensen's inequality, can we use it to show Cramer-Rao inequality? $\endgroup$
    – Boby
    Oct 24, 2017 at 13:42
  • $\begingroup$ Yes. It's a more general inequality. Cauchy-Schwartz is a special case of Jensen. $\endgroup$
    – Deane Yang
    Oct 24, 2017 at 13:52

2 Answers 2

4
$\begingroup$

In line with Deane's comment, this is an "answer" that also uses the Cauchy-Schwarz inequality but does so in a way that you might find more natural. I'll use different notation than yours (sorry; pushed for time and I'll probably mess it up if I attempt to translate quickly).

Take a family of probability density functions $f(-; \theta)$ parametrized by some real $\theta$, and an unbiased estimator $\hat{\theta}$ of $\theta$. The Cramér-Rao bound is a lower bound on $\text{Var}(\hat{\theta})$. Your implicit challenge is to derive it in a way that seems more natural than the proof you give.

Let's begin by writing down the definition of $\hat{\theta}$ being an unbiased estimator: $$ \theta = \int \hat{\theta}(x) f(x; \theta) \, dx $$ for all $\theta$. Because this holds for all $\theta$, we can differentiate both sides with respect to $\theta$, and I hope you'd agree that this is a fairly natural step. The result is $$ 1 = \int \hat{\theta}(x) \text{sc}(x; \theta) f(x; \theta) \, dx $$ where $\text{sc} = \frac{\partial f/\partial\theta}{f}$ (called the score). Now the right-hand side is a covariance: $$ 1 = \text{Cov}(\hat{\theta}, \text{sc}). $$ I hope you'd agree that it's natural to use the fact that correlation coefficients always have absolute value $\leq 1$. (That's basically the Cauchy-Schwarz inequality.) This gives $$ 1 \leq \sqrt{\text{Var}(\hat{\theta}) \text{Var}(\text{sc})}. $$ And rearranging, that's exactly the Cram\'er-Rao bound: $$ \text{Var}(\hat{\theta}) \geq 1/\text{Var}(\text{sc}). $$ (The variance of the score is called the Fisher information.)

$\endgroup$
2
  • $\begingroup$ This is the parametric case, right? I am looking at Bayesian. Does your proof still hold? $\endgroup$
    – Boby
    Oct 23, 2017 at 16:34
  • $\begingroup$ You should just try adapting the parameteric proof to the Bayesian case. $\endgroup$
    – Deane Yang
    Oct 23, 2017 at 22:19
1
$\begingroup$

Perhaps, the following re-arrangement of the argument will help remove the mystery of the choice of $g(x,y)=\frac{\partial}{\partial x} \ln f(x,y)=\frac{f'_x(x,y)}{f(x,y)}$, where $f:=f_{X,Y}$ and ${}'_x$ denotes the partial derivative in $x$.

Assume appropriate regularity conditions, whatever are needed for the manipulations below. Integrating by parts, we have
\begin{equation} \int x f'_x(x,y)\,dx=-\int f(x,y)\,dx, \end{equation} whence \begin{equation} EXg(X,Y)=\int\int x f'_x(x,y)\,dx\,dy=-1. \end{equation} Here, $\int:=\int_{-\infty}^\infty$.
Also, $\int f'_x(x,y)\,dx=0$ and hence \begin{equation} Eh(Y)g(X,Y)=\int dy\,h(y)\int f'_x(x,y)\,dx=0 \end{equation} for any function $h$ (satisfying appropriate regularity conditions). Therefore, $E(X-h(Y))g(X,Y)=-1$. Thus, by the Cauchy-- Schwarz inequality (hardly possible to do without it), \begin{equation} E(X-h(Y))^2\ge\frac1{Eg(X,Y)^2}. \end{equation} Of course, here one can take $h(Y)=E(X|Y)$ (which actually minimizes the left-hand side of the last inequality).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.