My question is:

Are there an alternative proof of Cramer-Rao lower bound that does not use Cauchy-Swartz inequality?

Let me outline the classical proof and explain why I am interested in this question.

Choose some function $g(X,Y)$. Then, \begin{align} E[ (X-E[X|Y]) g(X,Y)] \le \left| E[ (X-E[X|Y]) g(X,Y)] \right| \le \sqrt{ E \left[ (X-E[X|Y])^2 \right] E[g(X,Y)^2] }. \end{align}

Therefore, \begin{align} E \left[ (X-E[X|Y])^2 \right] \ge \frac{\left| E[ (X-E[X|Y]) g(X,Y)] \right|}{E[g(X,Y)^2]}. \end{align}

The proof is completed by choosing $g(x,y)=\frac{d}{dx} \log (f_{XY}(x,y) ) $ and noting that then $ E[ (X-E[X|Y]) g(X,Y)]=-1$. This gives us the Cramer-Rao lower bound \begin{align} E \left[ (X-E[X|Y])^2 \right] \ge \frac{1}{E \left[ \left(\frac{d}{dx} \log (f_{XY}(X,Y) ) \right)^2 \right]}. \end{align}

The choice of $g(X,Y)=\frac{d}{dx} \log (f_{XY}(x,y) ) $ always seemed mysterious to me (but this is not the main reason for ask this question). That is why I am wondering whether there is a more "natural" proof where the quantity $\frac{d}{dx} \log (f_{XY}(x,y) )$ appearance is more obvious.

For example, it would be nice if we can derive an inequality by showing that

\begin{align} E \left[ (X-E[X|Y])^2 \right] = \frac{1}{E \left[ \left(\frac{d}{dx} \log (f_{XY}(X,Y) ) \right)^2 \right]}+c, \end{align} where $c$ is non-negative.

  • $\begingroup$ It is possible to write this proof a bit differently in a way you find more natural. But the key step uses convexity, so there is no way to avoid Cauchy-Schwarz or an equivalent inequality. $\endgroup$ – Deane Yang Oct 23 '17 at 14:27
  • $\begingroup$ @DeaneYang I don't see how convexity comes in. Could you show this proof? Another question. Why do you think there is no proof that avoids Cauchy-Schwartz but uses some other inequality? $\endgroup$ – Boby Oct 23 '17 at 14:36
  • $\begingroup$ The Cauchy-Schwarz inequality is equivalent to saying that the function $x \mapsto x^2$ is convex. Any proof of the Cramer-Rao inequality has to use a convexity inequality like that, and the Cauchy-Schwarz inequality is the simplest possible one, since $x\mapsto x^2$ is the simplest convex function. $\endgroup$ – Deane Yang Oct 23 '17 at 22:15
  • $\begingroup$ @DeaneYang So, how about Jensen's inequality, can we use it to show Cramer-Rao inequality? $\endgroup$ – Boby Oct 24 '17 at 13:42
  • $\begingroup$ Yes. It's a more general inequality. Cauchy-Schwartz is a special case of Jensen. $\endgroup$ – Deane Yang Oct 24 '17 at 13:52

In line with Deane's comment, this is an "answer" that also uses the Cauchy-Schwarz inequality but does so in a way that you might find more natural. I'll use different notation than yours (sorry; pushed for time and I'll probably mess it up if I attempt to translate quickly).

Take a family of probability density functions $f(-; \theta)$ parametrized by some real $\theta$, and an unbiased estimator $\hat{\theta}$ of $\theta$. The Cramér-Rao bound is a lower bound on $\text{Var}(\hat{\theta})$. Your implicit challenge is to derive it in a way that seems more natural than the proof you give.

Let's begin by writing down the definition of $\hat{\theta}$ being an unbiased estimator: $$ \theta = \int \hat{\theta}(x) f(x; \theta) \, dx $$ for all $\theta$. Because this holds for all $\theta$, we can differentiate both sides with respect to $\theta$, and I hope you'd agree that this is a fairly natural step. The result is $$ 1 = \int \hat{\theta}(x) \text{sc}(x; \theta) f(x; \theta) \, dx $$ where $\text{sc} = \frac{\partial f/\partial\theta}{f}$ (called the score). Now the right-hand side is a covariance: $$ 1 = \text{Cov}(\hat{\theta}, \text{sc}). $$ I hope you'd agree that it's natural to use the fact that correlation coefficients always have absolute value $\leq 1$. (That's basically the Cauchy-Schwarz inequality.) This gives $$ 1 \leq \sqrt{\text{Var}(\hat{\theta}) \text{Var}(\text{sc})}. $$ And rearranging, that's exactly the Cram\'er-Rao bound: $$ \text{Var}(\hat{\theta}) \geq 1/\text{Var}(\text{sc}). $$ (The variance of the score is called the Fisher information.)

  • $\begingroup$ This is the parametric case, right? I am looking at Bayesian. Does your proof still hold? $\endgroup$ – Boby Oct 23 '17 at 16:34
  • $\begingroup$ You should just try adapting the parameteric proof to the Bayesian case. $\endgroup$ – Deane Yang Oct 23 '17 at 22:19

Perhaps, the following re-arrangement of the argument will help remove the mystery of the choice of $g(x,y)=\frac{\partial}{\partial x} \ln f(x,y)=\frac{f'_x(x,y)}{f(x,y)}$, where $f:=f_{X,Y}$ and ${}'_x$ denotes the partial derivative in $x$.

Assume appropriate regularity conditions, whatever are needed for the manipulations below. Integrating by parts, we have
\begin{equation} \int x f'_x(x,y)\,dx=-\int f(x,y)\,dx, \end{equation} whence \begin{equation} EXg(X,Y)=\int\int x f'_x(x,y)\,dx\,dy=-1. \end{equation} Here, $\int:=\int_{-\infty}^\infty$.
Also, $\int f'_x(x,y)\,dx=0$ and hence \begin{equation} Eh(Y)g(X,Y)=\int dy\,h(y)\int f'_x(x,y)\,dx=0 \end{equation} for any function $h$ (satisfying appropriate regularity conditions). Therefore, $E(X-h(Y))g(X,Y)=-1$. Thus, by the Cauchy-- Schwarz inequality (hardly possible to do without it), \begin{equation} E(X-h(Y))^2\ge\frac1{Eg(X,Y)^2}. \end{equation} Of course, here one can take $h(Y)=E(X|Y)$ (which actually minimizes the left-hand side of the last inequality).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.