# An alternative proof of Bayesian Cramer-Rao

My question is:

Are there an alternative proof of Cramer-Rao lower bound that does not use Cauchy-Swartz inequality?

Let me outline the classical proof and explain why I am interested in this question.

Choose some function $g(X,Y)$. Then, \begin{align} E[ (X-E[X|Y]) g(X,Y)] \le \left| E[ (X-E[X|Y]) g(X,Y)] \right| \le \sqrt{ E \left[ (X-E[X|Y])^2 \right] E[g(X,Y)^2] }. \end{align}

Therefore, \begin{align} E \left[ (X-E[X|Y])^2 \right] \ge \frac{\left| E[ (X-E[X|Y]) g(X,Y)] \right|}{E[g(X,Y)^2]}. \end{align}

The proof is completed by choosing $g(x,y)=\frac{d}{dx} \log (f_{XY}(x,y) )$ and noting that then $E[ (X-E[X|Y]) g(X,Y)]=-1$. This gives us the Cramer-Rao lower bound \begin{align} E \left[ (X-E[X|Y])^2 \right] \ge \frac{1}{E \left[ \left(\frac{d}{dx} \log (f_{XY}(X,Y) ) \right)^2 \right]}. \end{align}

The choice of $g(X,Y)=\frac{d}{dx} \log (f_{XY}(x,y) )$ always seemed mysterious to me (but this is not the main reason for ask this question). That is why I am wondering whether there is a more "natural" proof where the quantity $\frac{d}{dx} \log (f_{XY}(x,y) )$ appearance is more obvious.

For example, it would be nice if we can derive an inequality by showing that

\begin{align} E \left[ (X-E[X|Y])^2 \right] = \frac{1}{E \left[ \left(\frac{d}{dx} \log (f_{XY}(X,Y) ) \right)^2 \right]}+c, \end{align} where $c$ is non-negative.

• It is possible to write this proof a bit differently in a way you find more natural. But the key step uses convexity, so there is no way to avoid Cauchy-Schwarz or an equivalent inequality. – Deane Yang Oct 23 '17 at 14:27
• @DeaneYang I don't see how convexity comes in. Could you show this proof? Another question. Why do you think there is no proof that avoids Cauchy-Schwartz but uses some other inequality? – Boby Oct 23 '17 at 14:36
• The Cauchy-Schwarz inequality is equivalent to saying that the function $x \mapsto x^2$ is convex. Any proof of the Cramer-Rao inequality has to use a convexity inequality like that, and the Cauchy-Schwarz inequality is the simplest possible one, since $x\mapsto x^2$ is the simplest convex function. – Deane Yang Oct 23 '17 at 22:15
• @DeaneYang So, how about Jensen's inequality, can we use it to show Cramer-Rao inequality? – Boby Oct 24 '17 at 13:42
• Yes. It's a more general inequality. Cauchy-Schwartz is a special case of Jensen. – Deane Yang Oct 24 '17 at 13:52

In line with Deane's comment, this is an "answer" that also uses the Cauchy-Schwarz inequality but does so in a way that you might find more natural. I'll use different notation than yours (sorry; pushed for time and I'll probably mess it up if I attempt to translate quickly).

Take a family of probability density functions $f(-; \theta)$ parametrized by some real $\theta$, and an unbiased estimator $\hat{\theta}$ of $\theta$. The Cramér-Rao bound is a lower bound on $\text{Var}(\hat{\theta})$. Your implicit challenge is to derive it in a way that seems more natural than the proof you give.

Let's begin by writing down the definition of $\hat{\theta}$ being an unbiased estimator: $$\theta = \int \hat{\theta}(x) f(x; \theta) \, dx$$ for all $\theta$. Because this holds for all $\theta$, we can differentiate both sides with respect to $\theta$, and I hope you'd agree that this is a fairly natural step. The result is $$1 = \int \hat{\theta}(x) \text{sc}(x; \theta) f(x; \theta) \, dx$$ where $\text{sc} = \frac{\partial f/\partial\theta}{f}$ (called the score). Now the right-hand side is a covariance: $$1 = \text{Cov}(\hat{\theta}, \text{sc}).$$ I hope you'd agree that it's natural to use the fact that correlation coefficients always have absolute value $\leq 1$. (That's basically the Cauchy-Schwarz inequality.) This gives $$1 \leq \sqrt{\text{Var}(\hat{\theta}) \text{Var}(\text{sc})}.$$ And rearranging, that's exactly the Cram\'er-Rao bound: $$\text{Var}(\hat{\theta}) \geq 1/\text{Var}(\text{sc}).$$ (The variance of the score is called the Fisher information.)

• This is the parametric case, right? I am looking at Bayesian. Does your proof still hold? – Boby Oct 23 '17 at 16:34
• You should just try adapting the parameteric proof to the Bayesian case. – Deane Yang Oct 23 '17 at 22:19

Perhaps, the following re-arrangement of the argument will help remove the mystery of the choice of $g(x,y)=\frac{\partial}{\partial x} \ln f(x,y)=\frac{f'_x(x,y)}{f(x,y)}$, where $f:=f_{X,Y}$ and ${}'_x$ denotes the partial derivative in $x$.

Assume appropriate regularity conditions, whatever are needed for the manipulations below. Integrating by parts, we have
\begin{equation} \int x f'_x(x,y)\,dx=-\int f(x,y)\,dx, \end{equation} whence \begin{equation} EXg(X,Y)=\int\int x f'_x(x,y)\,dx\,dy=-1. \end{equation} Here, $\int:=\int_{-\infty}^\infty$.
Also, $\int f'_x(x,y)\,dx=0$ and hence \begin{equation} Eh(Y)g(X,Y)=\int dy\,h(y)\int f'_x(x,y)\,dx=0 \end{equation} for any function $h$ (satisfying appropriate regularity conditions). Therefore, $E(X-h(Y))g(X,Y)=-1$. Thus, by the Cauchy-- Schwarz inequality (hardly possible to do without it), \begin{equation} E(X-h(Y))^2\ge\frac1{Eg(X,Y)^2}. \end{equation} Of course, here one can take $h(Y)=E(X|Y)$ (which actually minimizes the left-hand side of the last inequality).