An alternative proof of Bayesian Cramer-Rao

Question

My question is:

Are there an alternative proof of Cramer-Rao lower bound that does not use Cauchy-Swartz inequality?

Let me outline the classical proof and explain why I am interested in this question.

Choose some function $g(X,Y)$. Then, \begin{align} E[ (X-E[X\mid Y]) g(X,Y)] \le \left| E[ (X-E[X\mid Y]) g(X,Y)] \right| \le \sqrt{ E \left[ (X-E[X\mid Y])^2 \right] E[g(X,Y)^2] }. \end{align}

Therefore, \begin{align} E \left[ (X-E[X\mid Y])^2 \right] \ge \frac{\left| E[ (X-E[X\mid Y]) g(X,Y)] \right|}{E[g(X,Y)^2]}. \end{align}

The proof is completed by choosing $g(x,y)=\frac{d}{dx} \log (f_{XY}(x,y) ) $ and noting that then $ E[ (X-E[X\mid Y]) g(X,Y)]=-1$. This gives us the Cramer-Rao lower bound \begin{align} E \left[ (X-E[X\mid Y])^2 \right] \ge \frac{1}{E \left[ \left(\frac{d}{dx} \log (f_{XY}(X,Y) ) \right)^2 \right]}. \end{align}

The choice of $g(X,Y)=\frac{d}{dx} \log (f_{XY}(x,y) ) $ always seemed mysterious to me (but this is not the main reason for ask this question). That is why I am wondering whether there is a more "natural" proof where the quantity $\frac{d}{dx} \log (f_{XY}(x,y) )$ appearance is more obvious.

For example, it would be nice if we can derive an inequality by showing that

\begin{align} E \left[ (X-E[X\mid Y])^2 \right] = \frac{1}{E \left[ \left(\frac{d}{dx} \log (f_{XY}(X,Y) ) \right)^2 \right]}+c, \end{align} where $c$ is non-negative.

Answer 1

In line with Deane's comment, this is an "answer" that also uses the Cauchy-Schwarz inequality but does so in a way that you might find more natural. I'll use different notation than yours (sorry; pushed for time and I'll probably mess it up if I attempt to translate quickly).

Take a family of probability density functions $f(-; \theta)$ parametrized by some real $\theta$, and an unbiased estimator $\hat{\theta}$ of $\theta$. The Cramér-Rao bound is a lower bound on $\text{Var}(\hat{\theta})$. Your implicit challenge is to derive it in a way that seems more natural than the proof you give.

Let's begin by writing down the definition of $\hat{\theta}$ being an unbiased estimator: $$ \theta = \int \hat{\theta}(x) f(x; \theta) \, dx $$ for all $\theta$. Because this holds for all $\theta$, we can differentiate both sides with respect to $\theta$, and I hope you'd agree that this is a fairly natural step. The result is $$ 1 = \int \hat{\theta}(x) \text{sc}(x; \theta) f(x; \theta) \, dx $$ where $\text{sc} = \frac{\partial f/\partial\theta}{f}$ (called the score). Now the right-hand side is a covariance: $$ 1 = \text{Cov}(\hat{\theta}, \text{sc}). $$ I hope you'd agree that it's natural to use the fact that correlation coefficients always have absolute value $\leq 1$. (That's basically the Cauchy-Schwarz inequality.) This gives $$ 1 \leq \sqrt{\text{Var}(\hat{\theta}) \text{Var}(\text{sc})}. $$ And rearranging, that's exactly the Cram\'er-Rao bound: $$ \text{Var}(\hat{\theta}) \geq 1/\text{Var}(\text{sc}). $$ (The variance of the score is called the Fisher information.)

Answer 2

Perhaps, the following re-arrangement of the argument will help remove the mystery of the choice of $g(x,y)=\frac{\partial}{\partial x} \ln f(x,y)=\frac{f'_x(x,y)}{f(x,y)}$, where $f:=f_{X,Y}$ and ${}'_x$ denotes the partial derivative in $x$.

Assume appropriate regularity conditions, whatever are needed for the manipulations below. Integrating by parts, we have
\begin{equation} \int x f'_x(x,y)\,dx=-\int f(x,y)\,dx, \end{equation} whence \begin{equation} EXg(X,Y)=\int\int x f'_x(x,y)\,dx\,dy=-1. \end{equation} Here, $\int:=\int_{-\infty}^\infty$.
Also, $\int f'_x(x,y)\,dx=0$ and hence \begin{equation} Eh(Y)g(X,Y)=\int dy\,h(y)\int f'_x(x,y)\,dx=0 \end{equation} for any function $h$ (satisfying appropriate regularity conditions). Therefore, $E(X-h(Y))g(X,Y)=-1$. Thus, by the Cauchy-- Schwarz inequality (hardly possible to do without it), \begin{equation} E(X-h(Y))^2\ge\frac1{Eg(X,Y)^2}. \end{equation} Of course, here one can take $h(Y)=E(X|Y)$ (which actually minimizes the left-hand side of the last inequality).