Fisher Information of variance of difference between random variable and gaussian

Question

I'm reading through the following paper: https://arxiv.org/pdf/0704.1751.pdf

I'm stuck in the middle of page 8, at the statement:

$$E[||S(X)-S^*(X)||^2] = J(X) - J(X^*)$$

Where $S(X)$ is the score of the random vector $X$, $S^*(X)$ is the score of the gaussian r.v. with the same covariance as $X$, and $J(X)$ is the Fisher information of $X$ defined to be $\text{tr}(\text{Cov}(S(X), S(X)))$.

What I have so far is:

\begin{align} E[||S(X)-S^*(X)||^2] &= E[||S(X)||^2 - 2(S(X)\cdot S^*(X)) + ||S^*(X)||^2]\\ &= E[||S(X)||^2] - 2E[S(X)\cdot S^*(X)] + E[||S^*(X)||^2]\\ &= J(X) - 2E[S(X)\cdot S^*(X)] + J(X^*) \end{align}

From this point it suffices to show that: $$E[S(X)\cdot S^*(X)] = J(X^*)$$

It can be shown that the score of the associated Gaussian is $-K^{-1}x$, where $K=\text{Cov}(X),$ so in other words, we look to show: $$E[S(X)\cdot (-K^{-1}X)] = E[(K^{-1}X)\cdot (K^{-1}X)]$$

How can this be shown? Or is there an alternative way to show this?

Thanks.

Answer 1

$\newcommand{\R}{\mathbb R}\newcommand{\tr}{\operatorname{tr}}$Let $p$ be the pdf of $X$, so that \begin{equation*} S(x)=\frac{\nabla p(x)}{p(x)} \tag{1}\label{1} \end{equation*} for all $x\in\R^n$ and \begin{equation*} \int_{\R^n} p(x-t)x\,dx=[E(X+t)=]t \tag{2}\label{2} \end{equation*} for all $t\in\R^n$.

If (say) $\int|\nabla p(x)|\,(|x|+1)dx<\infty$, then, differentiating \eqref{2} in $t$ at $t=0$ and using the Fubini theorem, we get \begin{equation*} -\int_{\R^n} \nabla p(x)x^\top\,dx=I, \tag{3}\label{3} \end{equation*} where $I$ is the identity matrix; that is, \begin{equation*} ES(X)X^\top=-I=EX S(X)^\top. \end{equation*}

So, \begin{equation*} ES(X)\cdot(K^{-1}X)=ES(X)^\top K^{-1}X=E\tr S(X)^\top K^{-1}X =E\tr K^{-1}XS(X)^\top=\tr K^{-1}EXS(X)^\top =-\tr K^{-1}. \end{equation*} Also, \begin{equation*} E(K^{-1}X)\cdot(K^{-1}X)=EX^\top K^{-2}X=E\tr K^{-2}XX^\top =\tr K^{-2}EXX^\top =\tr K^{-1}. \end{equation*}

Thus, indeed \begin{equation*} ES(X)\cdot(-K^{-1}X)=E(K^{-1}X)\cdot(K^{-1}X). \end{equation*}