6
$\begingroup$

I'm reading 《Algebraic geometry and statistical learning theory》.My problem is why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?In the page 8
detail

detail

$\endgroup$

1 Answer 1

8
$\begingroup$

$\newcommand{\R}{\mathbb{R}} \newcommand{\p}{\partial}$

Let $(p_w)_{w\in W}$ be a family of pdfs, where $W$ is an open subset of $\R^k$. Take any $w_*\in W$. The Kullback-–Leibler "distance" from $w=(w_1,\dots,w_k)\in W$ to $w_*$ is \begin{equation} d(w):=D(w,w_*):=\int p_{w_*} \ln\frac{p_{w_*}}{p_{w}}. \end{equation} Letting $\p_j:=\frac{\p}{\p w_j}$ and assuming regularity conditions allowing the necessary differentiation with respect to the parameter $w$ under the integral sign, we have \begin{equation} \p_j d(w)=-\int p_{w_*} \frac{\p_j p_w}{p_{w}}, \end{equation} \begin{equation} \p_i\p_j d(w)|_{w=w_*}=\int p_{w_*} \Big(\frac{\p_i p_w\,\p_j p_w}{p_w^2} -\frac{\p_i\p_j p_w}{p_w}\Big)\Big|_{w=w_*} =\int\frac{\p_i p_w\,\p_j p_w}{p_{w_*}}\Big|_{w=w_*}=I_{ij}(w_*), \end{equation} the $ij$-entry of the Fisher information matrix $I(w_*)$, as desired. Here $i,j=1,\dots,k$, and we used the following: \begin{equation} \int p_{w_*} \frac{\p_i\p_j p_w}{p_w}\Big|_{w=w_*} =\int (\p_i\p_j p_w)|_{w=w_*}=\Big(\p_i\p_j\int p_w\Big)\Big|_{w=w_*}=\p_i\p_j 1=0. \end{equation}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.