Why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?

Question

I'm reading 《Algebraic geometry and statistical learning theory》.My problem is why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?In the page 8
detail

detail

Answer 1

$\newcommand{\R}{\mathbb{R}} \newcommand{\p}{\partial}$

Let $(p_w)_{w\in W}$ be a family of pdfs, where $W$ is an open subset of $\R^k$. Take any $w_*\in W$. The Kullback-–Leibler "distance" from $w=(w_1,\dots,w_k)\in W$ to $w_*$ is \begin{equation} d(w):=D(w,w_*):=\int p_{w_*} \ln\frac{p_{w_*}}{p_{w}}. \end{equation} Letting $\p_j:=\frac{\p}{\p w_j}$ and assuming regularity conditions allowing the necessary differentiation with respect to the parameter $w$ under the integral sign, we have \begin{equation} \p_j d(w)=-\int p_{w_*} \frac{\p_j p_w}{p_{w}}, \end{equation} \begin{equation} \p_i\p_j d(w)|_{w=w_*}=\int p_{w_*} \Big(\frac{\p_i p_w\,\p_j p_w}{p_w^2} -\frac{\p_i\p_j p_w}{p_w}\Big)\Big|_{w=w_*} =\int\frac{\p_i p_w\,\p_j p_w}{p_{w_*}}\Big|_{w=w_*}=I_{ij}(w_*), \end{equation} the $ij$-entry of the Fisher information matrix $I(w_*)$, as desired. Here $i,j=1,\dots,k$, and we used the following: \begin{equation} \int p_{w_*} \frac{\p_i\p_j p_w}{p_w}\Big|_{w=w_*} =\int (\p_i\p_j p_w)|_{w=w_*}=\Big(\p_i\p_j\int p_w\Big)\Big|_{w=w_*}=\p_i\p_j 1=0. \end{equation}