3
$\begingroup$

I'm reading 《Algebraic geometry and statistical learning theory》.My problem is why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?In the page 8
detail

detail

$\endgroup$
3
$\begingroup$

$\newcommand{\R}{\mathbb{R}} \newcommand{\p}{\partial}$

Let $(p_w)_{w\in W}$ be a family of pdfs, where $W$ is an open subset of $\R^k$. Take any $w_*\in W$. The Kullback-–Leibler "distance" from $w=(w_1,\dots,w_k)\in W$ to $w_*$ is \begin{equation} d(w):=D(w,w_*):=\int p_{w_*} \ln\frac{p_{w_*}}{p_{w}}. \end{equation} Letting $\p_j:=\frac{\p}{\p w_j}$ and assuming regularity conditions allowing the necessary differentiation with respect to the parameter $w$ under the integral sign, we have \begin{equation} \p_j d(w)=-\int p_{w_*} \frac{\p_j p_w}{p_{w}}, \end{equation} \begin{equation} \p_i\p_j d(w)|_{w=w_*}=\int p_{w_*} \Big(\frac{\p_i p_w\,\p_j p_w}{p_w^2} -\frac{\p_i\p_j p_w}{p_w}\Big)\Big|_{w=w_*} =\int\frac{\p_i p_w\,\p_j p_w}{p_{w_*}}\Big|_{w=w_*}=I_{ij}(w_*), \end{equation} the $ij$-entry of the Fisher information matrix $I(w_*)$, as desired. Here $i,j=1,\dots,k$, and we used the following: \begin{equation} \int p_{w_*} \frac{\p_i\p_j p_w}{p_w}\Big|_{w=w_*} =\int (\p_i\p_j p_w)|_{w=w_*}=\Big(\p_i\p_j\int p_w\Big)\Big|_{w=w_*}=\p_i\p_j 1=0. \end{equation}

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.