2
$\begingroup$

Consider a parametric family $p_\theta(x)$ of distributions, with parameter $\theta \in \Theta \subseteq \mathbb R^p$.

If the mapping $\theta \mapsto p_\theta(x)$ is continuously differentiable at $\theta_0 \in \Theta$ for $p_{\theta_0}$-almost all $x$, then the Fisher information at $\theta_0$ is the $p$-by-$p$ psd matrix given by $F(\theta_0) = \mathbb E_{x \sim p_{\theta_0}}[s_{\theta_0}(x)s_{\theta_0}(x)^T]$, where $x \mapsto s_{\theta_0}(x) := \partial_\theta \log(p_\theta(x))|_{\theta=\theta_0} \in \mathbb R^p$ is the score function at $\theta_0$.

Question

What is the geometric interpretation of the trace of $F(\theta_0)$ ?

Observation

If is well-known that $KL(p_{\theta_0 + \Delta \theta}\|p_{\theta_0}) = \frac{1}{2}\Delta \theta^T F(\theta_0)\Delta \theta + o(\|\Delta\theta\|^2)$.

$\endgroup$
3
  • $\begingroup$ The Fisher information defines a quadratic form on the tangent space at $\theta_0$. What do you mean by trace? $\endgroup$
    – S.Surace
    Oct 23, 2019 at 11:47
  • $\begingroup$ The trace of the $p$-by-$p$ psd matrix $F_{\theta_0}$ sure makes sense. No ? $\endgroup$
    – dohmatob
    Oct 23, 2019 at 15:01
  • $\begingroup$ Yes, but it is not clear to me what geometric entity, if any, this will correspond to. The geometrical status of the Fisher information matrix is a Riemannian metric, i.e. a tensor whose components have two covariant indices. Taking the trace of such a tensor requires raising one of the indices. If you do it with the Fisher metric, you get $p$. If you use some other metric, you get something else, but it depends on what the other metric is. $\endgroup$
    – S.Surace
    Oct 25, 2019 at 14:07

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.