7
$\begingroup$

It seems that there are two separate definitions for the Fisher information, and I'm wondering if there is some relationship between the two.

The first is the so-called Fisher information which appears in some versions of the log-Sobolev inequality. It has the form $I(f) = \int_X \frac{|\nabla f|^2}{f} dx$. Notice that the derivatives are in the observation space $X$. This quantity seems to be important in functional analysis.

The second is the so called Fisher information metric or Fisher-Rao metric. For a parameterized family of probability densities $f(x,\theta)$, we can express it in the following way: $$g^{FR}_f(\theta_i, \theta_j) = \int_X \frac{\nabla_{\theta_i} f \,\nabla_{\theta_j} f}{f} dx.$$

Note that the derivatives here are in the statistical manifold, not the observation space. It's straightforward to generalize this to a non-parametrized model with Frechet derivatives. This metric is important in statistics and probability because it is in some sense a canonical metric.

What I'm trying to understand is whether there some relationship between these two. I can see that if $X$ is the real line, and we perturb $f$ by translating it, then the $I(f)$ and the norm of the translation in $g_f^{FR}$ are the same. However, I don't see much else relating them.

Does anyone have any pointers?

Improve this question
$\endgroup$
8
  • $\begingroup$ Possibly relevant: stats.stackexchange.com/questions/154724/… $\endgroup$
    – SMD
    Commented Jun 5, 2018 at 18:39
  • $\begingroup$ Thanks for the reference. However, that question is entirely about the second type of Fisher information, where the derivatives are in the statistical manifold and doesn't use the first type, where the derivatives are in the state space. $\endgroup$
    – Gabe K
    Commented Jun 6, 2018 at 13:01
  • 1
    $\begingroup$ Have you heard about the fact that the Fisher information functional is the derivative of entropy along the heat semigroup? $\endgroup$
    – S.Surace
    Commented Jul 10, 2018 at 20:51
  • $\begingroup$ Thanks for bringing that up. That's actually exactly what I've been looking into recently. It appears this yields some interesting relationships, but let me work out the details before I say anything too publicly. $\endgroup$
    – Gabe K
    Commented Jul 11, 2018 at 2:24
  • 1
    $\begingroup$ After spending some more times with these concepts, I think the deepest that I can say so far is: the Fisher information functional is the derivative of the entropy along the heat semigroup, while the Fisher-Rao metric is the Hessian of the relative entropy $H(\nu||\mu)$ wrt. the first argument. I don't know whether this helps your quest. If you want, I can add another answer to expand on this. $\endgroup$
    – S.Surace
    Commented Oct 27, 2018 at 13:56

1 Answer 1

Reset to default
1
$\begingroup$

This is not a complete answer, but a comment that's gotten too long plus a hint.

My hunch is that it is unlikely that there is a general, simple connection. My reasoning is the following:

Let $X$ be a smooth manifold. The Fisher-Rao metric is invariant under the action of diffeomorphisms by pushforward, while the Fisher information functional $I$ is not; even under scalings on $X=\mathbb{R}^n$, the squared norm of the gradient will pick up a non-trivial scaling factor.

But maybe there is hope to find an interesting relation when the statistical manifold is closed under isometries, as you hint at with your translation example.

ADDENDUM: I found the following paper: B. Khesin, G. Misiolek, and K. Modin, “Geometric hydrodynamics via Madelung transform,” Proc. Natl. Acad. Sci. U.S.A., vol. 115, no. 24, pp. 6165–6170, Jun. 2018.

In Proposition 13 there, the Fisher metric and Fisher functional interact to produce some sort of wave equation that seems to be of interest. I'm afraid I cannot say much more. Maybe someone else will read this and be able to expand on this direction.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .