48
$\begingroup$

I am now doing an article survey on the field of information geometry started by S.Amari and Barndorff-Nielson. I want to know some research situation in this field. I have read (4) and parts of (3).

As the comment in (1)'s answer said"I am seeking more of an expert's perspective on the field." I want some comments from experts too since I am totally new to this field.(I would say that it is lying between math and statistics.)

It seems like a young branch which starts at 1960s and reached its peak at around 1990s(5). According to (2), I still feel there is much potentiality in this field at the first sight. However, the critiques (6) certainly make sense but I doubt that these small flaws (like the lack of independence assumption) will affect its future development since this can probably be remedied be adding slightly more restrictive priori assumptions.

And I feel it rather insightful that some researchers has presented a new framework by using Fisher matrix as a convergence criterion(7).

Yet there is still few recent research papers in this field, which is contradictory to my first impression.

Question

My question is how the direction of researches is going on in the field of Information Geometry today? Is it a fancy field to be explored or it is just a dead end with some severe flaws I didn't catch? (If so, please point it out.)

Reference

(1)Related post on math.SE:Applications of IG

(2)Tutorial of IG:Information Theory and Statistics: A Tutorial

(3)Introduction written by S.Amari:Methods of Information Geometry

(4)Another readable introduction:Differential Geometry and Statistics

(5)The paper collections:Differential Geometry in Statistical Inference

(6)Two critical papers of IG:Critique of information geometry Failures of information geometry

(7)Works of S.Watanabe

(8)Wikipedia.org:wiki:IG

(9)A most recent paper by Amari talking about the interplay between information geometry, statistics and machine learning. Information geometry in optimization, machine learning and statistical inference

$\endgroup$
8
  • 3
    $\begingroup$ I don't want to write an answer since I'm not an expert, but I can give the opinion of someone with a geometry background who tried to get a sense of what IG is all about (by looking at Murray and Rice's book, and a few other articles). My opinion is that the theory seems quite rich and elegant, but I couldn't tell you what the main examples are or what the theory would hope to say about them. In principle I think the examples are supposed to come from parametrized families of random variables, and it is natural to ask if such a family can be taken seriously as a geometric object... $\endgroup$ Aug 30, 2015 at 22:22
  • 2
    $\begingroup$ ...but it's hard to think of a natural example where the parameter space is something other than a finite set, $\mathbb{R}^n$, or a lattice in $\mathbb{R}^n$. And if you take a problem associated to a family with a more familiar parameter space (e.g. the central limit theorem) it is not clear what the corresponding problem would be in IG. Answers to these questions may appear somewhere in the literature, but in my experience the literature is more preoccupied with building the theory than motivating it with examples. $\endgroup$ Aug 30, 2015 at 22:30
  • $\begingroup$ @PaulSiegel I think what you talked about is just 1980s' situation in Murray's text. If my memory is correct, there're a few papers attempting to apply ergodic theory onto more general situations besides parametrizable families. But thank you for your time and attention to this problem. Actually I just felt strange that such an elegant theory attracted so little attention...Thanks! $\endgroup$
    – Henry.L
    Aug 30, 2015 at 23:04
  • 1
    $\begingroup$ I would think it's natural to have a parameter space given by a Lie group, as in, e.g. mathoverflow.net/questions/14729 $\endgroup$ Aug 31, 2015 at 0:09
  • 1
    $\begingroup$ @SteveHuntsman You mean a multi-variate case with matrix parameters? But how the Lie group structure helps in this case? I am confused. $\endgroup$
    – Henry.L
    Aug 31, 2015 at 0:43

2 Answers 2

20
$\begingroup$

I'm not sure I would say I'm an expert in information geometry. However, I worked for several years on the subject as a postdoc. As a disclaimer, this is entirely my own opinion and others may disagree.

Since you asked this question, the research situation in the field has improved. Firstly, two separate books ([1$ $], [2$ $]) have been published, both of which are good references for the material. In particular, the second gives a rigorous mathematical treatment for the basic theory. Secondly, a new journal, Information Geometry, has been released. Thus far several issues have been published and they contain some interesting papers.

However, information geometry is definitely a relatively niche mathematical field. As to the reason for this, in my opinion IG is really an interdisciplinary field and not simply a branch of mathematics. Many of the people working in the field are not mathematicians by background. As a result, information geometry embodies a wide range of research. Some papers are mathematical, but many others are really statistics, computer science, or some hybrid thereof. Many of the publishing conventions are differ from math, as well. For instance, it's common to publish short papers without proofs in conference proceedings and, generally speaking, the main theorems are not stated in the introduction.

While there is a lot of good work being done in the field, there is also too much research that is not really serious. Most of this is not done in bad faith, but due to a lack of experience and background in geometry. Furthermore, a lot of the work is published in a for-profit journal whose peer review process is minimal. Without giving examples, some papers boil down to slightly modifying known results and treating them as novel. Other papers try to use really big ideas without understanding the underlying theory or really proving anything. Furthermore, what is considered acceptable overlap between publications is far greater than in pure math. Needless to say, these issues create serious problems for the field, and makes it much less likely to be taken seriously.

Even with the good papers, they often seem to lack a good punchline. As was mentioned in the comments, the math in IG has built up a very general foundational theory, often without providing mathematical or statistical motivation for this theory. My impression is that quite a few of the researchers in the field were heavily influenced by the "structural point of view" pioneered by Nomizu and Kobayashi. I suppose the motivation for these structures might be self-evident to a statistician, but as a geometer oftentimes it's completely lost on me. In my experience, I only really started to understand what was going on when I worked through some important examples of statistical manifolds, instead of trying to learn the theory from the ground up.

Related to the point above, it's difficult to find explicit conjectures in the field. There isn't something similar to Yau's list of open problems in geometry to guide progress in the field. As such, when I was learning the field it was hard to tell what was considered an important problem and to understand the motivations for the research.

As a result of all of these factors, information geometry has remained a specialized sub-field. I think this will remain the case unless it is used to solve a big problem or it evolves to be more in line with standard mathematical conventions. All that being said, I've learned a lot from information geometry, and there is definitely a fair amount of low-hanging fruit to be picked. Furthermore, the field seems to be making progress in recent years, so hopefully my critiques will soon be obsolete.

To end on a positive note, let me give an example of a paper that I think does things well [3$ $]. This work studies necessary conditions for a Riemannian manifold to locally be written as the Hessian of a convex potential. I really like this paper and have found it helpful for my intuition.

P.S. If anyone is interested, I was able to find a list of open problems from 1998, some of which have since been solved.

References

[1$ $] Amari, S. I. (2016). Information geometry and its applications (Vol. 194). Tokyo: Springer.

[2$ $] Ay, N., Jost, J., Vân Lê, H., & Schwachhöfer, L. (2017). Information geometry (Vol. 8). Berlin: Springer.

[3$ $] Amari, S. I., & Armstrong, J. (2014). Curvature of Hessian manifolds. Differential Geometry and its Applications, 33, 1-12.

$\endgroup$
2
  • 1
    $\begingroup$ Could you say more about why Hessian manifolds are useful to information geometry? $\endgroup$
    – Deane Yang
    Aug 18, 2020 at 18:52
  • $\begingroup$ Of course! The basic reason is that any exponential family is a Hessian manifold when you use its natural parameters as the coordinates and the Fisher-Rao metric as the Riemannian metric. I'd be happy to fill in more details, but it might be better to do that in a separate question as opposed to a long series of comments. $\endgroup$
    – Gabe K
    Aug 18, 2020 at 19:00
-1
$\begingroup$

The two 'critique' papers you mention have atrocious flaws:

The author comments on how the distances defined by the information metric are a metric space, and how relative entropy is not, and say that as a consequence information geometry is wrong.

There natural flaw with his reasoning is the fact that information geometry is not supposed to be used as relative entropy. Then he presents his 'examples' that are nothing more than unnecessarily complicated versions of problems in which usually relative entropy is used and the results from minimizing distances do not match. So what?

He say information geometry is wrong, but apparently he has no knowledge of differential geometry and think is not applicable. TL,DR: He says Rao is wrong but he doesn't understand Riemann/Gauss yet.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.