Questions tagged [information-geometry]

Information geometry is a branch of mathematics that applies the techniques of differential geometry to the field of probability theory. This is done by taking probability distributions for a statistical model as the points of a Riemannian manifold, forming a statistical manifold. The Fisher information metric provides the Riemannian metric.

Filter by
Sorted by
Tagged with
48 votes
2 answers
13k views

Research situation in the field of Information Geometry

I am now doing an article survey on the field of information geometry started by S.Amari and Barndorff-Nielson. I want to know some research situation in this field. I have read (4) and parts of (3). ...
Henry.L's user avatar
  • 7,951
23 votes
1 answer
4k views

Relation between information geometry and geometric deep learning

Disclaimer: This is a cross-post from a very similar question on math.SE. I allowed myself to post it here after reading this meta post about cross-posting between mathoverflow and math.SE, I did ...
Blupon's user avatar
  • 333
11 votes
1 answer
3k views

Which books should I read in order to be prepared to study information geometry?

At the moment, I am preparing my master's thesis (in statistics) and I intend to keep studying in order to pursue a doctoral degree. To be precise, I am mainly interested in studying Information ...
user avatar
9 votes
5 answers
2k views

Reviews of Probability in High Dimension not by Van Handel

I'm completely in love with Ramon van Handel's lecture notes Probability in High Dimension and I would like to find more learning resources. Lecture notes or reviews would be ideal as anything in this ...
user41147's user avatar
  • 263
8 votes
1 answer
291 views

Upper-bound on the Fisher-Rao distance between multivariate Gaussian measures by the KL-divergence

Let $\mu$ and $\nu$ be two multivariate Gaussian measures on $\mathbb{R}^d$ with non-singular covariance matrices. Can the Fisher-Rao distance $d(\mu,\nu)$ computed on the information manifold of non-...
Justin_other_PhD's user avatar
8 votes
2 answers
938 views

Approximation of Wasserstein distance between $p_\theta$ and $p_{\theta + d\theta}$

Given a parametric family of distributions $\{p_\theta\mid\theta \in \Theta\}$, one can show that under some regularity conditions, the following approximation is valid $$\operatorname{KL}(p_\theta\...
dohmatob's user avatar
  • 6,716
8 votes
3 answers
1k views

Introduction to information geometry and/or geometric control theory

Some background: I'vebeen searching for a research project to work through my grad studies and I found information geometry like a strong candidate but the amount of work out there is overwhelming. I ...
Santiago Gil's user avatar
7 votes
1 answer
315 views

What is the correct notion of morphism between statistical manifolds?

Given two statistical manifolds, is there a notion of "isomorphic"? What are morphisms?
user168590's user avatar
7 votes
1 answer
353 views

Singular Fisher information matrix and existence of unbiased estimators

I'm doing some research into the Cramer-Rao bound for time of arrival localization and have come across a rather strange result: the FIM is singular, but there exists an unbiased estimator. My ...
JNL's user avatar
  • 75
7 votes
2 answers
483 views

Projections in infinite dimensional statistical manifolds

I'm struggling to understand the geometry of projection for infinite dimensional statistical manifolds. In finite dimensions, a strictly convex smooth function $F$ defines a Bregman divergence. From ...
John Rached's user avatar
6 votes
1 answer
784 views

reverse KL-divergence: Bregman or not?

I am having a little trouble getting my head around the two "directions" of the Kullback-Leibler divergence: Definition (Kullback-Leibler divergence) For discrete probability distributions $...
jw7642's user avatar
  • 91
6 votes
1 answer
343 views

Is the gradient of a strictly convex, continuously differentiable function a homeomorphism?

Let $X\subseteq\mathbb{R}^n$ be a convex set. Let $f:X\to\mathbb{R}$ be a strictly convex function that is differentiable on the (non-empty) relative interior of $X$. $\nabla f$ is a bijection, but is ...
rick's user avatar
  • 121
6 votes
1 answer
157 views

Smallest $\mathrm{D}(Q\|P)$ given fixed marginals $\mathrm{D}(Q_X\|P_X)$ and $\mathrm{D}(Q_Y\|P_Y)$

Let $P$ be a distribution on a set $U\times V$ with marginal distributions $P_X$ and $P_Y$. Suppose we have two values $d_x, d_y\in\mathbb R$, and we want to find the distribution $Q$ absolutely ...
Thomas Dybdahl Ahle's user avatar
6 votes
0 answers
261 views

Covariance operator analogue for manifolds and respective measure manifolds

Assume $E$ is a connected riemannian manifold with geodesic metric space structure given by $d$ and $P$ is a probability measure over $E$ with Borel sigma-algebra given by this metric structure. Also ...
Nik Bren's user avatar
  • 499
5 votes
2 answers
718 views

Comparison of Information and Wasserstein Topologies

There are many possible metrics one can place on the space of Gaussian probability measures on $\mathbb{R}^n$, with strictly positive definite co-variance matrices. Let's denote this space by $X$. I'...
Catologist_who_flies_on_Monday's user avatar
5 votes
2 answers
1k views

How to study to learn differential geometry for applying it to statistics

Basically I want to learn information geometry or specifically the application of differential geometry in statistics to do a project. I am from a statistical background and have a knowledge about ...
Avishek Dutta's user avatar
5 votes
2 answers
1k views

Relationship between $\alpha$-divergences?

I am working with $\alpha$-divergences and was wondering how understand the relationship between the definitions of Renyi and Amari? Renyi: $D_{\alpha}[p||q] = \frac{1}{\alpha - 1} \log \int p^{\...
nico's user avatar
  • 91
5 votes
1 answer
404 views

Pythagorean theorems for other distances

Question The usual projection in $\mathbb{R}^n$ on a subspace can be defined as the point that minimizes the squared distance to the subspace. I'll call the Pythagorean theorem the easy fact that, ...
Aaron Bergman's user avatar
5 votes
0 answers
187 views

Divergence for Bhattacharya Information matrix

The Fisher information matrix (in the scalar parameter case) can be obtained from the Kullback-Leibler divergence by $$g(\theta) = -\frac{\partial}{\partial \theta}\frac{\partial}{\partial \theta'}D(...
Ashok's user avatar
  • 779
5 votes
0 answers
339 views

Distance measures that preserve Pythagoras' theorem but break the triangle inequality

In information geometry, we can think of the Kullback-Leibler divergence as being "something like a squared distance." The sense of this is that if we have three probability measures, $P$, $Q$ and $R$...
N. Virgo's user avatar
  • 1,316
4 votes
1 answer
115 views

Gromov hyperbolicity for (non-geodesic) metrics on the upper-half plane invariant with respect to SL(2, R) action

$\DeclareMathOperator\SL{SL}$Let $d$ be a metric on the upper-half plane $\mathbb H = \{(x,y) : y > 0\}$ which is invariant with respect to the action of $\SL(2, \mathbb R)$ to $\mathbb H$ which is ...
Kazuki OKAMURA's user avatar
4 votes
2 answers
750 views

About optimization with Renyi divergence

Can someone link me to some pedagogic example of computing the Renyi divergence between two discrete/continuous distributions? Like examples where someone has been able to obtain a neat closed form or ...
gradstudent's user avatar
  • 2,136
4 votes
1 answer
303 views

Information monotonicity of divergence => function of $f$-divergence

It is well-known that $f$-divergences defined on $\mathcal P(\mathcal X)$ where $\mathcal X$ is a measure space with $\sigma$-algebra $\mathcal B$ satisfy the property of information monotonicity: ...
Lance's user avatar
  • 203
4 votes
0 answers
117 views

Exponential families closed under affine transformations

Let $(\Omega,\Sigma,\mu)$ be a probability space and let $\mathcal{M}$ be an exponential family of probability distributions for $\mu$ of the following form: There are $\varphi_1,\dots,\varphi_n:\...
ABIM's user avatar
  • 5,019
3 votes
2 answers
328 views

Direct calculation of the Fisher distance via Riemannian geodesics

I'm looking for a reference for a direct calculation of the Fisher distance (to avoid overloading the term "metric") $d_F(x,y) := 2 \cos^{-1} \sum_i \sqrt{x_i y_i}$ as the geodesic distance ...
Steve Huntsman's user avatar
3 votes
1 answer
119 views

Conditions for: (local) lipschitz stability of I-projection

The following post builds on this post; I'll begin by quoting the setting. Background from Previous Question: $\newcommand\SS{P}\newcommand\TT{Q}$Call a Gaussian probability measure $\SS$ on $\mathbb{...
Math_Newbie's user avatar
3 votes
1 answer
138 views

Interpolation inequality related to the 5/3-Laplace operator

I'm having trouble with an estimate that would be helpful in information geometry. The background is the following. Suppose we have a smooth positive function $g:X \to \mathbb{R}^+$ where $X$ is a ...
Gabe K's user avatar
  • 5,364
3 votes
0 answers
455 views

From $f$-divergence to its dual: the transformation of convex functions on $\mathbb R_+$ by $f^*(t) = 1 f(\frac 1 t)$

I would like to understand the relationship between minimising the KL divergence $P \mapsto D_{KL}[P,Q]$ and the reverse KL divergence $P\mapsto D_{KL}^*[P,Q]=D_{KL}[Q,P]$ for probability measures $P$ ...
Lance's user avatar
  • 203
3 votes
0 answers
232 views

Parametric distances on product spaces of measures

Disclaimer: Please excuse my loose language. I'm neither an expert in geometry nor probability. Please ask for clarification if something appears unclear or awkward to you. Let $X$ be a topological ...
dohmatob's user avatar
  • 6,716
3 votes
0 answers
214 views

Partial information decomposition for tangle machines

In (Williams and Beer, 2010), they define the partial information decomposition (PID) as a generalization of Shannon's Mutual Information for multiple information sources. Their key insight is that ...
momeara's user avatar
  • 211
2 votes
1 answer
246 views

Is Bregman divergence independent of coordinates?

Question Is Bregman divergence free of coordinates? Although it is invariant w.r.t. which local affine coordinate you take, is it possible to prove that it does not change w.r.t. an arbitrary change ...
diadochos's user avatar
  • 163
2 votes
1 answer
414 views

Complete statistical manifolds

Here, by a statistical manifold I mean a $d$-dimensional Riemannian manifold whose points are probability measures on $\mathbb{R}^n$. What are some well-studied/interesting examples of statistical ...
Catologist_who_flies_on_Monday's user avatar
2 votes
1 answer
200 views

Ideas on how to prove Pythagorean identity involving Wasserstein distances?

I conjectured earlier that if $P$ and $Q$ were two probability measures, then we could show $$W^2(P,Q) = \min_{T} [d^2(P,T_{\#}P) + W^2(T_{\#}P,Q)]$$ where $W^2(P,Q)$ denotes the squared Wasserstein-2 ...
Kashif's user avatar
  • 343
2 votes
1 answer
265 views

How much can KL divergence decrease by diluting the reference distribution

Let $\Omega$ be a countable set and $\mu,\nu\colon\Omega\to[0,1]$ be distributions on $\Omega$, that is we have $\sum_{x\in\Omega}\mu(x)=1$ and likewise for $\nu$. The Kullback-Leibler divergence of $\...
boinkboink's user avatar
2 votes
1 answer
100 views

What's meant by this measure in the sample space of a probability distribution?

In Amari's "Information Geometry and its Applications", in section 2.1 they define the exponential family as $$ p(x, \pmb\theta) = \exp \{ \theta^i h_i (x) + k(x) - \psi (\pmb\theta) \}$$ (...
ham burglar's user avatar
2 votes
1 answer
275 views

Convexity of exponential family

It is known that (given a $\sigma$-finite Borel reference measure $\nu$ on $\mathbb{R}$) the parameter space of an exponential family is convex in Euclidean space. However, my question is, for an the ...
ABIM's user avatar
  • 5,019
2 votes
1 answer
215 views

(quasi)metric on Riemannian manifolds via Brownian Motion?

Given points $a$ and $b$ on a Riemannian manifold $M$, I would like a (quasi)metric that corresponds to some property of Brownian Motion from $a$ to $b$ (or rather, to $\epsilon$-ball $B = \{ x : |x - ...
gusl's user avatar
  • 47
2 votes
0 answers
78 views

Joint lower semicontinuity of the Rényi divergence in all three arguments

Let $X$ be a standard Borel space (I'm already interested in the case where $X$ is finite, i.e., $X=\lbrace 1,\cdots,n\rbrace$). Let $P,Q$ be probability measures on $X$ such that $P\ll Q$. Then the ...
Lau's user avatar
  • 729
2 votes
0 answers
76 views

Is there any correspondence between Jacobi forms and automorphic forms on the unit ball in $\mathbb{C}^2$?

Apologies in advance if this question is obvious (or obviously false); number theory is far from my area of expertise. Let me state my questions and then I'll explain the motivation for asking them. ...
Gabe K's user avatar
  • 5,364
2 votes
0 answers
83 views

Rotationally symmetric manifold with statistical structure

Statistical manifolds are riemannian manifolds endowed with an affine torsion-free connection which is not necessarily Levi-Civita but verifies a weaker compatibility condition. More precisely, a ...
user2002's user avatar
  • 181
2 votes
0 answers
193 views

Inequality on the Kullback-Leibler divergence

Let us define the arithmetic, geometric, and harmonic means of $x,y \in \mathbb{R}$ weighted by $\alpha =(\alpha_x,\alpha_y) \in [0,1]$, respectively as \begin{equation} a_\alpha(x,y) = \frac{\...
Apprentice's user avatar
2 votes
0 answers
363 views

Connecting Wasserstein distance with mutual information?

Suppose I have Markov chains: $$X \rightarrow f(X) \rightarrow g(X)$$ $$Y \rightarrow f(Y) \rightarrow g(Y)$$ where it is known that minimizing the $\mathbb{E}(g(X)) - \mathbb{E}(g(Y))$ minimizes the ...
minimore99's user avatar
2 votes
1 answer
314 views

Does a 1-Lipschitz function preserve mutual information between two random variables?

Suppose we have a 1-Lipschitz function $f$ such that 1-Lipschitzness is preserved, with $D_A(f(X), f(Y)) \leq D_B(X, Y)$ for some metric spaces $A$ and $B$. Does this also imply that $I(f(X); f(Y)) = ...
minimore99's user avatar
2 votes
0 answers
633 views

Is there any geometric interpretation for the trace of Fisher information matrix?

Consider a parametric family $p_\theta(x)$ of distributions, with parameter $\theta \in \Theta \subseteq \mathbb R^p$. If the mapping $\theta \mapsto p_\theta(x)$ is continuously differentiable at $\...
dohmatob's user avatar
  • 6,716
2 votes
0 answers
128 views

Variational inference: Does the natural gradient follow (Fisher-Rao) geodesics locally?

Amari's natural gradient descent is a well-known optimisation algorithm for functionals defined on statistical manifolds. It consists of preconditioning the vanilla gradient descent update rule with ...
Lance's user avatar
  • 203
2 votes
0 answers
52 views

Minimizer of a class of SDEs

Setup Let $\mathscr{H}$ be a separable Hilbert space, $\mathcal{X}\triangleq \langle \Omega,\mathscr{F},\mathscr{F}_t,\mathbb{P}\rangle$ be a stochastic base and $X_t$ be an $H$-valued stochastic ...
ABIM's user avatar
  • 5,019
1 vote
1 answer
108 views

References: error and stability estimates for information projection

$\newcommand\SS{P}\newcommand\TT{Q}$I will call a Gaussian probability measure $\SS$ on $\mathbb{R}^d$ isotropic if its covariance matrix is diagonal with non-vanishing determinant; i.e. $\Sigma_{i,i}&...
Math_Newbie's user avatar
1 vote
1 answer
371 views

Relationship between the Fisher distance and Kulback Leibler divergence

I am reading the 2017 book "Information geometry" by Ay, Jost, Lê, Schwachhöfer. The Fisher distance is given by $$ d^F(\mu, \nu) := \inf_{\gamma} L(\gamma) $$ for curves $\gamma:[0,1]\to P$ ...
Felix B.'s user avatar
  • 347
1 vote
1 answer
241 views

Parameterization of exponential family

Let $\{\mathbb{P}_{\theta}\}_{\theta}$ be an exponential family of probability measures, all with finite mean. Under what conditions is the parameterization map $\theta\mapsto \mathbb{P}_{\theta}$ ...
ABIM's user avatar
  • 5,019
1 vote
0 answers
79 views

Representation theory for symmetries of probability distribution functions

I would like to parameterize all the possible modifications to a probability density function. Is there a representation theory for this? Something along the lines of, these are all the operators $L$ ...
Alex's user avatar
  • 119