# Questions tagged [information-geometry]

Information geometry is a branch of mathematics that applies the techniques of differential geometry to the field of probability theory. This is done by taking probability distributions for a statistical model as the points of a Riemannian manifold, forming a statistical manifold. The Fisher information metric provides the Riemannian metric.

60
questions

6
votes

1
answer

331
views

### Is the gradient of a strictly convex, continuously differentiable function a homeomorphism?

Let $X\subseteq\mathbb{R}^n$ be a convex set. Let $f:X\to\mathbb{R}$ be a strictly convex function that is differentiable on the (non-empty) relative interior of $X$.
$\nabla f$ is a bijection, but is ...

1
vote

0
answers

77
views

### Representation theory for symmetries of probability distribution functions

I would like to parameterize all the possible modifications to a probability density function. Is there a representation theory for this? Something along the lines of, these are all the operators $L$ ...

2
votes

0
answers

72
views

### Joint lower semicontinuity of the Rényi divergence in all three arguments

Let $X$ be a standard Borel space (I'm already interested in the case where $X$ is finite, i.e., $X=\lbrace 1,\cdots,n\rbrace$). Let $P,Q$ be probability measures on $X$ such that $P\ll Q$. Then the ...

1
vote

1
answer

308
views

### Relationship between the Fisher distance and Kulback Leibler divergence

I am reading the 2017 book "Information geometry" by Ay, Jost, Lê, Schwachhöfer. The Fisher distance is given by
$$
d^F(\mu, \nu) := \inf_{\gamma} L(\gamma)
$$
for curves $\gamma:[0,1]\to P$ ...

0
votes

0
answers

33
views

### Obtaining metric and compatible differential equations on codimension one foliation of $n$-cube

Essentially the same content as this post on Math stack exchange: https://math.stackexchange.com/q/4741835/460999. I don't expect an answer there and after waiting a few days I've decided to post here....

0
votes

0
answers

22
views

### Canonical information geometry for probability distributions on different parameter spaces

I am interested in a canonical information geometry on spaces of probability distributions containing distributions with different parameter spaces. Let me give some context and practical motivation ...

7
votes

1
answer

324
views

### Singular Fisher information matrix and existence of unbiased estimators

I'm doing some research into the Cramer-Rao bound for time of arrival localization and have come across a rather strange result: the FIM is singular, but there exists an unbiased estimator. My ...

2
votes

0
answers

74
views

### Is there any correspondence between Jacobi forms and automorphic forms on the unit ball in $\mathbb{C}^2$?

Apologies in advance if this question is obvious (or obviously false); number theory is far from my area of expertise. Let me state my questions and then I'll explain the motivation for asking them.
...

3
votes

1
answer

118
views

### Conditions for: (local) lipschitz stability of I-projection

The following post builds on this post; I'll begin by quoting the setting.
Background from Previous Question:
$\newcommand\SS{P}\newcommand\TT{Q}$Call a Gaussian probability measure $\SS$ on $\mathbb{...

1
vote

1
answer

108
views

### References: error and stability estimates for information projection

$\newcommand\SS{P}\newcommand\TT{Q}$I will call a Gaussian probability measure $\SS$ on $\mathbb{R}^d$ isotropic if its covariance matrix is diagonal with non-vanishing determinant; i.e. $\Sigma_{i,i}&...

8
votes

1
answer

276
views

### Upper-bound on the Fisher-Rao distance between multivariate Gaussian measures by the KL-divergence

Let $\mu$ and $\nu$ be two multivariate Gaussian measures on $\mathbb{R}^d$ with non-singular covariance matrices. Can the Fisher-Rao distance $d(\mu,\nu)$ computed on the information manifold of non-...

1
vote

0
answers

56
views

### Question about canonical divergence on a dually flat manifold

I am reading "Methods of Information geometry by Shun-Ichi-Amari" (chapter 3 sec 3.4) and I am stuck here, can someone explain or give any resource about how we got equation $(3.53)$?

1
vote

0
answers

33
views

### Reference for a general theory of spaces of one-directional rays?

There is a lot of work done on projective spaces, over real, complex numbers or over an abstract field. But I do not find a reference for similar theory where the vectors are projected to the same &...

5
votes

0
answers

186
views

### Divergence for Bhattacharya Information matrix

The Fisher information matrix (in the scalar parameter case) can be obtained from the Kullback-Leibler divergence by
$$g(\theta) = -\frac{\partial}{\partial \theta}\frac{\partial}{\partial \theta'}D(...

4
votes

0
answers

112
views

### Exponential families closed under affine transformations

Let $(\Omega,\Sigma,\mu)$ be a probability space and let $\mathcal{M}$ be an exponential family of probability distributions for $\mu$ of the following form: There are $\varphi_1,\dots,\varphi_n:\...

2
votes

1
answer

99
views

### What's meant by this measure in the sample space of a probability distribution?

In Amari's "Information Geometry and its Applications", in section 2.1 they define the exponential family as
$$ p(x, \pmb\theta) = \exp \{ \theta^i h_i (x) + k(x) - \psi (\pmb\theta) \}$$
(...

3
votes

2
answers

322
views

### Direct calculation of the Fisher distance via Riemannian geodesics

I'm looking for a reference for a direct calculation of the Fisher distance (to avoid overloading the term "metric") $d_F(x,y) := 2 \cos^{-1} \sum_i \sqrt{x_i y_i}$ as the geodesic distance ...

4
votes

1
answer

114
views

### Gromov hyperbolicity for (non-geodesic) metrics on the upper-half plane invariant with respect to SL(2, R) action

$\DeclareMathOperator\SL{SL}$Let $d$ be a metric on the upper-half plane $\mathbb H = \{(x,y) : y > 0\}$ which is invariant with respect to the action of $\SL(2, \mathbb R)$ to $\mathbb H$ which is ...

2
votes

0
answers

81
views

### Rotationally symmetric manifold with statistical structure

Statistical manifolds are riemannian manifolds endowed with an affine torsion-free connection which is not necessarily Levi-Civita but verifies a weaker compatibility condition.
More precisely, a ...

1
vote

0
answers

72
views

### intuition about Gaussian processes over a finite space

In a paper that I am reading the authors defines $\mathbb P(n,q)$ the space of covariance tensors for $\mathbb R^q$-valued Gaussian processes on an abstract finite space $K=\{x_1,\dots,x_n\}$. In his ...

1
vote

0
answers

175
views

### Statistical manifolds with trivial statistical structure after quotienting

A statistical manifold $(M,g,\nabla)$ is a Riemannian manifold with a torsion-free affine connection $\nabla$ such that $\nabla g$ is symmetric in all entries. Equivalently, there is a dual affine ...

6
votes

1
answer

763
views

### reverse KL-divergence: Bregman or not?

I am having a little trouble getting my head around the two "directions" of the Kullback-Leibler divergence:
Definition (Kullback-Leibler divergence) For discrete probability distributions $...

1
vote

1
answer

238
views

### Parameterization of exponential family

Let $\{\mathbb{P}_{\theta}\}_{\theta}$ be an exponential family of probability measures, all with finite mean. Under what conditions is the parameterization map $\theta\mapsto \mathbb{P}_{\theta}$ ...

5
votes

2
answers

682
views

### Comparison of Information and Wasserstein Topologies

There are many possible metrics one can place on the space of Gaussian probability measures on $\mathbb{R}^n$, with strictly positive definite co-variance matrices. Let's denote this space by $X$.
I'...

2
votes

1
answer

406
views

### Complete statistical manifolds

Here, by a statistical manifold I mean a $d$-dimensional Riemannian manifold whose points are probability measures on $\mathbb{R}^n$. What are some well-studied/interesting examples of statistical ...

7
votes

2
answers

481
views

### Projections in infinite dimensional statistical manifolds

I'm struggling to understand the geometry of projection for infinite dimensional statistical manifolds. In finite dimensions, a strictly convex smooth function $F$ defines a Bregman divergence. From ...

7
votes

1
answer

315
views

### What is the correct notion of morphism between statistical manifolds?

Given two statistical manifolds, is there a notion of "isomorphic"? What are morphisms?

4
votes

1
answer

296
views

### Information monotonicity of divergence => function of $f$-divergence

It is well-known that $f$-divergences defined on $\mathcal P(\mathcal X)$ where $\mathcal X$ is a measure space with $\sigma$-algebra $\mathcal B$ satisfy the property of information monotonicity:
...

3
votes

0
answers

432
views

### From $f$-divergence to its dual: the transformation of convex functions on $\mathbb R_+$ by $f^*(t) = 1 f(\frac 1 t)$

I would like to understand the relationship between minimising the KL divergence $P \mapsto D_{KL}[P,Q]$ and the reverse KL divergence $P\mapsto D_{KL}^*[P,Q]=D_{KL}[Q,P]$ for probability measures $P$ ...

2
votes

1
answer

193
views

### Ideas on how to prove Pythagorean identity involving Wasserstein distances?

I conjectured earlier that if $P$ and $Q$ were two probability measures, then we could show
$$W^2(P,Q) = \min_{T} [d^2(P,T_{\#}P) + W^2(T_{\#}P,Q)]$$ where $W^2(P,Q)$ denotes the squared Wasserstein-2 ...

6
votes

1
answer

155
views

### Smallest $\mathrm{D}(Q\|P)$ given fixed marginals $\mathrm{D}(Q_X\|P_X)$ and $\mathrm{D}(Q_Y\|P_Y)$

Let $P$ be a distribution on a set $U\times V$ with marginal distributions $P_X$ and $P_Y$.
Suppose we have two values $d_x, d_y\in\mathbb R$, and we want to find the distribution $Q$ absolutely ...

5
votes

2
answers

1k
views

### How to study to learn differential geometry for applying it to statistics

Basically I want to learn information geometry or specifically the application of differential geometry in statistics to do a project. I am from a statistical background and have a knowledge about ...

11
votes

1
answer

3k
views

### Which books should I read in order to be prepared to study information geometry?

At the moment, I am preparing my master's thesis (in statistics) and I intend to keep studying in order to pursue a doctoral degree. To be precise, I am mainly interested in studying Information ...

1
vote

0
answers

166
views

### Bounding the total variation metric between Gaussian mixtures

Let $\mathcal{P}(\mathbb{R}^d)$ the space of probability measures on $\mathbb{R}^d$ with total variation metric $\delta$, fix $k \in \mathbb{N}$, and let $\mathcal{P}'\subset \mathcal{P}(\mathbb{R}^d)$...

2
votes

0
answers

189
views

### Inequality on the Kullback-Leibler divergence

Let us define the arithmetic, geometric, and harmonic means of $x,y \in \mathbb{R}$ weighted by $\alpha =(\alpha_x,\alpha_y) \in [0,1]$, respectively as
\begin{equation}
a_\alpha(x,y) = \frac{\...

5
votes

1
answer

400
views

### Pythagorean theorems for other distances

Question
The usual projection in $\mathbb{R}^n$ on a subspace can be defined as the point that minimizes the squared distance to the subspace. I'll call the Pythagorean theorem the easy fact that, ...

2
votes

0
answers

358
views

### Connecting Wasserstein distance with mutual information?

Suppose I have Markov chains:
$$X \rightarrow f(X) \rightarrow g(X)$$
$$Y \rightarrow f(Y) \rightarrow g(Y)$$
where it is known that minimizing the $\mathbb{E}(g(X)) - \mathbb{E}(g(Y))$ minimizes the ...

2
votes

1
answer

302
views

### Does a 1-Lipschitz function preserve mutual information between two random variables?

Suppose we have a 1-Lipschitz function $f$ such that 1-Lipschitzness is preserved, with $D_A(f(X), f(Y)) \leq D_B(X, Y)$ for some metric spaces $A$ and $B$.
Does this also imply that $I(f(X); f(Y)) = ...

8
votes

3
answers

1k
views

### Introduction to information geometry and/or geometric control theory

Some background: I'vebeen searching for a research project to work through my grad studies and I found information geometry like a strong candidate but the amount of work out there is overwhelming. I ...

2
votes

0
answers

615
views

### Is there any geometric interpretation for the trace of Fisher information matrix?

Consider a parametric family $p_\theta(x)$ of distributions, with parameter $\theta \in \Theta \subseteq \mathbb R^p$.
If the mapping $\theta \mapsto p_\theta(x)$ is continuously differentiable at $\...

23
votes

1
answer

4k
views

### Relation between information geometry and geometric deep learning

Disclaimer: This is a cross-post from a very similar question on math.SE. I allowed myself to post it here after reading this meta
post about cross-posting between mathoverflow and math.SE, I did
...

5
votes

2
answers

1k
views

### Relationship between $\alpha$-divergences?

I am working with $\alpha$-divergences and was wondering how understand the relationship between the definitions of Renyi and Amari?
Renyi:
$D_{\alpha}[p||q] = \frac{1}{\alpha - 1} \log \int p^{\...

2
votes

1
answer

275
views

### Convexity of exponential family

It is known that (given a $\sigma$-finite Borel reference measure $\nu$ on $\mathbb{R}$) the parameter space of an exponential family is convex in Euclidean space. However, my question is, for an the ...

2
votes

0
answers

125
views

### Variational inference: Does the natural gradient follow (Fisher-Rao) geodesics locally?

Amari's natural gradient descent is a well-known optimisation algorithm for functionals defined on statistical manifolds. It consists of preconditioning the vanilla gradient descent update rule with ...

1
vote

0
answers

90
views

### Binary search extension for determining a hyperplane splitting a set of points in $\mathbb{R}^d$

We are given a set $S$ of $n$ points in $\mathbb{R}^d$ and a (hidden) vector $\mathbf{w}\in\mathbb{R}^d$, where each point $\mathbf{v}\in S$ is associated with a binary label equal to the sign of $\...

1
vote

0
answers

202
views

### Jensen-Shannon Divergence of Sample Distributions

Given normal distributions with a single positional and variation parameter each, $p_1=\big[\mu_1, \sigma_1\big]$, $p_2=\big[\mu_2, \sigma_2\big]$, we define their Jensen-Shannon divergence as:
$$
\...

8
votes

2
answers

930
views

### Approximation of Wasserstein distance between $p_\theta$ and $p_{\theta + d\theta}$

Given a parametric family of distributions $\{p_\theta\mid\theta \in \Theta\}$, one can show that under some regularity conditions, the following approximation is valid
$$\operatorname{KL}(p_\theta\...

3
votes

1
answer

138
views

### Interpolation inequality related to the 5/3-Laplace operator

I'm having trouble with an estimate that would be helpful in information geometry.
The background is the following. Suppose we have a smooth positive function $g:X \to \mathbb{R}^+$ where $X$ is a ...

9
votes

5
answers

2k
views

### Reviews of Probability in High Dimension not by Van Handel

I'm completely in love with Ramon van Handel's lecture notes Probability in High Dimension and I would like to find more learning resources. Lecture notes or reviews would be ideal as anything in this ...

3
votes

0
answers

232
views

### Parametric distances on product spaces of measures

Disclaimer: Please excuse my loose language. I'm neither an expert in geometry nor probability. Please ask for clarification if something appears unclear or awkward to you.
Let $X$ be a topological ...