Questions tagged [information-geometry]

Information geometry is a branch of mathematics that applies the techniques of differential geometry to the field of probability theory. This is done by taking probability distributions for a statistical model as the points of a Riemannian manifold, forming a statistical manifold. The Fisher information metric provides the Riemannian metric.

Filter by
Sorted by
Tagged with
6 votes
1 answer
331 views

Is the gradient of a strictly convex, continuously differentiable function a homeomorphism?

Let $X\subseteq\mathbb{R}^n$ be a convex set. Let $f:X\to\mathbb{R}$ be a strictly convex function that is differentiable on the (non-empty) relative interior of $X$. $\nabla f$ is a bijection, but is ...
rick's user avatar
  • 121
1 vote
0 answers
77 views

Representation theory for symmetries of probability distribution functions

I would like to parameterize all the possible modifications to a probability density function. Is there a representation theory for this? Something along the lines of, these are all the operators $L$ ...
Alex's user avatar
  • 119
2 votes
0 answers
72 views

Joint lower semicontinuity of the Rényi divergence in all three arguments

Let $X$ be a standard Borel space (I'm already interested in the case where $X$ is finite, i.e., $X=\lbrace 1,\cdots,n\rbrace$). Let $P,Q$ be probability measures on $X$ such that $P\ll Q$. Then the ...
Lau's user avatar
  • 729
1 vote
1 answer
308 views

Relationship between the Fisher distance and Kulback Leibler divergence

I am reading the 2017 book "Information geometry" by Ay, Jost, Lê, Schwachhöfer. The Fisher distance is given by $$ d^F(\mu, \nu) := \inf_{\gamma} L(\gamma) $$ for curves $\gamma:[0,1]\to P$ ...
Felix B.'s user avatar
  • 347
0 votes
0 answers
33 views

Obtaining metric and compatible differential equations on codimension one foliation of $n$-cube

Essentially the same content as this post on Math stack exchange: https://math.stackexchange.com/q/4741835/460999. I don't expect an answer there and after waiting a few days I've decided to post here....
geocalc33's user avatar
0 votes
0 answers
22 views

Canonical information geometry for probability distributions on different parameter spaces

I am interested in a canonical information geometry on spaces of probability distributions containing distributions with different parameter spaces. Let me give some context and practical motivation ...
Lance's user avatar
  • 203
7 votes
1 answer
324 views

Singular Fisher information matrix and existence of unbiased estimators

I'm doing some research into the Cramer-Rao bound for time of arrival localization and have come across a rather strange result: the FIM is singular, but there exists an unbiased estimator. My ...
JNL's user avatar
  • 75
2 votes
0 answers
74 views

Is there any correspondence between Jacobi forms and automorphic forms on the unit ball in $\mathbb{C}^2$?

Apologies in advance if this question is obvious (or obviously false); number theory is far from my area of expertise. Let me state my questions and then I'll explain the motivation for asking them. ...
Gabe K's user avatar
  • 5,324
3 votes
1 answer
118 views

Conditions for: (local) lipschitz stability of I-projection

The following post builds on this post; I'll begin by quoting the setting. Background from Previous Question: $\newcommand\SS{P}\newcommand\TT{Q}$Call a Gaussian probability measure $\SS$ on $\mathbb{...
Math_Newbie's user avatar
1 vote
1 answer
108 views

References: error and stability estimates for information projection

$\newcommand\SS{P}\newcommand\TT{Q}$I will call a Gaussian probability measure $\SS$ on $\mathbb{R}^d$ isotropic if its covariance matrix is diagonal with non-vanishing determinant; i.e. $\Sigma_{i,i}&...
Math_Newbie's user avatar
8 votes
1 answer
276 views

Upper-bound on the Fisher-Rao distance between multivariate Gaussian measures by the KL-divergence

Let $\mu$ and $\nu$ be two multivariate Gaussian measures on $\mathbb{R}^d$ with non-singular covariance matrices. Can the Fisher-Rao distance $d(\mu,\nu)$ computed on the information manifold of non-...
Justin_other_PhD's user avatar
1 vote
0 answers
56 views

Question about canonical divergence on a dually flat manifold

I am reading "Methods of Information geometry by Shun-Ichi-Amari" (chapter 3 sec 3.4) and I am stuck here, can someone explain or give any resource about how we got equation $(3.53)$?
Andyale's user avatar
  • 111
1 vote
0 answers
33 views

Reference for a general theory of spaces of one-directional rays?

There is a lot of work done on projective spaces, over real, complex numbers or over an abstract field. But I do not find a reference for similar theory where the vectors are projected to the same &...
Michal Kupsa's user avatar
5 votes
0 answers
186 views

Divergence for Bhattacharya Information matrix

The Fisher information matrix (in the scalar parameter case) can be obtained from the Kullback-Leibler divergence by $$g(\theta) = -\frac{\partial}{\partial \theta}\frac{\partial}{\partial \theta'}D(...
Ashok's user avatar
  • 779
4 votes
0 answers
112 views

Exponential families closed under affine transformations

Let $(\Omega,\Sigma,\mu)$ be a probability space and let $\mathcal{M}$ be an exponential family of probability distributions for $\mu$ of the following form: There are $\varphi_1,\dots,\varphi_n:\...
ABIM's user avatar
  • 5,069
2 votes
1 answer
99 views

What's meant by this measure in the sample space of a probability distribution?

In Amari's "Information Geometry and its Applications", in section 2.1 they define the exponential family as $$ p(x, \pmb\theta) = \exp \{ \theta^i h_i (x) + k(x) - \psi (\pmb\theta) \}$$ (...
ham burglar's user avatar
3 votes
2 answers
322 views

Direct calculation of the Fisher distance via Riemannian geodesics

I'm looking for a reference for a direct calculation of the Fisher distance (to avoid overloading the term "metric") $d_F(x,y) := 2 \cos^{-1} \sum_i \sqrt{x_i y_i}$ as the geodesic distance ...
Steve Huntsman's user avatar
4 votes
1 answer
114 views

Gromov hyperbolicity for (non-geodesic) metrics on the upper-half plane invariant with respect to SL(2, R) action

$\DeclareMathOperator\SL{SL}$Let $d$ be a metric on the upper-half plane $\mathbb H = \{(x,y) : y > 0\}$ which is invariant with respect to the action of $\SL(2, \mathbb R)$ to $\mathbb H$ which is ...
Kazuki OKAMURA's user avatar
2 votes
0 answers
81 views

Rotationally symmetric manifold with statistical structure

Statistical manifolds are riemannian manifolds endowed with an affine torsion-free connection which is not necessarily Levi-Civita but verifies a weaker compatibility condition. More precisely, a ...
user2002's user avatar
  • 181
1 vote
0 answers
72 views

intuition about Gaussian processes over a finite space

In a paper that I am reading the authors defines $\mathbb P(n,q)$ the space of covariance tensors for $\mathbb R^q$-valued Gaussian processes on an abstract finite space $K=\{x_1,\dots,x_n\}$. In his ...
leo monsaingeon's user avatar
1 vote
0 answers
175 views

Statistical manifolds with trivial statistical structure after quotienting

A statistical manifold $(M,g,\nabla)$ is a Riemannian manifold with a torsion-free affine connection $\nabla$ such that $\nabla g$ is symmetric in all entries. Equivalently, there is a dual affine ...
user2002's user avatar
  • 181
6 votes
1 answer
763 views

reverse KL-divergence: Bregman or not?

I am having a little trouble getting my head around the two "directions" of the Kullback-Leibler divergence: Definition (Kullback-Leibler divergence) For discrete probability distributions $...
jw7642's user avatar
  • 91
1 vote
1 answer
238 views

Parameterization of exponential family

Let $\{\mathbb{P}_{\theta}\}_{\theta}$ be an exponential family of probability measures, all with finite mean. Under what conditions is the parameterization map $\theta\mapsto \mathbb{P}_{\theta}$ ...
ABIM's user avatar
  • 5,069
5 votes
2 answers
682 views

Comparison of Information and Wasserstein Topologies

There are many possible metrics one can place on the space of Gaussian probability measures on $\mathbb{R}^n$, with strictly positive definite co-variance matrices. Let's denote this space by $X$. I'...
Catologist_who_flies_on_Monday's user avatar
2 votes
1 answer
406 views

Complete statistical manifolds

Here, by a statistical manifold I mean a $d$-dimensional Riemannian manifold whose points are probability measures on $\mathbb{R}^n$. What are some well-studied/interesting examples of statistical ...
Catologist_who_flies_on_Monday's user avatar
7 votes
2 answers
481 views

Projections in infinite dimensional statistical manifolds

I'm struggling to understand the geometry of projection for infinite dimensional statistical manifolds. In finite dimensions, a strictly convex smooth function $F$ defines a Bregman divergence. From ...
John Rached's user avatar
7 votes
1 answer
315 views

What is the correct notion of morphism between statistical manifolds?

Given two statistical manifolds, is there a notion of "isomorphic"? What are morphisms?
user168590's user avatar
4 votes
1 answer
296 views

Information monotonicity of divergence => function of $f$-divergence

It is well-known that $f$-divergences defined on $\mathcal P(\mathcal X)$ where $\mathcal X$ is a measure space with $\sigma$-algebra $\mathcal B$ satisfy the property of information monotonicity: ...
Lance's user avatar
  • 203
3 votes
0 answers
432 views

From $f$-divergence to its dual: the transformation of convex functions on $\mathbb R_+$ by $f^*(t) = 1 f(\frac 1 t)$

I would like to understand the relationship between minimising the KL divergence $P \mapsto D_{KL}[P,Q]$ and the reverse KL divergence $P\mapsto D_{KL}^*[P,Q]=D_{KL}[Q,P]$ for probability measures $P$ ...
Lance's user avatar
  • 203
2 votes
1 answer
193 views

Ideas on how to prove Pythagorean identity involving Wasserstein distances?

I conjectured earlier that if $P$ and $Q$ were two probability measures, then we could show $$W^2(P,Q) = \min_{T} [d^2(P,T_{\#}P) + W^2(T_{\#}P,Q)]$$ where $W^2(P,Q)$ denotes the squared Wasserstein-2 ...
Kashif's user avatar
  • 343
6 votes
1 answer
155 views

Smallest $\mathrm{D}(Q\|P)$ given fixed marginals $\mathrm{D}(Q_X\|P_X)$ and $\mathrm{D}(Q_Y\|P_Y)$

Let $P$ be a distribution on a set $U\times V$ with marginal distributions $P_X$ and $P_Y$. Suppose we have two values $d_x, d_y\in\mathbb R$, and we want to find the distribution $Q$ absolutely ...
Thomas Dybdahl Ahle's user avatar
5 votes
2 answers
1k views

How to study to learn differential geometry for applying it to statistics

Basically I want to learn information geometry or specifically the application of differential geometry in statistics to do a project. I am from a statistical background and have a knowledge about ...
Avishek Dutta's user avatar
11 votes
1 answer
3k views

Which books should I read in order to be prepared to study information geometry?

At the moment, I am preparing my master's thesis (in statistics) and I intend to keep studying in order to pursue a doctoral degree. To be precise, I am mainly interested in studying Information ...
user avatar
1 vote
0 answers
166 views

Bounding the total variation metric between Gaussian mixtures

Let $\mathcal{P}(\mathbb{R}^d)$ the space of probability measures on $\mathbb{R}^d$ with total variation metric $\delta$, fix $k \in \mathbb{N}$, and let $\mathcal{P}'\subset \mathcal{P}(\mathbb{R}^d)$...
ABIM's user avatar
  • 5,069
2 votes
0 answers
189 views

Inequality on the Kullback-Leibler divergence

Let us define the arithmetic, geometric, and harmonic means of $x,y \in \mathbb{R}$ weighted by $\alpha =(\alpha_x,\alpha_y) \in [0,1]$, respectively as \begin{equation} a_\alpha(x,y) = \frac{\...
Apprentice's user avatar
5 votes
1 answer
400 views

Pythagorean theorems for other distances

Question The usual projection in $\mathbb{R}^n$ on a subspace can be defined as the point that minimizes the squared distance to the subspace. I'll call the Pythagorean theorem the easy fact that, ...
Aaron Bergman's user avatar
2 votes
0 answers
358 views

Connecting Wasserstein distance with mutual information?

Suppose I have Markov chains: $$X \rightarrow f(X) \rightarrow g(X)$$ $$Y \rightarrow f(Y) \rightarrow g(Y)$$ where it is known that minimizing the $\mathbb{E}(g(X)) - \mathbb{E}(g(Y))$ minimizes the ...
minimore99's user avatar
2 votes
1 answer
302 views

Does a 1-Lipschitz function preserve mutual information between two random variables?

Suppose we have a 1-Lipschitz function $f$ such that 1-Lipschitzness is preserved, with $D_A(f(X), f(Y)) \leq D_B(X, Y)$ for some metric spaces $A$ and $B$. Does this also imply that $I(f(X); f(Y)) = ...
minimore99's user avatar
8 votes
3 answers
1k views

Introduction to information geometry and/or geometric control theory

Some background: I'vebeen searching for a research project to work through my grad studies and I found information geometry like a strong candidate but the amount of work out there is overwhelming. I ...
Santiago Gil's user avatar
2 votes
0 answers
615 views

Is there any geometric interpretation for the trace of Fisher information matrix?

Consider a parametric family $p_\theta(x)$ of distributions, with parameter $\theta \in \Theta \subseteq \mathbb R^p$. If the mapping $\theta \mapsto p_\theta(x)$ is continuously differentiable at $\...
dohmatob's user avatar
  • 6,674
23 votes
1 answer
4k views

Relation between information geometry and geometric deep learning

Disclaimer: This is a cross-post from a very similar question on math.SE. I allowed myself to post it here after reading this meta post about cross-posting between mathoverflow and math.SE, I did ...
Blupon's user avatar
  • 333
5 votes
2 answers
1k views

Relationship between $\alpha$-divergences?

I am working with $\alpha$-divergences and was wondering how understand the relationship between the definitions of Renyi and Amari? Renyi: $D_{\alpha}[p||q] = \frac{1}{\alpha - 1} \log \int p^{\...
nico's user avatar
  • 91
2 votes
1 answer
275 views

Convexity of exponential family

It is known that (given a $\sigma$-finite Borel reference measure $\nu$ on $\mathbb{R}$) the parameter space of an exponential family is convex in Euclidean space. However, my question is, for an the ...
ABIM's user avatar
  • 5,069
2 votes
0 answers
125 views

Variational inference: Does the natural gradient follow (Fisher-Rao) geodesics locally?

Amari's natural gradient descent is a well-known optimisation algorithm for functionals defined on statistical manifolds. It consists of preconditioning the vanilla gradient descent update rule with ...
Lance's user avatar
  • 203
1 vote
0 answers
90 views

Binary search extension for determining a hyperplane splitting a set of points in $\mathbb{R}^d$

We are given a set $S$ of $n$ points in $\mathbb{R}^d$ and a (hidden) vector $\mathbf{w}\in\mathbb{R}^d$, where each point $\mathbf{v}\in S$ is associated with a binary label equal to the sign of $\...
Penelope Benenati's user avatar
1 vote
0 answers
202 views

Jensen-Shannon Divergence of Sample Distributions

Given normal distributions with a single positional and variation parameter each, $p_1=\big[\mu_1, \sigma_1\big]$, $p_2=\big[\mu_2, \sigma_2\big]$, we define their Jensen-Shannon divergence as: $$ \...
Matt Cuffaro's user avatar
8 votes
2 answers
930 views

Approximation of Wasserstein distance between $p_\theta$ and $p_{\theta + d\theta}$

Given a parametric family of distributions $\{p_\theta\mid\theta \in \Theta\}$, one can show that under some regularity conditions, the following approximation is valid $$\operatorname{KL}(p_\theta\...
dohmatob's user avatar
  • 6,674
3 votes
1 answer
138 views

Interpolation inequality related to the 5/3-Laplace operator

I'm having trouble with an estimate that would be helpful in information geometry. The background is the following. Suppose we have a smooth positive function $g:X \to \mathbb{R}^+$ where $X$ is a ...
Gabe K's user avatar
  • 5,324
9 votes
5 answers
2k views

Reviews of Probability in High Dimension not by Van Handel

I'm completely in love with Ramon van Handel's lecture notes Probability in High Dimension and I would like to find more learning resources. Lecture notes or reviews would be ideal as anything in this ...
user41147's user avatar
  • 263
3 votes
0 answers
232 views

Parametric distances on product spaces of measures

Disclaimer: Please excuse my loose language. I'm neither an expert in geometry nor probability. Please ask for clarification if something appears unclear or awkward to you. Let $X$ be a topological ...
dohmatob's user avatar
  • 6,674