Questions tagged [information-geometry]
Information geometry is a branch of mathematics that applies the techniques of differential geometry to the field of probability theory. This is done by taking probability distributions for a statistical model as the points of a Riemannian manifold, forming a statistical manifold. The Fisher information metric provides the Riemannian metric.
48
questions
1
vote
0
answers
29
views
Reference for a general theory of spaces of one-directional rays?
There is a lot of work done on projective spaces, over real, complex numbers or over an abstract field. But I do not find a reference for similar theory where the vectors are projected to the same &...
- 330
5
votes
1
answer
137
views
Divergence for Bhattacharya Information Matrix
The Fisher information matrix (in the scalar parameter case) can be obtained from the Kullback-Leibler divergence by
$$g(\theta) = -\frac{\partial}{\partial \theta}\frac{\partial}{\partial \theta'}D(...
- 769
4
votes
0
answers
55
views
Exponential families closed under affine transformations
Let $(\Omega,\Sigma,\mu)$ be a probability space and let $\mathcal{M}$ be an exponential family of probability distributions for $\mu$ of the following form: There are $\varphi_1,\dots,\varphi_n:\...
- 5,124
2
votes
1
answer
80
views
What's meant by this measure in the sample space of a probability distribution?
In Amari's "Information Geometry and its Applications", in section 2.1 they define the exponential family as
$$ p(x, \pmb\theta) = \exp \{ \theta^i h_i (x) + k(x) - \psi (\pmb\theta) \}$$
(...
- 41
2
votes
2
answers
246
views
Direct calculation of the Fisher distance via Riemannian geodesics
I'm looking for a reference for a direct calculation of the Fisher distance (to avoid overloading the term "metric") $d_F(x,y) := 2 \cos^{-1} \sum_i \sqrt{x_i y_i}$ as the geodesic distance ...
- 14.8k
4
votes
1
answer
100
views
Gromov hyperbolicity for (non-geodesic) metrics on the upper-half plane invariant with respect to SL(2, R) action
$\DeclareMathOperator\SL{SL}$Let $d$ be a metric on the upper-half plane $\mathbb H = \{(x,y) : y > 0\}$ which is invariant with respect to the action of $\SL(2, \mathbb R)$ to $\mathbb H$ which is ...
2
votes
0
answers
64
views
Rotationally symmetric manifold with statistical structure
Statistical manifolds are riemannian manifolds endowed with an affine torsion-free connection which is not necessarily Levi-Civita but verifies a weaker compatibility condition.
More precisely, a ...
- 181
1
vote
0
answers
63
views
intuition about Gaussian processes over a finite space
In a paper that I am reading the authors defines $\mathbb P(n,q)$ the space of covariance tensors for $\mathbb R^q$-valued Gaussian processes on an abstract finite space $K=\{x_1,\dots,x_n\}$. In his ...
- 3,755
1
vote
0
answers
159
views
Statistical manifolds with trivial statistical structure after quotienting
A statistical manifold $(M,g,\nabla)$ is a Riemannian manifold with a torsion-free affine connection $\nabla$ such that $\nabla g$ is symmetric in all entries. Equivalently, there is a dual affine ...
- 181
6
votes
1
answer
477
views
reverse KL-divergence: Bregman or not?
I am having a little trouble getting my head around the two "directions" of the Kullback-Leibler divergence:
Definition (Kullback-Leibler divergence) For discrete probability distributions $...
- 91
1
vote
1
answer
138
views
Parameterization of exponential family
Let $\{\mathbb{P}_{\theta}\}_{\theta}$ be an exponential family of probability measures, all with finite mean. Under what conditions is the parameterization map $\theta\mapsto \mathbb{P}_{\theta}$ ...
- 5,124
5
votes
2
answers
329
views
Comparison of Information and Wasserstein Topologies
There are many possible metrics one can place on the space of Gaussian probability measures on $\mathbb{R}^n$, with strictly positive definite co-variance matrices. Let's denote this space by $X$.
I'...
1
vote
1
answer
195
views
Complete statistical manifolds
Here, by a statistical manifold I mean a $d$-dimensional Riemannian manifold whose points are probability measures on $\mathbb{R}^n$. What are some well-studied/interesting examples of statistical ...
7
votes
2
answers
417
views
Projections in infinite dimensional statistical manifolds
I'm struggling to understand the geometry of projection for infinite dimensional statistical manifolds. In finite dimensions, a strictly convex smooth function $F$ defines a Bregman divergence. From ...
- 131
6
votes
1
answer
268
views
What is the correct notion of morphism between statistical manifolds?
Given two statistical manifolds, is there a notion of "isomorphic"? What are morphisms?
- 123
3
votes
1
answer
135
views
Information monotonicity of divergence => function of $f$-divergence
It is well-known that $f$-divergences defined on $\mathcal P(\mathcal X)$ where $\mathcal X$ is a measure space with $\sigma$-algebra $\mathcal B$ satisfy the property of information monotonicity:
...
- 191
2
votes
0
answers
173
views
From $f$-divergence to its dual: the transformation of convex functions on $\mathbb R_+$ by $f^*(t) = 1 f(\frac 1 t)$
I would like to understand the relationship between minimising the KL divergence $P \mapsto D_{KL}[P,Q]$ and the reverse KL divergence $P\mapsto D_{KL}^*[P,Q]=D_{KL}[Q,P]$ for probability measures $P$ ...
- 191
2
votes
1
answer
137
views
Ideas on how to prove Pythagorean identity involving Wasserstein distances?
I conjectured earlier that if $P$ and $Q$ were two probability measures, then we could show
$$W^2(P,Q) = \min_{T} [d^2(P,T_{\#}P) + W^2(T_{\#}P,Q)]$$ where $W^2(P,Q)$ denotes the squared Wasserstein-2 ...
- 323
6
votes
1
answer
136
views
Smallest $\mathrm{D}(Q\|P)$ given fixed marginals $\mathrm{D}(Q_X\|P_X)$ and $\mathrm{D}(Q_Y\|P_Y)$
Let $P$ be a distribution on a set $U\times V$ with marginal distributions $P_X$ and $P_Y$.
Suppose we have two values $d_x, d_y\in\mathbb R$, and we want to find the distribution $Q$ absolutely ...
5
votes
2
answers
873
views
How to study to learn differential geometry for applying it to statistics
Basically I want to learn information geometry or specifically the application of differential geometry in statistics to do a project. I am from a statistical background and have a knowledge about ...
10
votes
1
answer
1k
views
Which books should I read in order to be prepared to study information geometry?
At the moment, I am preparing my master's thesis (in statistics) and I intend to keep studying in order to pursue a doctoral degree. To be precise, I am mainly interested in studying Information ...
user127351
1
vote
0
answers
78
views
Bounding the total variation metric between Gaussian mixtures
Let $\mathcal{P}(\mathbb{R}^d)$ the space of probability measures on $\mathbb{R}^d$ with total variation metric $\delta$, fix $k \in \mathbb{N}$, and let $\mathcal{P}'\subset \mathcal{P}(\mathbb{R}^d)$...
- 5,124
2
votes
0
answers
162
views
Inequality on the Kullback-Leibler divergence
Let us define the arithmetic, geometric, and harmonic means of $x,y \in \mathbb{R}$ weighted by $\alpha =(\alpha_x,\alpha_y) \in [0,1]$, respectively as
\begin{equation}
a_\alpha(x,y) = \frac{\...
- 235
5
votes
1
answer
350
views
Pythagorean theorems for other distances
Question
The usual projection in $\mathbb{R}^n$ on a subspace can be defined as the point that minimizes the squared distance to the subspace. I'll call the Pythagorean theorem the easy fact that, ...
- 3,858
2
votes
0
answers
229
views
Connecting Wasserstein distance with mutual information?
Suppose I have Markov chains:
$$X \rightarrow f(X) \rightarrow g(X)$$
$$Y \rightarrow f(Y) \rightarrow g(Y)$$
where it is known that minimizing the $\mathbb{E}(g(X)) - \mathbb{E}(g(Y))$ minimizes the ...
- 57
2
votes
1
answer
227
views
Does a 1-Lipschitz function preserve mutual information between two random variables?
Suppose we have a 1-Lipschitz function $f$ such that 1-Lipschitzness is preserved, with $D_A(f(X), f(Y)) \leq D_B(X, Y)$ for some metric spaces $A$ and $B$.
Does this also imply that $I(f(X); f(Y)) = ...
- 57
6
votes
3
answers
775
views
Introduction to information geometry and/or geometric control theory
Some background: I'vebeen searching for a research project to work through my grad studies and I found information geometry like a strong candidate but the amount of work out there is overwhelming. I ...
- 61
2
votes
0
answers
391
views
Is there any geometric interpretation for the trace of Fisher information matrix?
Consider a parametric family $p_\theta(x)$ of distributions, with parameter $\theta \in \Theta \subseteq \mathbb R^p$.
If the mapping $\theta \mapsto p_\theta(x)$ is continuously differentiable at $\...
- 5,830
22
votes
1
answer
2k
views
Relation between information geometry and geometric deep learning
Disclaimer: This is a cross-post from a very similar question on math.SE. I allowed myself to post it here after reading this meta
post about cross-posting between mathoverflow and math.SE, I did
...
- 323
3
votes
2
answers
360
views
Relationship between $\alpha$-divergences?
I am working with $\alpha$-divergences and was wondering how understand the relationship between the definitions of Renyi and Amari?
Renyi:
$D_{\alpha}[p||q] = \frac{1}{\alpha - 1} \log \int p^{\...
- 31
2
votes
1
answer
209
views
Convexity of exponential family
It is known that (given a $\sigma$-finite Borel reference measure $\nu$ on $\mathbb{R}$) the parameter space of an exponential family is convex in Euclidean space. However, my question is, for an the ...
- 5,124
2
votes
0
answers
77
views
Variational inference: Does the natural gradient follow (Fisher-Rao) geodesics locally?
Amari's natural gradient descent is a well-known optimisation algorithm for functionals defined on statistical manifolds. It consists of preconditioning the vanilla gradient descent update rule with ...
- 191
1
vote
0
answers
85
views
Binary search extension for determining a hyperplane splitting a set of points in $\mathbb{R}^d$
We are given a set $S$ of $n$ points in $\mathbb{R}^d$ and a (hidden) vector $\mathbf{w}\in\mathbb{R}^d$, where each point $\mathbf{v}\in S$ is associated with a binary label equal to the sign of $\...
- 1,173
1
vote
0
answers
174
views
Jensen-Shannon Divergence of Sample Distributions
Given normal distributions with a single positional and variation parameter each, $p_1=\big[\mu_1, \sigma_1\big]$, $p_2=\big[\mu_2, \sigma_2\big]$, we define their Jensen-Shannon divergence as:
$$
\...
- 417
8
votes
2
answers
806
views
Approximation of Wasserstein distance between $p_\theta$ and $p_{\theta + d\theta}$
Given a parametric family of distributions $\{p_\theta\mid\theta \in \Theta\}$, one can show that under some regularity conditions, the following approximation is valid
$$\operatorname{KL}(p_\theta\...
- 5,830
3
votes
1
answer
134
views
Interpolation inequality related to the 5/3-Laplace operator
I'm having trouble with an estimate that would be helpful in information geometry.
The background is the following. Suppose we have a smooth positive function $g:X \to \mathbb{R}^+$ where $X$ is a ...
- 4,381
7
votes
5
answers
2k
views
Reviews of Probability in High Dimension not by Van Handel
I'm completely in love with Ramon van Handel's lecture notes Probability in High Dimension and I would like to find more learning resources. Lecture notes or reviews would be ideal as anything in this ...
- 243
3
votes
0
answers
187
views
Parametric distances on product spaces of measures
Disclaimer: Please excuse my loose language. I'm neither an expert in geometry nor probability. Please ask for clarification if something appears unclear or awkward to you.
Let $X$ be a topological ...
- 5,830
4
votes
0
answers
202
views
Distance measures that preserve Pythagoras' theorem but break the triangle inequality
In information geometry, we can think of the Kullback-Leibler divergence as being "something like a squared distance."
The sense of this is that if we have three probability measures, $P$, $Q$ and $R$...
- 1,210
2
votes
1
answer
208
views
Is Bregman divergence independent of coordinates?
Question
Is Bregman divergence free of coordinates?
Although it is invariant w.r.t. which local affine coordinate you take, is it possible to prove that it does not change w.r.t. an arbitrary change ...
- 163
3
votes
0
answers
187
views
Covariance operator analogue for manifolds and respective measure manifolds
Assume $E$ is a connected riemannian manifold with geodesic metric space structure given by $d$ and $P$ is a probability measure over $E$ with Borel sigma-algebra given by this metric structure. Also ...
- 327
3
votes
0
answers
207
views
Partial information decomposition for tangle machines
In (Williams and Beer, 2010), they define the partial information decomposition (PID) as a generalization of Shannon's Mutual Information for multiple information sources. Their key insight is that ...
- 211
2
votes
0
answers
47
views
Minimizer of a class of SDEs
Setup
Let $\mathscr{H}$ be a separable Hilbert space, $\mathcal{X}\triangleq \langle \Omega,\mathscr{F},\mathscr{F}_t,\mathbb{P}\rangle$ be a stochastic base and $X_t$ be an $H$-valued stochastic ...
- 5,124
2
votes
1
answer
239
views
How much can KL divergence decrease by diluting the reference distribution
Let $\Omega$ be a countable set and $\mu,\nu\colon\Omega\to[0,1]$ be distributions on $\Omega$, that is we have $\sum_{x\in\Omega}\mu(x)=1$ and likewise for $\nu$. The Kullback-Leibler divergence of $\...
- 83
2
votes
1
answer
192
views
(quasi)metric on Riemannian manifolds via Brownian Motion?
Given points $a$ and $b$ on a Riemannian manifold $M$, I would like a (quasi)metric that corresponds to some property of Brownian Motion from $a$ to $b$ (or rather, to $\epsilon$-ball $B = \{ x : |x - ...
- 21
4
votes
2
answers
510
views
About optimization with Renyi divergence
Can someone link me to some pedagogic example of computing the Renyi divergence between two discrete/continuous distributions? Like examples where someone has been able to obtain a neat closed form or ...
- 2,006
1
vote
0
answers
295
views
Relation between Aitchison Distance on a Simplex and Geodesic distance on the multinomial manifold [closed]
I am trying to understand the difference/relation between the Aitchison distance on a simplex
$$\left[ \sum^D_{k=1} (\log{\frac{x_{ik}}{g(\mathbf{x}_i)}} - \log{\frac{x_{jk}}{g(\mathbf{x}_j)}})^2 \...
- 111
45
votes
2
answers
12k
views
Research situation in the field of Information Geometry
I am now doing an article survey on the field of information geometry started by S.Amari and Barndorff-Nielson. I want to know some research situation in this field.
I have read (4) and parts of (3). ...
- 7,523