Questions tagged [information-geometry]
Information geometry is a branch of mathematics that applies the techniques of differential geometry to the field of probability theory. This is done by taking probability distributions for a statistical model as the points of a Riemannian manifold, forming a statistical manifold. The Fisher information metric provides the Riemannian metric.
30
questions
3
votes
1answer
188 views
Which books should I read in order to be prepared to study information geometry?
At the moment, I am preparing my master's thesis (in statistics) and I intend to keep studying in order to pursue a doctoral degree. To be precise, I am mainly interested in studying Information ...
0
votes
0answers
55 views
Manifold structure of Gaussian mixtures
Fix $l$ a positive integer. Let $\mathcal{M}$ denote the set of Gaussian mixtures of the form
$$
\sum_{i=1}^l k_i \mu_i,
$$
where $\mu_i $ is a non-degenerate Gaussian measure on $\mathbb{R}^k$ and ...
1
vote
0answers
38 views
Bounding the total variation metric between Gaussian mixtures
Let $\mathcal{P}(\mathbb{R}^d)$ the space of probability measures on $\mathbb{R}^d$ with total variation metric $\delta$, fix $k \in \mathbb{N}$, and let $\mathcal{P}'\subset \mathcal{P}(\mathbb{R}^d)$...
2
votes
0answers
91 views
Inequality on the Kullback-Leibler divergence
Let us define the arithmetic, geometric, and harmonic means of $x,y \in \mathbb{R}$ weighted by $\alpha =(\alpha_x,\alpha_y) \in [0,1]$, respectively as
\begin{equation}
a_\alpha(x,y) = \frac{\...
0
votes
0answers
29 views
Relation between the covariance of a random variable taking values in an embedded submanifold and the covariance matrix in the ambient Euclidean space
Let $M^m \subset \mathbb{R}^d, m < d $ be an $m$-dimensional embedded submanifold. Let $X: \Omega \to M^m$ be a manifold valued random variable. Then we've apparently two different notions of ...
5
votes
1answer
185 views
Pythagorean theorems for other distances
Question
The usual projection in $\mathbb{R}^n$ on a subspace can be defined as the point that minimizes the squared distance to the subspace. I'll call the Pythagorean theorem the easy fact that, ...
1
vote
0answers
93 views
Connecting Wasserstein distance with mutual information?
Suppose I have Markov chains:
$$X \rightarrow f(X) \rightarrow g(X)$$
$$Y \rightarrow f(Y) \rightarrow g(Y)$$
where it is known that minimizing the $\mathbb{E}(g(X)) - \mathbb{E}(g(Y))$ minimizes the ...
1
vote
1answer
91 views
Does a 1-Lipschitz function preserve mutual information between two random variables?
Suppose we have a 1-Lipschitz function $f$ such that 1-Lipschitzness is preserved, with $D_A(f(X), f(Y)) \leq D_B(X, Y)$ for some metric spaces $A$ and $B$.
Does this also imply that $I(f(X); f(Y)) = ...
3
votes
3answers
286 views
Introduction to information geometry and/or geometric control theory
Some background: I'vebeen searching for a research project to work through my grad studies and I found information geometry like a strong candidate but the amount of work out there is overwhelming. I ...
2
votes
0answers
123 views
Is there any geometric interpretation for the trace of Fisher information matrix?
Consider a parametric family $p_\theta(x)$ of distributions, with parameter $\theta \in \Theta \subseteq \mathbb R^p$.
If the mapping $\theta \mapsto p_\theta(x)$ is continuously differentiable at $\...
11
votes
0answers
478 views
Relation between information geometry and geometric deep learning
Disclaimer: This is a cross-post from a very similar question on math.SE. I allowed myself to post it here after reading this meta
post about cross-posting between mathoverflow and math.SE, I did
...
2
votes
1answer
115 views
Relationship between $\alpha$-divergences?
I am working with $\alpha$-divergences and was wondering how understand the relationship between the definitions of Renyi and Amari?
Renyi:
$D_{\alpha}[p||q] = \frac{1}{\alpha - 1} \log \int p^{\...
4
votes
1answer
158 views
Convexity of exponential family
It is known that (given a $\sigma$-finite Borel reference measure $\nu$ on $\mathbb{R}$) the parameter space of an exponential family is convex in Euclidean space. However, my question is, for an the ...
2
votes
0answers
41 views
Variational inference: Does the natural gradient follow (Fisher-Rao) geodesics locally?
Amari's natural gradient descent is a well-known optimisation algorithm for functionals defined on statistical manifolds. It consists of preconditioning the vanilla gradient descent update rule with ...
1
vote
0answers
78 views
Binary search extension for determining a hyperplane splitting a set of points in $\mathbb{R}^d$
We are given a set $S$ of $n$ points in $\mathbb{R}^d$ and a (hidden) vector $\mathbf{w}\in\mathbb{R}^d$, where each point $\mathbf{v}\in S$ is associated with a binary label equal to the sign of $\...
1
vote
0answers
121 views
Jensen-Shannon Divergence of Sample Distributions
Given normal distributions with a single positional and variation parameter each, $p_1=\big[\mu_1, \sigma_1\big]$, $p_2=\big[\mu_2, \sigma_2\big]$, we define their Jensen-Shannon divergence as:
$$
\...
6
votes
2answers
554 views
Approximation of Wasserstein distance between $p_\theta$ and $p_{\theta + d\theta}$
Given a parametric family of distributions $\{p_\theta\mid\theta \in \Theta\}$, one can show that under some regularity conditions, the following approximation is valid
$$\operatorname{KL}(p_\theta\...
3
votes
1answer
122 views
Interpolation inequality related to the 5/3-Laplace operator
I'm having trouble with an estimate that would be helpful in information geometry.
The background is the following. Suppose we have a smooth positive function $g:X \to \mathbb{R}^+$ where $X$ is a ...
6
votes
5answers
1k views
Reviews of Probability in High Dimension not by Van Handel
I'm completely in love with Ramon van Handel's lecture notes Probability in High Dimension and I would like to find more learning resources. Lecture notes or reviews would be ideal as anything in this ...
2
votes
0answers
131 views
Parametric distances on product spaces of measures
Disclaimer: Please excuse my loose language. I'm neither an expert in geometry nor probability. Please ask for clarification if something appears unclear or awkward to you.
Let $X$ be a topological ...
3
votes
0answers
151 views
Distance measures that preserve Pythagoras' theorem but break the triangle inequality
In information geometry, we can think of the Kullback-Leibler divergence as being "something like a squared distance."
The sense of this is that if we have three probability measures, $P$, $Q$ and $R$...
2
votes
1answer
153 views
Is Bregman divergence independent of coordinates?
Question
Is Bregman divergence free of coordinates?
Although it is invariant w.r.t. which local affine coordinate you take, is it possible to prove that it does not change w.r.t. an arbitrary change ...
3
votes
0answers
138 views
Covariance operator analogue for manifolds and respective measure manifolds
Assume $E$ is a connected riemannian manifold with geodesic metric space structure given by $d$ and $P$ is a probability measure over $E$ with Borel sigma-algebra given by this metric structure. Also ...
3
votes
0answers
197 views
Partial information decomposition for tangle machines
In (Williams and Beer, 2010), they define the partial information decomposition (PID) as a generalization of Shannon's Mutual Information for multiple information sources. Their key insight is that ...
2
votes
0answers
43 views
Minimizer of a class of SDEs
Setup
Let $\mathscr{H}$ be a separable Hilbert space, $\mathcal{X}\triangleq \langle \Omega,\mathscr{F},\mathscr{F}_t,\mathbb{P}\rangle$ be a stochastic base and $X_t$ be an $H$-valued stochastic ...
2
votes
1answer
222 views
How much can KL divergence decrease by diluting the reference distribution
Let $\Omega$ be a countable set and $\mu,\nu\colon\Omega\to[0,1]$ be distributions on $\Omega$, that is we have $\sum_{x\in\Omega}\mu(x)=1$ and likewise for $\nu$. The Kullback-Leibler divergence of $\...
2
votes
1answer
184 views
(quasi)metric on Riemannian manifolds via Brownian Motion?
Given points $a$ and $b$ on a Riemannian manifold $M$, I would like a (quasi)metric that corresponds to some property of Brownian Motion from $a$ to $b$ (or rather, to $\epsilon$-ball $B = \{ x : |x - ...
4
votes
2answers
342 views
About optimization with Renyi divergence
Can someone link me to some pedagogic example of computing the Renyi divergence between two discrete/continuous distributions? Like examples where someone has been able to obtain a neat closed form or ...
1
vote
0answers
198 views
Relation between Aitchison Distance on a Simplex and Geodesic distance on the multinomial manifold [closed]
I am trying to understand the difference/relation between the Aitchison distance on a simplex
$$\left[ \sum^D_{k=1} (\log{\frac{x_{ik}}{g(\mathbf{x}_i)}} - \log{\frac{x_{jk}}{g(\mathbf{x}_j)}})^2 \...
36
votes
1answer
11k views
Research situation in the field of Information Geometry
I am now doing an article survey on the field of information geometry started by S.Amari and Barndorff-Nielson. I want to know some research situation in this field.
I have read (4) and parts of (3). ...
