# Questions tagged [it.information-theory]

Theoretical and experimental aspects of information theory and coding theory. This tag covers but is not limited to following branches: information theory, information geometry, optimal transportation theory, coding theory.

453 questions
Filter by
Sorted by
Tagged with
111 views

### $\varepsilon$-net of a $d$-dimensional unit ball formed by power set of $V = \{+1, 0 -1\}^d$

I have a set of $d$-dimensional vectors $V = \{+1, 0, -1\}^d$. Then $P(V)$ constitutes the power set of $V$. I now construct a set of unit vectors $V_{\mathrm{sum}}$ from the power set $P(V)$ such ...
183 views

### Can information be extracted more precisely using more random trials?

Write $n$ iid draws of $(x,y)$ as $(x^n, y^n)$. Fix $R\in (0,H(y))$. What is the min of $n^{-1}H(x^n|f(y^n))$ over $f$ with $H(f(y^n))\leq nR$, taking $n\to \infty$?
59 views

### Manifold structure of Gaussian mixtures

Fix $l$ a positive integer. Let $\mathcal{M}$ denote the set of Gaussian mixtures of the form $$\sum_{i=1}^l k_i \mu_i,$$ where $\mu_i$ is a non-degenerate Gaussian measure on $\mathbb{R}^k$ and ...
39 views

### Bounding the total variation metric between Gaussian mixtures

Let $\mathcal{P}(\mathbb{R}^d)$ the space of probability measures on $\mathbb{R}^d$ with total variation metric $\delta$, fix $k \in \mathbb{N}$, and let $\mathcal{P}'\subset \mathcal{P}(\mathbb{R}^d)$...
121 views

### Books to develop a unified view of statistics and information theory?

I hope to understand the connection between statistics and information theory in a deep philosophical sense. I suppose the best place to start would be David MacKay's Information Theory, Inference, ...
93 views

### Inequality on the Kullback-Leibler divergence

Let us define the arithmetic, geometric, and harmonic means of $x,y \in \mathbb{R}$ weighted by $\alpha =(\alpha_x,\alpha_y) \in [0,1]$, respectively as \begin{equation} a_\alpha(x,y) = \frac{\...
31 views

### Do d “moments of surprisal” determine a probability distribution on d events?

(The following question was transferred from stack-exchange with the hope of obtain a useful hint here). Consider a probability distribution on $d$ events, with the probabilities $p_j$ gathered in a ...
138 views

### Statistical divergence

Does anyone know about a statistical divergence of this type? \begin{equation} \text{D}(P||Q) = \frac{1}{2} \left[\text{KL}(M||P) + \text{KL}(M||Q)\right] \end{equation} where $M = \frac{1}{2} [P+Q]$....
49 views