All Questions
Tagged with it.information-theory measure-theory
20 questions
15
votes
3
answers
3k
views
Entropy of a measure
Let $\mu$ be a probability measure on a set of $n$ elements and let $p_i$ be the measure of the $i$-th element. Its Shannon entropy is defined by
$$
E(\mu)=-\sum_{i=1}^np_i\log(p_i)
$$
with the ...
- 6,034
12
votes
2
answers
1k
views
Entropy for Haar measure on $O(n)$
Let $G$ be a locally compact group. A measure $\mu$ is the right-Haar measure on $G$ if for every $g\in G$ and $E\subseteq G$ Borel set $\mu(Eg)=\mu(E)$. It is known that every locally compact group ...
- 980
10
votes
2
answers
1k
views
Continuity of the mutual information
The mutual information $I(\mathfrak A_1;\mathfrak A_2)$ of two complete $\sigma$-algebras $\mathfrak A_1$ and $\mathfrak A_2$ in a Lebesgue probability space $(X,m)$ is the integral of the logarithm ...
- 17k
9
votes
0
answers
223
views
Cramer's theorem in Hilbert spaces
I am interested under what conditions Cramer's theorem applies in random variables taking values in Hilbert spaces. Following these lecture notes, but using a Hilbert space:
Let $X_1,X_2,\cdots$, be ...
- 619
8
votes
1
answer
355
views
Lower Bound of KL-Divergence Between Two Gibbs Measures
Suppose we have two Gibbs measures with densities
$$
p_f(x) \propto \exp(f(x)),\quad q_g(x)\propto \exp(g(x)).
$$
Consider the KL-divergence between $p_f$ and $q_g$, as a functional of $f$ and $g$, ...
- 1,127
6
votes
3
answers
1k
views
Entropy of a general prob. measure [closed]
How is entropy of a general probability measure defined?
- 779
5
votes
2
answers
848
views
Can the differential entropy of a continuous distribution with lebesgue integrable density be negative infinity?
Define the (differential) entropy for a density $f$ as
$$ H(f) :=-\int_{0}^{1} f(x) \log_{2}(f(x)) dx \, .$$
I am trying to find a $f \in L_{1}([0,1])$ such that $f\geq 0, \int_{0}^{1} f(x) dx = 1$...
5
votes
2
answers
679
views
distance in terms of the variance between two absolutely continuous probability measures
Consider two probability measures $\mu_0$ and $\mu_1$ on $\mathbb{R}^n$, such that $\mu_0\ll \mu_1$. Then I can define a "distance" like quantitiy
$$
\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\...
- 1,081
5
votes
0
answers
137
views
Large Deviations: Exponential decay in normed spaces
Let $(X_1,X_2,\cdots)$ be a sequence of independent and identically distributed random variables taking values in some general normed space $(V,||\cdot||)$. Denote $\mu=E[X_1]$ and $S_n=\frac{1}{n}[...
- 619
4
votes
1
answer
352
views
Measure of the rate of convergence for filtration and conditional expectations
This question is cross-posted at MSE with a soon to expire bounty that hasn't generated much discussion.
Let $(\Omega, \mathcal{F},P)$ be a probability space and $(\mathcal{F}_n)_n$ a filtration that ...
- 869
3
votes
2
answers
1k
views
A general inequality for KL divergence of functions of variables
The question concerns a very general setting and a very general inequality about KL divergence. I'm writing this thread to verify whether my intuition is correct.
Let $E_1, E_2$ be two measurable ...
- 83
3
votes
0
answers
428
views
When is the entropy of a $\sigma$-algebra finite?
Let two (countably-generated) $\sigma-$algebras $\mathscr{F,G}$ on the event space $\mathbb{R}$ be given. I believe we also need the atoms of $\mathscr{F,G}$ to be the points of $\mathbb{R}$.
Let $\...
- 2,680
3
votes
0
answers
133
views
What distribution(s) of delays make(s) timing attacks hardest?
$H$ is (Shannon) entropy.
In terms of the positive real number $t$, what distribution(s) $\hspace{.01 in}X$ on $\:[0\hspace{.005 in},\hspace{-0.03 in}\scriptsize+\normalsize\infty\hspace{-0.02 in})$...
user5810
2
votes
1
answer
293
views
Information theory for uncountably infinite-dimensional continuous random variable
I'm exploring the possibility to apply information theory on an uncountably infinite-dimensional scenario. I found the concept of generalized entropy for continuous random variables defined on finite-...
- 263
2
votes
1
answer
181
views
Conditional entropy - solve example
Given a random variable $X$ that is uniformly distributed on $[-b,b]$ and $Y=g(X)$ with
$$g(x) = \begin{cases} 0, ~~~ x\in [-c,c] \\ x, ~~~ \text{else}\end{cases}$$
Now I want to compute the ...
- 131
2
votes
0
answers
234
views
Characterizing the optimimum over the space of probability measures
Consider the following optimization problem:
\begin{equation}
\max_{\mu \in \mathcal{M}} \int \log\left( \int e^{\alpha U(x,y)} d\mu(y) \right) d\nu(x)
\end{equation}
where $\mathcal{M}$ is the space ...
- 21
1
vote
1
answer
3k
views
basic measure theory question - measure on the natural numbers [closed]
I am looking for a succinct way to describe a subset of the natural numbers which has ``measure zero" in the following sense: Let X_1 \subset X_2 \subset .... be any strictly nesting sequence of ...
1
vote
1
answer
137
views
Gelfand–Yaglom–Perez Theorem for product space
In On a Gel'fand-Yaglom-Peres theorem for f-divergences, Gilardoni proved the Gelfand–Yaglom–Perez Theorem for general $f$-divergence, i.e. $f$-divergence between two probability measures $P$ and $Q$ ...
- 113
1
vote
1
answer
134
views
KL-divergence and sub-$\sigma$-algebras
I am trying to understand if the following claim is true:
Let $P$, $Q$ be probability measures on $\mathcal{X}$. For any $\sigma$-algebra $\mathcal{G}$, with countably many atoms (sets with $\...
- 13
1
vote
0
answers
184
views
Bounding the total variation metric between Gaussian mixtures
Let $\mathcal{P}(\mathbb{R}^d)$ the space of probability measures on $\mathbb{R}^d$ with total variation metric $\delta$, fix $k \in \mathbb{N}$, and let $\mathcal{P}'\subset \mathcal{P}(\mathbb{R}^d)$...
- 5,405