All Questions
14 questions
3
votes
2
answers
1k
views
A general inequality for KL divergence of functions of variables
The question concerns a very general setting and a very general inequality about KL divergence. I'm writing this thread to verify whether my intuition is correct.
Let $E_1, E_2$ be two measurable ...
- 83
1
vote
1
answer
135
views
KL-divergence and sub-$\sigma$-algebras
I am trying to understand if the following claim is true:
Let $P$, $Q$ be probability measures on $\mathcal{X}$. For any $\sigma$-algebra $\mathcal{G}$, with countably many atoms (sets with $\...
- 13
2
votes
1
answer
181
views
Conditional entropy - solve example
Given a random variable $X$ that is uniformly distributed on $[-b,b]$ and $Y=g(X)$ with
$$g(x) = \begin{cases} 0, ~~~ x\in [-c,c] \\ x, ~~~ \text{else}\end{cases}$$
Now I want to compute the ...
- 131
2
votes
1
answer
294
views
Information theory for uncountably infinite-dimensional continuous random variable
I'm exploring the possibility to apply information theory on an uncountably infinite-dimensional scenario. I found the concept of generalized entropy for continuous random variables defined on finite-...
- 263
1
vote
0
answers
184
views
Bounding the total variation metric between Gaussian mixtures
Let $\mathcal{P}(\mathbb{R}^d)$ the space of probability measures on $\mathbb{R}^d$ with total variation metric $\delta$, fix $k \in \mathbb{N}$, and let $\mathcal{P}'\subset \mathcal{P}(\mathbb{R}^d)$...
- 5,405
8
votes
1
answer
355
views
Lower Bound of KL-Divergence Between Two Gibbs Measures
Suppose we have two Gibbs measures with densities
$$
p_f(x) \propto \exp(f(x)),\quad q_g(x)\propto \exp(g(x)).
$$
Consider the KL-divergence between $p_f$ and $q_g$, as a functional of $f$ and $g$, ...
- 1,127
4
votes
1
answer
352
views
Measure of the rate of convergence for filtration and conditional expectations
This question is cross-posted at MSE with a soon to expire bounty that hasn't generated much discussion.
Let $(\Omega, \mathcal{F},P)$ be a probability space and $(\mathcal{F}_n)_n$ a filtration that ...
- 869
3
votes
0
answers
428
views
When is the entropy of a $\sigma$-algebra finite?
Let two (countably-generated) $\sigma-$algebras $\mathscr{F,G}$ on the event space $\mathbb{R}$ be given. I believe we also need the atoms of $\mathscr{F,G}$ to be the points of $\mathbb{R}$.
Let $\...
- 2,680
9
votes
0
answers
223
views
Cramer's theorem in Hilbert spaces
I am interested under what conditions Cramer's theorem applies in random variables taking values in Hilbert spaces. Following these lecture notes, but using a Hilbert space:
Let $X_1,X_2,\cdots$, be ...
- 619
5
votes
0
answers
137
views
Large Deviations: Exponential decay in normed spaces
Let $(X_1,X_2,\cdots)$ be a sequence of independent and identically distributed random variables taking values in some general normed space $(V,||\cdot||)$. Denote $\mu=E[X_1]$ and $S_n=\frac{1}{n}[...
- 619
3
votes
0
answers
133
views
What distribution(s) of delays make(s) timing attacks hardest?
$H$ is (Shannon) entropy.
In terms of the positive real number $t$, what distribution(s) $\hspace{.01 in}X$ on $\:[0\hspace{.005 in},\hspace{-0.03 in}\scriptsize+\normalsize\infty\hspace{-0.02 in})$...
user5810
15
votes
3
answers
3k
views
Entropy of a measure
Let $\mu$ be a probability measure on a set of $n$ elements and let $p_i$ be the measure of the $i$-th element. Its Shannon entropy is defined by
$$
E(\mu)=-\sum_{i=1}^np_i\log(p_i)
$$
with the ...
- 6,034
5
votes
2
answers
679
views
distance in terms of the variance between two absolutely continuous probability measures
Consider two probability measures $\mu_0$ and $\mu_1$ on $\mathbb{R}^n$, such that $\mu_0\ll \mu_1$. Then I can define a "distance" like quantitiy
$$
\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\...
- 1,081
10
votes
2
answers
1k
views
Continuity of the mutual information
The mutual information $I(\mathfrak A_1;\mathfrak A_2)$ of two complete $\sigma$-algebras $\mathfrak A_1$ and $\mathfrak A_2$ in a Lebesgue probability space $(X,m)$ is the integral of the logarithm ...
- 17k