Questions tagged [entropy]
The entropy tag has no usage guidance.
36
questions
1
vote
1
answer
124
views
Limiting Entropy of deterministic sequences - 1
Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$.
Consider distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_{a,m}$ be distribution at ...
- 13.7k
17
votes
4
answers
2k
views
Geometric interpretations of the exponential of entropy
Question:
Might there be a natural geometric interpretation of the exponential of entropy in Classical and Quantum Information theory? This question occurred to me recently via a geometric inequality ...
- 3,827
6
votes
1
answer
3k
views
Covering number of Lipschitz functions
What do we know about the covering number of $L$-Lipschitz functions mapping say, $\mathbb{R}^n \rightarrow \mathbb{R}$ for some $L >0$?
Only 2 results I have found so far are,
That the $\infty$-...
- 2,146
4
votes
0
answers
2k
views
metric entropy for Lipschitz functions
Suppose $(X,d)$ is a metric space of unit diameer and let $F$ be the collection of all $1$-Lipschitz functions mapping $X$ to $[-1,1]$, equipped with the sup-norm $||\cdot||_\infty$.
I am interested ...
- 6,061
2
votes
1
answer
126
views
Entropy dominance
Let $0<a<b<c$ be distinct positive reals.
Define four different probability distributions:
$$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$
$$\mathcal{P}_{bc}:P_{b,bc}=\frac{b}{b+c}=1-...
- 13.7k
1
vote
1
answer
394
views
joining or coupling
given two shift invariant measures in the Bernoulli space $\{0,1\}^{\mathbb{N}}$, is there a way to construct joinings of them? It's very diffcult, in general, to find exactly the minimal joining i.e, ...
1
vote
1
answer
101
views
Entropy difference dominance of sequences
Consider a collection of positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$.
Similarly for the collection $\{a_i\}_{i=1}^{m+1}$ form the distribution $q_i=\...
- 13.7k
37
votes
3
answers
3k
views
An entropy inequality
Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...
- 11.3k
32
votes
3
answers
2k
views
Intrinsic significance of differential entropy
Many commentators (e.g. Jaynes, Rota) argue that the notion of "differential entropy" is problematic (as commonly defined by $ h(X) = \int ( \log\frac{1}{p(x)} ) p(x) \, dx $, where $X$ is a random ...
- 1,203
18
votes
8
answers
3k
views
When do people actually use the maximum entropy distribution?
One of the standard problems in convex optimization is the calculation of the maximum entropy distribution that satisfies some set of criteria. For example, if $\mathbf{x} \in \mathbb R^n$ is an ...
- 183
18
votes
3
answers
3k
views
Entropy and total variation distance
Let $X$, $Y$ be discrete random variables taking values within the same set of $N$ elements. Let the total variation distance $|P-Q|$ (which is half the $L_1$ distance between the distributions of $P$ ...
- 19.3k
17
votes
2
answers
914
views
Is there an axiomatic characterization of the entropy of a continuous random variable?
Let $X$ be a random variable taking values in $\{1,\ldots,n\}$, and let $p_i$ denote the probability of the event $\{X = i\}$. Shannon defined the entropy of $X$ to be the quantity
$$H(X) = -\sum_i ...
- 28.8k
9
votes
1
answer
601
views
Heat flow, decay of the Fisher information, and $\lambda$-displacement convexity
In the whole post I will work in the flat torus $\mathbb T^d=\mathbb R^d/\mathbb Z^d$ and $\rho$ will stand for any probability measure $\mathcal P(\mathbb T^d)$. This question is strongly related to ...
- 5,163
9
votes
3
answers
2k
views
Gaussian distribution, maximum entropy and the heat equation
I have asked this question on MathSE, but I got no replies, so I thought of trying here.
Consider the Gaussian distribution on $\mathbb{R}$ with mean $m$ and variance $t=\sigma^2$. This has the ...
- 547
9
votes
1
answer
480
views
Higher moments of information and Renyi entropy
For a given discrete probability distribution, Shannon entropy can be though as an expectation value $\langle - \log p \rangle$ (see also: What is entropy, really?, What is the role of the logarithm ...
- 1,592
6
votes
2
answers
1k
views
Do binary symmetric channels maximize mutual information?
Consider the following setup: $(X, Y)$ is a doubly symmetric binary source with parameter $0 < p < 1/2$, i.e., $X \sim \text{Bernoulli}(1/2)$, $Z \sim \text{Bernoulli}(p)$ and $Y = X \oplus Z$. ...
5
votes
1
answer
414
views
Fokker-Planck: equivalence between linear spectral gap and nonlinear displacement convexity?
In a smooth, bounded and convex domain $\Omega\subset \mathbb R^d$ consider the usual linear Fokker-Planck equation with Neumann (some would say Robin) boundary conditions
\begin{equation}
\label{FP}
\...
- 5,163
5
votes
3
answers
490
views
Maximizing entropy under constraints
This question is about an extension of the variational principle in thermodynamical formalism when one adds linear constraints to the measures.
Consider the one-sided shift $\sigma:\mathcal{A}^\...
- 14.2k
4
votes
2
answers
2k
views
Convexity and semicontinuity of the relative entropy function
There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.
Let $M$ be a closed manifold and $\mathcal{...
4
votes
1
answer
223
views
Asymptotics for 'generalized" Kasteleyn's formula
A follow up on an earlier MO question.
Kasteleyn's formula for the number of domino tilings of a $2n\times 2n$ square
$\prod_{j=1}^n\prod_{k=1}^n \left( 4\cos^2(\pi j/(2n+1))+4\cos^2(\pi k/(2n+1))\...
- 41.8k
4
votes
3
answers
1k
views
Incremental entropy computation
After a quick internet search I found no method for incremental entropy computation.
Question 1
Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...
- 161
3
votes
1
answer
119
views
Conditions for: (local) lipschitz stability of I-projection
The following post builds on this post; I'll begin by quoting the setting.
Background from Previous Question:
$\newcommand\SS{P}\newcommand\TT{Q}$Call a Gaussian probability measure $\SS$ on $\mathbb{...
- 310
3
votes
3
answers
1k
views
bracketing number vs covering number
Just want to double check if the lemma on page 9 of this slides is correct:
http://www.math.leidenuniv.nl/~avdvaart/talks/09hilversum.pdf
Lemma: $N(\epsilon,\cal F,||\cdot||)\leq N_{[]}(2\epsilon,\cal ...
- 90
2
votes
0
answers
249
views
Prove or disprove a mutual information inequality
I have $n$ IID Bernoulli random variables denoted by $X_1,X_2,\ldots X_n$ with parameter $p$.
I am interested in knowing if the following inequality involving mutual information holds :
$\boxed{\max_{...
- 83
2
votes
1
answer
224
views
Sum of squares of middle binomial sums or 'Truncated mean' of binomial coefficients under binomial distribution
$\mu=1+\epsilon$ where $\epsilon>0$ holds.
1.Is there a good bound for $$T=\frac{\sum_{i=-\sqrt{\mu n\ln n}}^{\sqrt{\mu n\ln n}}\binom{n}{\frac n2 +i}^2}{2^n}?$$
This quantity can be ...
- 1,816
2
votes
2
answers
789
views
Defining a measure of uniformity for measurable subsets of $[0,1]^2$ w.r.t dimension $\alpha\in[0,2]$
Let $(X,d)$ be a metric space. If set $A\subseteq X$, let $H^{\alpha}$ be the $\alpha$-dimensional Hausdorff measure on $A$, where $\alpha\in[0,2]$ and $\text{dim}_{\text{H}}(A)$ is the Hausdorff ...
- 1
2
votes
0
answers
409
views
Interpretation of Shannon Entropy Application
Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Let entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$ be given
by $$H(\mathcal{A}...
- 13.7k
2
votes
1
answer
146
views
Tight sublinear estimates for a triple partial binomial summation
Is there tight estimates for the following logarithmic summation ($\gamma,\gamma'\in(0,1)$ and $\mu,\mu'>0$)
$$\log_2\Bigg(\sum_{t=\frac{n^{}}2-n^\gamma\sqrt{\mu\ln n}}^{\frac{n^{}}2+n^\gamma\sqrt{...
- 1,816
1
vote
1
answer
109
views
References: error and stability estimates for information projection
$\newcommand\SS{P}\newcommand\TT{Q}$I will call a Gaussian probability measure $\SS$ on $\mathbb{R}^d$ isotropic if its covariance matrix is diagonal with non-vanishing determinant; i.e. $\Sigma_{i,i}&...
- 310
1
vote
2
answers
146
views
A conjecture on 'truncated joint moments' of binomial coefficients under binomial distribution
This is similar in spirit to Sum of squares of middle binomial sums or 'Truncated mean' of binomial coefficients under binomial distribution but gives some total estimates. Though the other ...
- 1,816
1
vote
0
answers
286
views
Incremental computation of a conditional entropy
Is it possible to compute a conditional entropy (see the two following formulas) in an incremental manner ? That is, the sets C and K are not fix: each time we have a new element c, set K may increase ...
- 123
1
vote
0
answers
284
views
Can a lower bound for this weakening of Goldbach's conjecture be reached?
Say a non negative integer $r$ is a primality radius of $n$ if both $n-r$ and $n+r$ are prime, and that a non negative integer $w$ is a Galois radius of $m$ if $\omega(m-w)=\omega(m+w)=1$, where $\...
- 6,934
1
vote
0
answers
49
views
Limiting Entropy of deterministic sequences - 2
Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_a$ be limiting distribution $m\...
- 13.7k
1
vote
1
answer
70
views
Entropy dominance of certain restricted sequenes
Say you have positive $\{a_i\}_{i=1}^n$ and you have $p_i=\frac{a_i}{\sum_{i=1}^na_i}$, then assume you have a $C$ such that $C<2a_n\ll\sum_{i=1}^na_i$ (that is $C$ is not very large), then define $...
- 13.7k
1
vote
0
answers
398
views
When inequality in Mrs. Gerber's lemma is almost equality?
Let $X=x_1\ldots x_n$ be a random variable.
Assume that every $x_i$ takes values in $\{0,1\}$.
Assume also that for every $I \subseteq \{1,\ldots, n\}$ the Shannon entropy of random value $X_I$
[if $I ...
- 2,683
0
votes
1
answer
161
views
order of convergence of the conditional entropy
Let $X_n$ be a random variable distributed on $A_n:=\{1, \ldots, n\}$ and $g_n\colon A_n \to A_n$ such that $\Pr\big(X_n \neq g_n(X_n)\big) \to 0$. Putting $Y_n=g(X_n)$ then by Fano's inequality $$\...
- 2,433