20

There is a good reason you were having difficulties in proving this.
This was an old question of Rohlin (MR0126526) which was first disproved in the topological setting by Goodwyn (MR0314023) and later independently by Thouvenot and Ornstein. An explicit example appears in the paper by Ornstein and Weiss (MR0910005; page 133); this paper should be relevant ...

18

OK, having spent about 20 hours on the search of a nice proof (which extinguished my passion for beauty for the next several days at least), I'm resorting to the brute force. I will love to see someone else to avenge this pitiful defeat of mine...
As I mentioned in the comment, the key to the solution is the inequality
$$
(1+a)\log(1+b)+(1+b)\log(1+a)\ge 2(...

18

Many classically important probability distributions are maximum entropy distributions for suitable constraints, including the normal distribution, exponential distribution, and Poisson distribution. Viewing them as maximum entropy distributions gives a unified viewpoint for why such distributions occur often in practice.

13

The earliest reference I have found for this result is Entropy and maximal spacings for random partitions (E. Slud, 1978).
Theorem 2.2 states that the entropy $W_n=-\sum_{i=1}^n p_i \ln p_i$ of the random partition is asymptotically normally distributed for $n\rightarrow \infty$ as ${\cal N}(\ln n +\gamma-1,\alpha_n)$, with $\alpha_n={\cal O}(1/n)$.
(Note ...

answered Feb 13 '19 at 12:39

Carlo Beenakker

100k99 gold badges239239 silver badges350350 bronze badges

11

You can maximize the entropy using standard calculus of variations; you need to take into account the constraint that the probability distribution is properly normalized and that the mean is known, using Lagrange multipliers. You then find that the probability distribution is of the form:
$$p(x) = \frac{\alpha e^{\alpha x}} { (e^{\alpha b}-e^{\alpha a})}, ...

11

I'll only attempt to answer question 1.
Let's say you have 10 objects, divided into three blocks of respective sizes 2, 5 and 3. What's the probability that a random endomorphism of your set of objects preserves the blocks, i.e. sends each element of the set to an element of the same block? It's
$$
\frac{\text{number of block-preserving endos}}{\text{...

10

This is a theorem of K. Berg, that the Haar measure is the measure of maximal entropy for automorphisms of compact groups. See, for example, these lecture notes.
An information-theoretic approach has been developed in the context of scattering theory, mainly for the unitary group, but I imagine the results are readily transposed to the orthogonal group. The ...

answered Apr 18 '14 at 19:49

Carlo Beenakker

100k99 gold badges239239 silver badges350350 bronze badges

9

For metrics on $S^{2}$ with positive curvature, Hamilton introduced the
entropy $N\left( g\right) =-\int\ln(R\operatorname{Area})Rd\mu.$ If the
initial metric has $R>0,$ he proved that this is nondecreasing under the Ricci
flow on surfaces; note that $Rd\mu$ satisfies $(\frac{\partial}{\partial t}-\Delta
)(Rd\mu)=0.$ Let $T$ be the singular time; then
$$...

9

In finance, finding risk neutral probabilities can be done via max-entropy methods.
In short, you observe prices $p_i$ of a finite number of instruments $\phi_i$, and you seek a probability measure $P$ such that $p_i = \int\phi_i\,\mathrm{d}P$. There is in general no uniqueness of $P$, and finding such a $P$ can be challenging. Finding $P$ by a max-entropy ...

9

Claim. If $\|P-Q\|\leq\varepsilon\leq\frac{1}{2}$, then $|H(P)-H(Q)| \leq H(\varepsilon) + \varepsilon\log N$.
Proof.
Let $\varepsilon':=\|P-Q\|$.
Let $(X,Y)$ be an optimal coupling of $P$ and $Q$, so that
\begin{align}
\mathbb{P}(X\neq Y) = \|P-Q\| \;.
\end{align}
Using a standard construction, we can assume that $X$ and $Y$ have the particular form
\...

9

Denote $f(p)=p(1-p)$, $H(p)=-p\log p$, let $p_1,\dots,p_n$ denote all positive probabilities of our distribution, then $\sum p_i=1$, finally denote $s=\sum_i f(p_i)$. Then we need to prove the inequality
$$
\sum_i H(f(p_i)/s)\geqslant \sum_i H(p_i).
$$
Since $H$ is concave, it suffices to prove that
the multiset $\{p_1,\dots,p_n\}$ majorizes the multiset $\{...

answered Jan 12 '19 at 12:10

Fedor Petrov

65.8k66 gold badges150150 silver badges276276 bronze badges

8

I would rather look at this from the opposite end. The key difference between the discrete and continuous cases is that in the discrete case there is a canonical reference measure, namely the counting one (whose presence is so natural that it's often forgotten). However, if you consider the classical entropy as the differential entropy with respect to the ...

8

We can find some attempts for generalizing the notion of entropy of discrete random variables to random variables with general distribution function.
A straightforward way is to employ Riemann sum of the distribution function. So we start with a discrete random variable and then by making the intervals small enough, the entropy function is obtained. Denote ...

8

Three tutorial lectures on entropy and counting by David Galvin (2014) provides an answer to Q1:
The key property of Shannonâ€™s entropy that makes it useful as an enumeration tool is that over all random variables that take on at most $n$ values with positive probability, the ones with the largest entropy are those which are uniform on their ranges, and ...

answered Jan 10 '18 at 21:51

Carlo Beenakker

100k99 gold badges239239 silver badges350350 bronze badges

8

$\newcommand{\ep}{\varepsilon}
$
Let $X$ be any nonnegative random variable (r.v.) with finite mean $\mu>0$ and variance $\sigma^2<\infty$. For any real $u>0$, we have $\ln\frac xu\le\frac xu-1$ for all real $x>0$, whence
$x\ln x\le\frac{x^2}u+x\ln\frac ue$ and
\begin{equation*}
EX\ln X\le\frac{EX^2}u+EX\ln\frac ue=\frac{\sigma^2+\mu^2}u+\mu\...

7

Yes. If $X$ is minimal (every orbit is dense) then the only subshift $Y\subset X$ is $X$ itself. The Jewett-Krieger theorem allows the construction of minimal subshifts with positive entropy, which therefore have the property you desire (albeit somewhat vacuously). Googling "minimal subshifts with positive entropy" brings up this paper by Henk Bruin, ...

answered Dec 6 '13 at 19:46

Vaughn Climenhaga

7,52022 gold badges2222 silver badges4444 bronze badges

7

The answer is as follows. We need to show that
$$I(Z; X)-I(X; Y)\leq H(Z|Y).$$ The left hand side of this is simplified as $H(X|Y)-H(X|Z)$, so we need to show that $H(X|Y)-H(X|Z)\leq H(Z|Y)$. Since conditioning reduces the entropy we have
$$H(X|Y)-H(X|Z)\leq H(X|Y)-H(X|Y,Z)=I(X;Z|Y)\leq H(Z|Y).$$

7

Entropy arguments are part of the toolkit of the probabilistic method. So the first step in answering the question of how someone just interested in combinatorics might be led to entropy is to ask how they might be led to consider the probabilistic method. I'm assuming you already have some feeling for this—arguments that just require approximate ...

answered Jan 11 '18 at 5:25

Timothy Chow

46.3k1515 gold badges201201 silver badges363363 bronze badges

7

Suppose that $k$ is a fixed natural number, $n\to\infty$, and
\begin{equation}
a_i=\frac nk+o(n^{2/3})
\end{equation}
for each $i=1,\dots,k$.
Let
\begin{equation}
h_i:=\frac kn\,a_i-1=o(n^{-1/3}).
\end{equation}
By Stirling's formula:
\begin{equation}
m!\sim\sqrt{2\pi m}(m/e)^m
\end{equation}
as $m\to\infty$,
\begin{equation}
\binom{n}{a_1,\ldots,...

6

There is a nice interpretation of Perelman's monotonicity formulas in terms of optimal transportation, see e.g. these lecture notes by Peter Topping
http://homepages.warwick.ac.uk/~maseq/grenoble_20100324.pdf
It seems helpful to look at the elliptic case first. As discovered by Lott-Villani and Sturm nonnegative Ricci curvature can be characterized by the ...

6

Perelman himself wrote about his entropy formula for the Ricci flow that "The interplay of statistical physics and (pseudo)-riemannian geometry occurs in the subject of Black Hole Thermodynamics, developed by Hawking et al. Unfortunately, this subject is beyond my understanding at the moment."
Subsequently, this connection has been explored in some detail ...

answered May 18 '13 at 18:55

Carlo Beenakker

100k99 gold badges239239 silver badges350350 bronze badges

6

I'm not sure if this qualifies as simple (or if this is helpful at all), but we have
$$
\frac{\phi'_M(t)}{\phi_M(t)}=\sum_n\frac{1}{t-\lambda_n}
$$
Using the residue theorem, we can write
$$
H[M]=\frac{-1}{2\pi i}\oint\frac{\phi'_M(z)}{\phi_M(z)}z\log(z)\,dz
$$
where the integral is taken over a closed contour containing all of the eigenvalues of $M$ (I ...

6

I have found this book a useful reference:
Maximum-entropy Models in Science and Engineering
It may contain some pointers to applied work that you will find convincing (I don't have it on me right now) if you can get a copy.

6

maximizing $S=\sum_{n=0}^\infty p_n \log p_n$ with the constraint $\sum_n n p_n=\mu$ and $\sum_n p_n =1$ gives $p_n = a b^n$, with Lagrange multipliers $a=1/(\mu+1)$ and $b=\mu/(\mu+1)$ determined by the constraints, so this is indeed a geometric distribution.
what is the source you are referring to that says the distribution is Poisson?

answered Jan 24 '15 at 12:44

Carlo Beenakker

100k99 gold badges239239 silver badges350350 bronze badges

6

Such a function cannot exist.
Let $I$ be the image of $H_n$. Then $I$ is a countable dense subset of $[0, \log_2 n]$. If such a function $H_n'$ would exist, then $H_n'(C_a) = f\big(H_n\big(\tfrac{C_a}{K_a}\big)\big)$, where $f$ is an injective monotonic function $f:I\to\mathbb{Z}$. Such a function $f$ cannot exist: For any $x,y\in I$ there is $z\in I$ ...

6

$\newcommand{\de}{\delta}
\newcommand{\ep}{\epsilon}$
Note that $p\ln p-p$ is decreasing in $p\in[0,1]$, so that $p\ln p-p\le q\ln q-q$ and hence $p\ln p-q\ln q\le p-q=|p-q|$ if $0\le q\le p\le1$.
Next, take any real $c\ge1$. Note that $g(p):=p\ln p+cp$ (with $g(0):=0$) is convex in $p\in[0,1]$. So, assuming $0\le p\le q\le1$ and $q\ge e^{-c}$, we have
$...

6

$\newcommand{\De}{\Delta}
\newcommand{\ep}{\epsilon}
\newcommand{\R}{\mathbb{R}}$
Here is yet another answer providing the exact bound. This answer is perhaps a bit more elementary than the excellent answer given by user Algernon. Another advantage of this approach is that it produces the exact bound for any convex function $f$ in place of the function $p\...

6

edit
I realized there is a simpler construction that achieves the same (examples of top. mixing zero-entropy homeos on $S^3$): Instead of the bi-directional flow through cuboids, have a flow from bottom to top on the cylinder $D^2 \times [0,1]$, and have it slow down to rate $0$ on the boundary. Also, instead of all the $C_i$ playing the same role, have a ...

5

There is work on infinite-dimensional exponential families of measures which might be what you are looking for.
There are these possible references:
Scricciolo (2006). Convergence rates for Bayesian density estimation of infinite-dimensional exponential families
Rivoirard and Rousseau (2012). Posterior concentration rates for infinite dimensional ...

5

I think I managed to prove the entire inequality analytically. The whole proof is a bit long to post here (about 7 pages) and involves ugly looking expressions. I'll outline the general strategy I used:
Start with the original inequality
$$-\frac{(1-2q)}{(p \star q) (1- p \star q)\log\left(\frac{1-p}{p}\right)} + \frac{\log\left(\frac{1- p \star q}{p \...

Only top voted, non community-wiki answers of a minimum length are eligible

#### Related Tags

entropy × 197it.information-theory × 60

pr.probability × 57

ergodic-theory × 38

ds.dynamical-systems × 34

probability-distributions × 27

co.combinatorics × 15

measure-theory × 12

st.statistics × 12

inequalities × 12

reference-request × 11

asymptotics × 11

symbolic-dynamics × 11

fa.functional-analysis × 7

approximation-theory × 7

nonlinear-optimization × 7

statistical-physics × 7

binomial-coefficients × 6

real-analysis × 5

quantum-mechanics × 4

nt.number-theory × 3

graph-theory × 3

gt.geometric-topology × 3

algorithms × 3

computational-complexity × 3