# Tag Info

Accepted

### Geometric interpretations of the exponential of entropy

With apologies for promoting my own work, there's a whole book on the mathematics of the exponentials of various entropies: Tom Leinster, Entropy and Diversity: The Axiomatic Approach. Cambridge ...
• 27.4k
Accepted

### Entropy of composition

There is a good reason you were having difficulties in proving this. This was an old question of Rohlin (MR0126526) which was first disproved in the topological setting by Goodwyn (MR0314023) and ...
Accepted

### Information inequalities

Yes. The set of $2^n$ (or $2^n-1$ excluding the empty set) dimensional vectors formed by entropies is called the entropic region [1]. Inequalities on the entropic region not implied by the ...
Accepted

### Relative Entropy and p-norm

The argument below is not very elegant,but it is, indeed, a standard exercise. Let $g=\max(f-1,0)$. We shall prove that $$f\log f\le 2g+\frac 2{p-1}g^p\,.$$ The integration and Holder then give the ...
• 60.6k
Accepted

• 112k
Accepted

### Bipartite version of Hamming bound (two families of codewords with large Hamming distance)?

If the Hamming distance between $A$ and $B$ is at least $d$, it yields that $B$ is disjoint from the $(d-1)$-neighborhood of $A$. By isoperimetric inequality for a Boolean cube (Harper's theorem), the ...
• 104k

### Is there an axiomatic characterization of the entropy of a continuous random variable?

$\newcommand{\Si}{\Sigma}\newcommand{\PP}{\mathscr P}\newcommand{\PPP}{\mathfrak P}$Let $X$ be a discrete random variable (r.v.) taking distinct values $x_1,x_2,\dots$, and let $p_i:=P(X = x_i)$. ...
• 121k
Accepted

### Explanation for why an ideal fluid doesn't have increasing entropy?

This is a very important issue, to which an answer must be made in mathematical terms, rather than by waving hands. Yes, the Euler system (conservation of mass, momentum and energy) is time-reversible....
• 51.8k

### Is there an entropy proof for bounding a weighted sum of binomial coefficients?

By the Chernoff–Hoeffding theorem, the sum in question is $\le\exp(-nD(a||p))$ for $a\le p$, where $a:=\alpha$ and $$D(a||p):=a\ln\frac ap+(1-a)\ln\frac{1-a}{1-p},$$ the Kullback–Leibler divergence ...
• 121k
Accepted

### Gaussian distribution, maximum entropy and the heat equation

Both the Gaussian maximum entropy distribution and the Gaussian solution of the diffusion equation (heat equation) follow from the central limit theorem, that the limiting distribution of the sum of i....
• 182k

### What's the motivation of entropy as a combinatorical tool? What problems is it able to solve?

Three tutorial lectures on entropy and counting by David Galvin (2014) provides an answer to Q1: The key property of Shannon’s entropy that makes it useful as an enumeration tool is that over all ...
• 182k
Accepted

### Asymptotics of multinomial coefficients

Suppose that $k$ is a fixed natural number, $n\to\infty$, and \begin{equation*} a_i=\frac nk+o(n^{2/3}) \end{equation*} for each $i$; here in what follows, $i\in\{1,\dots,k\}$. Let \begin{...
• 121k

### What's the motivation of entropy as a combinatorical tool? What problems is it able to solve?

Entropy arguments are part of the toolkit of the probabilistic method. So the first step in answering the question of how someone just interested in combinatorics might be led to entropy is to ask ...
• 79.5k
Accepted

### Shannon entropy of $p(x)(1-p(x))$ is no less than entropy of $p(x)$

Denote $f(p)=p(1-p)$, $H(p)=-p\log p$, let $p_1,\dots,p_n$ denote all positive probabilities of our distribution, then $\sum p_i=1$, finally denote $s=\sum_i f(p_i)$. Then we need to prove the ...
• 104k

$\newcommand{\fx}{\lfloor X\rfloor}$ $\newcommand\Z{\mathbb{Z}}$ We shall prove more than requested: that $H(\fx)<\infty$ if $E\ln(1+|X|)<\infty$. Indeed, let $$p_n:=P(\fx=n),$$ so that $$H(\fx)=... • 121k 7 votes Accepted ### Covering number of Lipschitz functions Here is a reference to a more general result: Lipschitz functions over a doubling metric space (rather than [0,1]^d). The ||\cdot||_\infty \epsilon-metric entropy of such functions is, ... • 6,101 7 votes ### Can the differential entropy of a continuous distribution with lebesgue integrable density be negative infinity? For x\in(0,1), let $$f(x):=\frac1{x\ln^2\frac ex}. \tag{1}$$ Then f\ge0 and \int_0^1f(x)\,dx=1. On the other hand, \ln f(x)\sim\ln\frac ex as x\downarrow0. So,... • 121k 7 votes Accepted ### Discrete entropy of the integer part of a random variable Since \lfloor X\rfloor has finite entropy if and only if |\lfloor X\rfloor| has finite entropy, it suffices to consider random variables taking values in the natural numbers. Write p_n for \... • 22.6k 7 votes ### Geometric interpretations of the exponential of entropy If I may broaden the question somewhat to include other interpretations of the exponential of entropy, it is commonly used in ecology to measure how many different species there are in a community. In ... • 182k 7 votes Accepted ### Is there an entropy proof for bounding a weighted sum of binomial coefficients? Yes, if \alpha<p (if \alpha>p, the sum is almost 1). To see this, write$$ \sum_{\ell = 0}^{\alpha n} \binom{n}{\ell}p^\ell(1-p)^{n-\ell}\leqslant t^{-\alpha n}(pt+(1-p))^n $$for every t\... • 104k 7 votes ### 3-periodic point implies positive topological entropy The result you are looking for is contained as a particular case in Theorem 4.58, point ii), in the very nice reference work by Sylvie Ruette on interval topological dynamics. I add some (hopefully) ... 7 votes Accepted ### Fokker-Planck: equivalence between linear spectral gap and nonlinear displacement convexity? One has the ordering of the three \lambda's, i.e.$$ \lambda_{\text{convex}} \leq \lambda_{\text{LSI}} \leq \lambda_{\text{SG}}, $$where \lambda_{\text{convex}} is the one from convexity, \... • 1,081 7 votes Accepted ### Relative entropy equality for a sequence of Bernoulli random variables This equality does not hold in general. E.g., suppose that n=2,$$\big(P_p(X_1=i,X_2=j)\colon\; i=0,1,\,j=0,1\big)= \frac1{30}\,\begin{pmatrix} 4 & 9 \\ 9 & 8 \\ \end{pmatrix},$$and$$\...
• 121k
In the setting you describe, for each $\alpha \in (0,1)$ the $(1-\alpha,\alpha)$-Bernoulli measure is the unique measure achieving the maximum. The function $\alpha \mapsto \eta(\alpha)$ is the ...